Create Federated Wikibase instance on Beta Commons
Closed, ResolvedPublic

Description

Our structured data for Commons test system (http://structured-commons.wmflabs.org) works fine, but now that we're approaching launch time, we'll need to have the system available on Beta too (with a plan to migrate it to production) Ideally it should take items and properties from Wikidata.

Restricted Application added a project: Wikidata. · View Herald TranscriptSep 18 2018, 6:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Ladsgroup I believe you set up the federated system on Labs for us. Any chance you could handle this one too? :)

@Ladsgroup I believe you set up the federated system on Labs for us. Any chance you could handle this one too? :)

Sure thing, let me get it done soon (today is a little bit crazy, hopefully tomorrow)

Jdforrester-WMF moved this task from To Do to Deployment/Config on the SDC Engineering board.

So I'm getting back to this (sorry for being so late). It's not possible to use the items and properties from Wikidata but it's possible to use from wikidata in beta cluster (https://wikidata.beta.wmflabs.org). It's also not possible to just make a new instance and install mediawiki on it but it's possible to turn the commons installation in beta cluster (https://commons.wikimedia.beta.wmflabs.org/wiki/Main_Page) into a wikibase repo with mediainfo extension enabled and federation working (basically doing whatever you're planning to do on commons, in beta cluster instead). Is it what you want? if yes, I do it.

So I'm getting back to this (sorry for being so late). It's not possible to use the items and properties from Wikidata but it's possible to use from wikidata in beta cluster (https://wikidata.beta.wmflabs.org). It's also not possible to just make a new instance and install mediawiki on it but it's possible to turn the commons installation in beta cluster (https://commons.wikimedia.beta.wmflabs.org/wiki/Main_Page) into a wikibase repo with mediainfo extension enabled and federation working (basically doing whatever you're planning to do on commons, in beta cluster instead). Is it what you want? if yes, I do it.

I believe this task is asking for WikibaseRepo to be installed to Beta Cluster Commons, yes. (NOT MediaInfo yet, please.)

I believe this task is asking for WikibaseRepo to be installed to Beta Cluster Commons, yes. (NOT MediaInfo yet, please.)

hmm, If we want to install WikibaseRepo on commons in beta cluster, we need to add at least one entity type. Item, Property and Lexeme are already defined and used by wikidata in beta. We can enable repo without having any entity type enabled (which I'm not sure even if it's possible) but in that case, it's not possible to test anything, there's nothing enabled to test. In other words, We can't enable it unless we have mediainfo enabled there too.

I believe this task is asking for WikibaseRepo to be installed to Beta Cluster Commons, yes. (NOT MediaInfo yet, please.)

hmm, If we want to install WikibaseRepo on commons in beta cluster, we need to add at least one entity type. Item, Property and Lexeme are already defined and used by wikidata in beta. We can enable repo without having any entity type enabled (which I'm not sure even if it's possible) but in that case, it's not possible to test anything, there's nothing enabled to test. In other words, We can't enable it unless we have mediainfo enabled there too.

Yes, that's T180981 which depends on this.

Given the reply @Jdforrester-WMF gave above, do you have all the answers you need to finish this, @Ladsgroup ?

Change 446843 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] Install but don't enable the WikibaseMediaInfo extension, part III

https://gerrit.wikimedia.org/r/446843

Change 446843 merged by jenkins-bot:
[operations/mediawiki-config@master] Install but don't enable the WikibaseMediaInfo extension, part III

https://gerrit.wikimedia.org/r/446843

Initial enablement (without federation) now merged into production config (to be used on Beta Cluster at first). Doing the real federation is needed for the next phase (statements), so we'll have to work out how to re-write those config files.

Ramsey-WMF triaged this task as High priority.Nov 15 2018, 6:39 PM

We're quickly approaching the point where we need this up on Beta because we're about to start doing dev on statements after testing file captions on Beta.

Abit added a subscriber: Abit.Nov 15 2018, 6:39 PM

I will do it tomorrow. Feel free to pick it up if it wasn't done by Wednesday.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptMon, Nov 19, 5:33 PM
Abit added a comment.Mon, Nov 19, 7:09 PM

Thanks so much, Amir! ^_^

Change 474889 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Add federation configs for beta

https://gerrit.wikimedia.org/r/474889

Change 474889 merged by jenkins-bot:
[operations/mediawiki-config@master] Add federation configs for beta

https://gerrit.wikimedia.org/r/474889

Mentioned in SAL (#wikimedia-operations) [2018-11-20T18:17:19Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Create Federated Wikibase instance on Beta Commons (T204748) (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2018-11-20T18:19:15Z] <ladsgroup@deploy1001> Synchronized wmf-config/Wikibase.php: Create Federated Wikibase instance on Beta Commons, part II (T204748) (duration: 00m 47s)

Change 474963 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] Use integer for namespace id

https://gerrit.wikimedia.org/r/474963

Change 474963 merged by jenkins-bot:
[operations/mediawiki-config@master] Use integer for namespace id

https://gerrit.wikimedia.org/r/474963

Change 474993 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[operations/mediawiki-config@master] labs: Add mediainfo to federation config

https://gerrit.wikimedia.org/r/474993

Change 474993 merged by jenkins-bot:
[operations/mediawiki-config@master] labs: Add mediainfo to federation config

https://gerrit.wikimedia.org/r/474993

Addshore moved this task from incoming to in progress on the Wikidata board.Wed, Nov 21, 8:19 AM

\o/
Done from our side. @Ramsey-WMF please close if also good from your side.

Jdforrester-WMF closed this task as Resolved.Mon, Nov 26, 3:55 PM
Jdforrester-WMF removed a project: Patch-For-Review.

Change 476391 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] Revert "labs: Add mediainfo to federation config"

https://gerrit.wikimedia.org/r/476391

Jdforrester-WMF reopened this task as Open.Wed, Nov 28, 9:15 PM

Unfortunately this triggered some Wikibase code that wasn't previously running, which broke all of WBMI. :-(

[W-8D66wQBHcAAAJ5GoUAAAAB] /wiki/File:Redsq.png Wikimedia\Assert\ParameterTypeException from line 89 of /srv/mediawiki/php-master/vendor/wikimedia/assert/src/Assert.php: Bad value for parameter $mediaInfoNamespace: must be a integer

Backtrace:

#0 /srv/mediawiki/php-master/extensions/WikibaseMediaInfo/src/Services/MediaInfoIdLookup.php(28): Wikimedia\Assert\Assert::parameterType(string, NULL, string)
#1 /srv/mediawiki/php-master/extensions/WikibaseMediaInfo/src/Services/MediaInfoServices.php(23): Wikibase\MediaInfo\Services\MediaInfoIdLookup->__construct(NULL)
#2 /srv/mediawiki/php-master/extensions/WikibaseMediaInfo/WikibaseMediaInfo.entitytypes.php(129): Wikibase\MediaInfo\Services\MediaInfoServices::getMediaInfoIdLookup()
#3 [internal function]: Closure$#6()
#4 /srv/mediawiki/php-master/extensions/Wikibase/repo/includes/Content/EntityContentFactory.php(217): call_user_func(Closure$#6;1820)
#5 /srv/mediawiki/php-master/extensions/Wikibase/repo/RepoHooks.php(136): Wikibase\Repo\Content\EntityContentFactory->getContentHandlerForType(string)
#6 /srv/mediawiki/php-master/includes/content/ContentHandler.php(320): Closure$Wikibase\RepoHooks::onSetupAfterCache(string)
#7 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(1464): ContentHandler::getForModelID(string)
#8 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(1628): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#9 [internal function]: Closure$MediaWiki\Revision\RevisionStore::loadSlotRecords(MediaWiki\Revision\SlotRecord)
#10 /srv/mediawiki/php-master/includes/Revision/SlotRecord.php(309): call_user_func(Closure$MediaWiki\Revision\RevisionStore::loadSlotRecords;471, MediaWiki\Revision\SlotRecord)
#11 /srv/mediawiki/php-master/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\SlotRecord->getContent()
#12 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(226): MediaWiki\Revision\RevisionRecord->getContent(string, integer, NULL)
#13 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(201): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#14 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(133): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#15 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#16 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(197): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;492, MediaWiki\Revision\RenderedRevision, array)
#17 /srv/mediawiki/php-master/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#18 /srv/mediawiki/php-master/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#19 /srv/mediawiki/php-master/includes/page/Article.php(774): PoolCounterWork->execute()
#20 /srv/mediawiki/php-master/includes/page/ImagePage.php(147): Article->view()
#21 /srv/mediawiki/php-master/includes/actions/ViewAction.php(68): ImagePage->view()
…

I'm reverting 474993 to start with, to see if that fixes it.

Change 476391 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "labs: Add mediainfo to federation config"

https://gerrit.wikimedia.org/r/476391

Change 476403 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/WikibaseMediaInfo@master] Don't check for integer namespace in MediaInfoIdLookup

https://gerrit.wikimedia.org/r/476403

Abbe98 added a subscriber: Abbe98.Thu, Nov 29, 6:46 AM

Change 476403 merged by Ladsgroup:
[mediawiki/extensions/WikibaseMediaInfo@master] Don't check for integer namespace in MediaInfoIdLookup

https://gerrit.wikimedia.org/r/476403

Ladsgroup closed this task as Resolved.Thu, Nov 29, 2:18 PM

It's resolved now.

Jdforrester-WMF reopened this task as Open.Tue, Dec 11, 7:26 PM
Jdforrester-WMF added a subscriber: daniel.

@daniel can give more details about the inner parts of Wikibase, but we had to disable this to unbreak WBMI :-(

Change 476403 merged by Ladsgroup:
[mediawiki/extensions/WikibaseMediaInfo@master] Don't check for integer namespace in MediaInfoIdLookup

https://gerrit.wikimedia.org/r/476403

That fix is wrong, and hiding an error. It should be reverted and the calling code fixed instead.

I don't think this is the reason that MediaInfo doesn't work on beta, though. That still needs more investigation.

Change 479417 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/WikibaseMediaInfo@master] Fail hard if no entity namespace is configured for MediaInfo.

https://gerrit.wikimedia.org/r/479417

Change 479419 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@master] Fail hard if an entity namespace is not configured.

https://gerrit.wikimedia.org/r/479419

If MediaInfo items only exist as slots, then I don't really understand why they need a namespace

I don't think this is the reason that MediaInfo doesn't work on beta, though. That still needs more investigation.

After some investigation, I now believe that they have a common cause: EntityNamespaceMapping is initialized with no namespace set for MediaInfo. That should be impossibly by virtue of MediaInfoHooks::onWikibaseRepoEntityNamespaces(). Further investigation is needed, it seems.

daniel added a comment.EditedThu, Dec 13, 12:52 PM

If MediaInfo items only exist as slots, then I don't really understand why they need a namespace

Because slots are on pages, and pages are in namespaces. Every entity type has a namespace and a slot.

Also, the way to configure what slot the entity goes into is done in the entity namespace mapping (for reasons of backwards compatibility), using the syntax <namespace>/<slot>. If the slot is omitted, the main slot is assumed.

MediaInfo should always be registered as 6/mediainfo, with 6 being the ID of the File namespace. This is done automatically by the hook. But something is not working right...

So should MediaInfo items have the page namespace, or their own namespace?

So should MediaInfo items have the page namespace, or their own namespace?

There is no page namespace (or rather, all namespaces are page namespaces). MediaInfo lives in the File names (ID 6), in the mediainfo slot.

Change 479497 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[operations/mediawiki-config@master] Allow MediaInfo on commons to reference Wikidata items.

https://gerrit.wikimedia.org/r/479497

Change 479497 merged by jenkins-bot:
[operations/mediawiki-config@master] [Beta Only] Allow MediaInfo on Commons to reference Wikidata items

https://gerrit.wikimedia.org/r/479497

Mentioned in SAL (#wikimedia-operations) [2018-12-13T18:22:41Z] <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: T204748 Rename repo-only Wikibase config for clarity [no-op] (duration: 00m 45s)

Mentioned in SAL (#wikimedia-operations) [2018-12-13T18:24:29Z] <jforrester@deploy1001> Synchronized wmf-config/Wikibase.php: T204748 [Beta only] Use newly-fixed config for Wikibase->Commons federation (duration: 00m 44s)

Change 479417 merged by jenkins-bot:
[mediawiki/extensions/WikibaseMediaInfo@master] Fail hard if no entity namespace is configured for MediaInfo.

https://gerrit.wikimedia.org/r/479417

Change 479419 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fail hard if an entity namespace is not configured.

https://gerrit.wikimedia.org/r/479419

daniel closed this task as Resolved.Thu, Dec 13, 7:59 PM

Thanks for testing, James!