Page MenuHomePhabricator

[Regression wmf.13] Wikidata localisation is broken
Closed, ResolvedPublicPRODUCTION ERROR

Description

Wikidata localisation seems to be broken quite a lot; when viewing in (e.g.) Czech (as always), I notice many broken things everywhere (e.g. a completely random example):

  • the sitelinks box is captioned “⧼wikibase-sitelinks-wikipedia⧽”
  • date statements show “http://www.wikidata.org/entity/Q1985727” instead of the calendar name
  • history displays things like “wbcreateclaim-create:1|” instead of the human-readable text

The same happens when I use e.g. uselang=de; on the other hand, the page looks fine with uselang=en.


Current status (2019-07-11Z20:57): Wikidata rolled back to wmf.11. Breakage visible on Test Wikidata, e.g. https://test.wikidata.org/wiki/Q64507

Event Timeline

Hrm. That is bizarre.

$ grep -c wikibase-sitelinks-wikipedia /srv/mediawiki-staging/php-1.34.0-wmf.13/cache/l10n/upstream/l10n_cache-cs.cdb.json
0
$ grep -c wikibase-sitelinks-wikipedia /srv/mediawiki-staging/php-1.34.0-wmf.11/cache/l10n/upstream/l10n_cache-cs.cdb.json
2

Nothing looks to have changed with the actual json in the extension:

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+/wmf/1.34.0-wmf.11/lib/i18n/cs.json#53

vs

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Wikibase/+/wmf/1.34.0-wmf.13/lib/i18n/cs.json#53

So either scap made a bad change, something scap shelled out to made a bad change, or there was some runtime error, methinks.

thcipriani raised the priority of this task from High to Unbreak Now!.Jul 11 2019, 8:27 PM

UBN since it's a deployment blocker.

It's not in the cdb file either

$ strings /srv/mediawiki-staging/php-1.34.0-wmf.13/cache/l10n//l10n_cache-cs.cdb | grep -c 'wikibase-sitelinks-wikipedia'
0
$ strings /srv/mediawiki-staging/php-1.34.0-wmf.11/cache/l10n/l10n_cache-cs.cdb | grep -c 'wikibase-sitelinks-wikipedia'
2

So it's not an error in the scap logic that generates json from cdb files, but in the logic/runtime that generates the cdbs.

Change 522181 had a related patch set uploaded (by Jeena Huneidi; owner: Jeena Huneidi):
[operations/mediawiki-config@master] Revert wikidata to 1.34.0-wmf.11 refs T227814

https://gerrit.wikimedia.org/r/522181

Change 522181 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert wikidata to 1.34.0-wmf.11 refs T227814

https://gerrit.wikimedia.org/r/522181

hrm, so WikibaseLib is in ExtensionMessages for 1.34.11, but not in ExtensionsMessages for 1.34.13

$ diff wmf-config/ExtensionMessages-1.34.0-wmf.{11,13}.php
...
918,925d922
<   'WikibaseLib' =>
<   array (
<     0 => "$IP/extensions/Wikibase/lib/../lib/i18n",
<   ),
<   'WikibaseView' =>
<   array (
<     0 => "$IP/extensions/Wikibase/view/../view/lib/wikibase-data-values-value-view/i18n",                                                           
<   ),
...

This file is generated by mergeMessageFileLists.php.

In the scap logs I see:

2019-07-09T19:44:11 Loading data from /srv/mediawiki-staging/php-1.34.0-wmf.13/extensions/Wikibase/client/WikibaseClient.php
2019-07-09T19:44:11 Loading data from /srv/mediawiki-staging/php-1.34.0-wmf.13/extensions/Wikibase/repo/Wikibase.php

The 2 previous week's output looks the same:

2019-06-25T19:51:39 Loading data from /srv/mediawiki-staging/php-1.34.0-wmf.11/extensions/Wikibase/client/WikibaseClient.php
2019-06-25T19:51:39 Loading data from /srv/mediawiki-staging/php-1.34.0-wmf.11/extensions/Wikibase/repo/Wikibase.php

Nothing obvious in the repo's logs.

Nothing obvious in the repo's logs.

I found https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/515031/ which looks suspect and is fairly recent; however, that should have affected 1.34.0-wmf.11 as well, I think.

Yeah, I remembered that one but as you say it went out in wmf.10.

[[ https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/maintenance/mergeMessageFileList.php | mergeMessageFileList ]] which is what scap shells out to to handle building ExtensionMessages does a require_once on each of the files listed in the extension-list file. It then dumps $wgMessagesDirs into the ExtensionMessages file.

This seems somewhat straightforward; however, maybe there's a possible race-condition here somewhere?

Mentioned in SAL (#wikimedia-operations) [2019-07-11T22:07:16Z] <thcipriani@deploy1001> Started scap: no op scap sync to rebuild l10n-cache (T227814)

Ran in another terminal window after I got the message 22:10:59 Updating ExtensionMessages-1.34.0-wmf.13.php from scap

$ grep -i wikibaselib wmf-config/ExtensionMessages-1.34.0-wmf.13.php 
$ grep -i wikibaselib wmf-config/ExtensionMessages-1.34.0-wmf.13.php 
  'WikibaseLib' => 

Mentioned in SAL (#wikimedia-operations) [2019-07-11T22:26:50Z] <thcipriani@deploy1001> Finished scap: no op scap sync to rebuild l10n-cache (T227814) (duration: 19m 34s)

greg claimed this task.
greg added a subscriber: greg.

Thanks and sorry all.

Really fixed? Testwikidata is still bad for me, the monolingual text statement’s content for example is not visible at all.

So did we figure out what happened? (The change @thcipriani linked is by me, and I also think it feels suspect, so I hope this is not my fault 😬)

Edit: And if this was only fixed by reverting the train, and the issue persists on Test Wikidata, shouldn’t the task remain open?

Test Wikidata works fine for me, the broken display disappeared on purge.

So did we figure out what happened? (The change @thcipriani linked is by me, and I also think it feels suspect, so I hope this is not my fault 😬)

Edit: And if this was only fixed by reverting the train, and the issue persists on Test Wikidata, shouldn’t the task remain open?

1.34.0-wmf.13 is on all wikis, this was fixed by re-running scap sync.

I'm not exactly happy about that outcome since I don't understand how it failed the first go-round. I looked closely at mergeMessageFileList.php today (which is the file that generates ExtensionsMessages-[version].php) and I'm not clear on how it works at all.

Essentially what mergeMessageFileList.php seems to do is do a require_once for all php files in extension-list (pre-seeding the $wgExtensionMessagesFiles and $wgMessagesDirs variables), then it extracts the global variables wgExtensionMessagesFiles and wgMessagesDirs from any json file in extension-list; however, when a file you've required adds a json file to the ExtensionRegistration queue (as in the Wikibase patch) when is that queue processed? I don't know, but it seemingly does (at least most of the time).

So did we figure out what happened? (The change @thcipriani linked is by me, and I also think it feels suspect, so I hope this is not my fault 😬)

Looking at https://www.mediawiki.org/wiki/Manual:Extension_registration#Migration_for_extension_developers it looks like you should keep $wgMessagesDirs specifically for mergeMessageFileList (see the comment in the example code on that page)

<?php
if ( function_exists( 'wfLoadExtension' ) ) {
	wfLoadExtension( 'FooBar' );
	// Keep i18n globals so mergeMessageFileList.php doesn't break
	$wgMessagesDirs['FooBar'] = __DIR__ . '/i18n';

Another example is Charlie Barnes (Q21005322):

Screenshot 2019-07-14 at 11.44.08.png (168×1 px, 37 KB)

Screenshot 2019-07-14 at 11.44.18.png (590×937 px, 117 KB)

Once again, this solved itself by purging the page, probably just a cached version of the broken state.

Looking at https://www.mediawiki.org/wiki/Manual:Extension_registration#Migration_for_extension_developers it looks like you should keep $wgMessagesDirs specifically for mergeMessageFileList (see the comment in the example code on that page)

<?php
if ( function_exists( 'wfLoadExtension' ) ) {
	wfLoadExtension( 'FooBar' );
	// Keep i18n globals so mergeMessageFileList.php doesn't break
	$wgMessagesDirs['FooBar'] = __DIR__ . '/i18n';

I asked about that on the talk page and the conclusion was that this is only necessary for extension-list entries, so removing the i18n globals in the WikibaseLib entry point should have been fine. If mergeMessageFileList.php does a require_once of the WikibaseRepo or -Client entry points, then it would also end up with the WikibaseLib entry point, and perhaps in that case we would need to keep the i18n globals after all… but it seems like, at least sometimes, having them only in the JSON file works? No idea why, though.

I think we should try to finish the Wikibase move to extension registration soon, and we’re actively working on it, but until we’re done with that perhaps it’s better to add the i18n globals back to the PHP entry points just to be sure?

Anyways, closing this task again since it seems like it’s still fixed, broken pages just need to be purged.

Change 524539 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@master] Define $wgMessagesDirs in WikibaseLib PHP entry point

https://gerrit.wikimedia.org/r/524539

Change 524539 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Define $wgMessagesDirs in WikibaseLib PHP entry point

https://gerrit.wikimedia.org/r/524539

Change 524573 had a related patch set uploaded (by Jforrester; owner: Lucas Werkmeister (WMDE)):
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.14] Define $wgMessagesDirs in WikibaseLib PHP entry point

https://gerrit.wikimedia.org/r/524573

Change 524573 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.34.0-wmf.14] Define $wgMessagesDirs in WikibaseLib PHP entry point

https://gerrit.wikimedia.org/r/524573

Mentioned in SAL (#wikimedia-operations) [2019-07-22T15:35:12Z] <jforrester@deploy1001> Synchronized php-1.34.0-wmf.14/extensions/Wikibase/lib/WikibaseLib.php: T227814 Wikibase: Define $wgMessagesDirs in WikibaseLib PHP entry point (duration: 00m 48s)

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:05 PM