Another outage this morning was reported here by @Josve05a: T106938

Again it was the language the wiki was in, sv on sv.wikipedia.org.

If we can't figure out what's causing this is there some other hack we can use? Or as @Ainali notes on that ticket, some way of monitoring this so we are alerted to it?

Can someone please put the purge suggestion in the description if that works?

In T93800#1502350, @Elitre wrote:

Can someone please put the purge suggestion in the description if that works?

Done. Feel free to edit further for clarity. Also might be a good idea to link this on-wiki and mediawiki docs where ever relevant; so far we've caught citoid being broken on en wiki, where we have a lot of testers/bug reporter, and sv wiki, where we have one very active tester/bug reporter, which makes me think this might be happening under the radar elsewhere.

Yesterday evening (CET) we had a citoid outage at Polish Wikipedia. No idea what went wrong, citoid window was simply not there in VE when you clicked on "Przypis" (Polish for "Reference"). I reported it at https://www.mediawiki.org/wiki/Topic:Sgikbv81nxsv7oy6, but it seems the problems disappeared on their own this morning.

It is my understanding that we need specific details to be able to understand what happens; Marielle, James etc., please list them here so that we know what to ask users, thank you.

In T93800#1502621, @Elitre wrote:

It is my understanding that we need specific details to be able to understand what happens; Marielle, James etc., please list them here so that we know what to ask users, thank you.

@Krinkle would have the best idea of what information we need- but for now I'd say just which wiki, when it happened, and what language (i.e. site language or if they were using a different language in preferences).

Happening on enwp right now for me. I haven't changed my lang.

Yup, just started happening on enwp for me, even in a private window.

I purged it, but it's still happening for me in a private window unless I add debug=true as a parameter.

Now it works on enwp again.

Currently Citoid is down for me at en-Wiki (FF 39.0, Windows XP, vector skin, desktop).
The main menu shows "Cite" with the usual small arrow to open the menu.
Inside the menu the list shows as a plain list with only 1 column: Website, Book, News, Journal, Basic, Re-use with their respective icons to the left. No "Citoid" function at all.

If you need further info, please let me know.

Quick update: purged per the advice on top of this thread (thanks) and Citoid and regular UI are available again.

Happening on live again

Now on sv.wp again

And again on enwiki just now.

If clicking on the link provided in the description is enough, that's what I did, but it took some mins for the change to be effective (maybe something we should add in the description).

Now reported on de.wp. I'm applying the workaround.

@Krinkle have you had a chance to examine the bug in production yet? Do you need us to stop clearing the cache so you have a live example?

In T93800#1576802, @Mvolz wrote:

@Krinkle have you had a chance to examine the bug in production yet? Do you need us to stop clearing the cache so you have a live example?

Indeed. I understand why we always clear it (it's production after all) but since I've exhausted all obvious places to look based on static analysis of the code, I can't narrow it down further until we have live specimen to hunt.

However I'm not always around to give it a try, so unless I'm online and available (ping me on IRC), it's okay to try purging it in order to restore Citoid functionality.

Having to ping you on IRC was a step I was not aware of. We can all consider doing that the next time this happens.

I actually just found one, and Krinkle is looking into it now.

That was nl.wp, and Krinkle just fixed it.

I confirmed that this bug is not caused by a race condition in Varnish during a deployment (e.g. the first request coming in when deployment sync is still busy). Even when bypassing Varnish with a cache-busting query parameter, the fresh response from MediaWiki directly has the broken message.

I did narrow it down to the MessageBlobStore database tables. When retrieving it from there directly, the cached blob consistently contains the broken <key> fallback value both for "nl" (the content language of nl.wikipedia.org) as well as for "en".

$ mwscript eval.php --wiki nlwiki
> $configFactory = ConfigFactory::getDefaultInstance();
> $rl = new ResourceLoader( $configFactory->makeConfig( 'main' ) );

> $m = MessageBlobStore::getInstance();
> require_once "$IP/tests/phpunit/includes/TestingAccessWrapper.php";
> $m = TestingAccessWrapper::newFromObject( $m );

> return $m->getFromDB( $rl, array( 'ext.citoid.visualEditor' ), 'nl' );
array(1) {
  ["ext.citoid.visualEditor"]=>
  string(1174) "{"citoid-520-error":"Het was niet mogelijk de gegevens van de opgegeven identifier op te halen.",
  .."citoid-template-type-map.json":"\u003Ccitoid-template-type-map.json\u003E",..}"
}
> return $m->getFromDB( $rl, array( 'ext.citoid.visualEditor' ), 'en' );
array(1) {
  ["ext.citoid.visualEditor"]=>
  string(1072) "{"citoid-520-error":"Unable to retrieve information from the provided identifier.",
  .."citoid-template-type-map.json":"\u003Ccitoid-template-type-map.json\u003E", ..}

Purging the MediaWiki namespace page on-wiki seems to fix it. Following that code path: ActionPurge -> doPurge::doPurge:

 MessageCache::singleton()->replace( $this->mTitle->getDBkey(), $text );
...
 new MessageBlobStore()
    ->updateMessage( $wgContLang->lcfirst( $msg ) );

Which replaces the database entry directly from the in-memory wikitext value during the purge. This then bumps the timestamp of the affected MessageBlobStore rows, which then propagates through the ResourceLoader startup module to actual users (within 5 minutes).

The problem must reside in the population of MessageBlobStore. The source for MessageBlobStore is wfMessage($key)->inLanguage($lang)->plain().

I suspect the following race condition exists

When the Citoid VE module is changed and this change is deployed (through the train or SWAT). For some reason, during this deployment there is a brief moment during which the localisation cache or message cache of one or more app servers is not working properly and misses this key. This then stays permanently in the MessageBlobStore until it has a reason to vacate the value (e.g. a purge or another deployment).

I also considered a race condition related to localisation cache being rebuild (during nightly LocalisationUpdate, or during the branch cut during first phase of the weekly train). However these values are not stored in LocalisationCache (it's a local wiki override, not from translatewiki). These are stored in MessageCache.

@Krinkle, does your explanation above also cover why only some languages are affected, while others aren't?

@Krinkle, if it's a race condition on deploy, wouldn't it show up immediately after a deploy? And deploy on en wiki occurs on Thursdays right? We had a report of citoid missing on en wiki (T104944) on a Tuesday and it's highly unlikely it went 5 days without being reported... although possible I guess... I also remember one that was a Sat morning or something.

@Elitre, I think so, since each message is a separate row in the DB, it could miss some languages and not others.

The new branch from the MediaWiki deployment train reaches enwiki on Thursday indeed. On Tuesday, when the new branch is created, we use Scap to regenerate the localisation caches. I believe this also re-creates the localisation cache for the "older" branch. And there is also the nightly localisation update.

Change 236028 had a related patch set uploaded (by Esanders):
Provide alternative access to citoid-template-type-map.json

https://gerrit.wikimedia.org/r/236028

Change 236028 merged by jenkins-bot:
Provide alternative access to citoid-template-type-map.json

https://gerrit.wikimedia.org/r/236028

Change 236218 had a related patch set uploaded (by Jforrester):
Provide alternative access to citoid-template-type-map.json

https://gerrit.wikimedia.org/r/236218

Change 236221 had a related patch set uploaded (by Alex Monk):
Actually use new backup message

https://gerrit.wikimedia.org/r/236221

Change 236221 merged by jenkins-bot:
Actually use new backup message

https://gerrit.wikimedia.org/r/236221

Change 236218 merged by jenkins-bot:
Provide alternative access to citoid-template-type-map.json

https://gerrit.wikimedia.org/r/236218

Change 236222 had a related patch set uploaded (by Jforrester):
Actually use new backup message

https://gerrit.wikimedia.org/r/236222

Change 236222 merged by jenkins-bot:
Actually use new backup message

https://gerrit.wikimedia.org/r/236222

Change 252159 had a related patch set uploaded (by Krinkle):
resourceloader: Log if MessageBlobStore failed to fetch a message

https://gerrit.wikimedia.org/r/252159

Change 252159 merged by jenkins-bot:
resourceloader: Log if MessageBlobStore failed to fetch a message

https://gerrit.wikimedia.org/r/252159

Change 252393 had a related patch set uploaded (by Krinkle):
resourceloader: Log if MessageBlobStore failed to fetch a message

https://gerrit.wikimedia.org/r/252393

Change 252393 merged by jenkins-bot:
resourceloader: Log if MessageBlobStore failed to fetch a message

https://gerrit.wikimedia.org/r/252393

Is this resolved? I notice the fall back message was removed from citoid, so I assume so?

🙂 Hello, I need help with citoid extension after installing its tells me this message on the photo in red can you tell me what to do to fix this problem?

@Doobwi This is not a correct place to ask for help. I suggest to try the resources in https://www.mediawiki.org/wiki/Project:Support_desk