krinkle@fluorine:/a/mw-log$ ack-grep c6a790a7 exception.log -A 20
2013-07-25 16:51:01 mw1177 enwikivoyage: [c6a790a7] /wiki/Tarragona Exception from line 315 of /usr/local/apache/common-local/php-1.22wmf11/includes/MagicWord.php: Error: invalid magic word 'noexternallanglinks'
#0 /usr/local/apache/common-local/php-1.22wmf11/includes/MagicWord.php(238): MagicWord->load('noexternallangl...')
#1 /usr/local/apache/common-local/php-1.22wmf11/includes/parser/Parser.php(4904): MagicWord::get('noexternallangl...')
#2 /usr/local/apache/common-local/php-1.22wmf11/extensions/Wikibase/client/WikibaseClient.hooks.php(673): Parser->setFunctionHook('noexternallangl...', '\Wikibase\NoLan...', 1)
#3 [internal function]: Wikibase\ClientHooks::onParserFirstCallInit(Object(Parser))
#4 /usr/local/apache/common-local/php-1.22wmf11/includes/Hooks.php(199): call_user_func_array('\Wikibase\Clien...', Array)
#5 /usr/local/apache/common-local/php-1.22wmf11/includes/GlobalFunctions.php(3835): Hooks::run('ParserFirstCall...', Array)
#6 /usr/local/apache/common-local/php-1.22wmf11/includes/parser/Parser.php(270): wfRunHooks('ParserFirstCall...', Array)
#7 [internal function]: Parser->firstCallInit()
#8 /usr/local/apache/common-local/php-1.22wmf11/includes/StubObject.php(79): call_user_func_array(Array, Array)
#9 /usr/local/apache/common-local/php-1.22wmf11/includes/StubObject.php(99): StubObject->_call('firstCallInit', Array)
#10 /usr/local/apache/common-local/php-1.22wmf11/includes/cache/MessageCache.php(996): StubObject->__call('firstCallInit', Array)
#11 /usr/local/apache/common-local/php-1.22wmf11/includes/cache/MessageCache.php(996): StubObject->firstCallInit()
#12 /usr/local/apache/common-local/php-1.22wmf11/includes/cache/MessageCache.php(974): MessageCache->getParser()
#13 /usr/local/apache/common-local/php-1.22wmf11/includes/Message.php(696): MessageCache->transform('$1 ??? Travel g...', true, Object(Language), Object(Title))
#14 /usr/local/apache/common-local/php-1.22wmf11/includes/Message.php(496): Message->transformText('$1 ??? Travel g...')
#15 /usr/local/apache/common-local/php-1.22wmf11/includes/Message.php(553): Message->toString()
#16 /usr/local/apache/common-local/php-1.22wmf11/includes/OutputPage.php(846): Message->text()
#17 /usr/local/apache/common-local/php-1.22wmf11/includes/OutputPage.php(889): OutputPage->setHTMLTitle(Object(Message))
#18 /usr/local/apache/common-local/php-1.22wmf11/includes/Article.php(555): OutputPage->setPageTitle('Tarragona')
#19 /usr/local/apache/common-local/php-1.22wmf11/includes/actions/ViewAction.php(44): Article->view()

Not limited to HTTPS. That url just happened to be cached on HTTP still and not over HTTPS. Any url that is not cached (or invalidated by purge/edit) will result in fatal error.

e.g.

• http://en.wikivoyage.org/wiki/Tarragona?action=history
• http://en.wikivoyage.org/wiki/Tarragona

We're back up, but I'm leaving this bug open for now to document what happened (for Aude and Reedy).

• the mergeMessageFiles script failed(?) due to wikibase bad initialization order
• we tried to define the i18n / namespace /magic word / alias fields explicitly in extension-list, to try and allow merge message files to work. Somehow it does not work for magic words (?) and wikibase repo messages were missing.
• would have loved to try rebuild and check it is okay for wmf12 only (test2 and test wikidata); instead it apparently also rebuilded and scapped the cache for wmf11 (wikidata + wikivoyage)
• Wikibase Client (wikivoyage) was missing a magic word and that causes exception. (would be awesome if a missing magic word does not do that or fails more gracefully)
• I removed the enforcement of initialization order and reverted change to extension-list, which fixes the issue for now.

PS, would be great to have a backup copy of the cache, so we have to wait for everything to rebuild again.

Change 75997 had a related patch set uploaded by Aude:
(bug 52038) catch exception when magic word not found and warn

https://gerrit.wikimedia.org/r/75997

• Bug 52040 has been marked as a duplicate of this bug. ***

(In reply to comment #3)

We're back up

I assume via reverting the change to extension-list?

Patch in comment 5 seems to require rework.

Site is up and functioning. Nothing blocking of highest priority about this bug now

Reedy: can you do a quick write up of this under https://wikitech.wikimedia.org/wiki/Incident_documentation ? Explaining what caused it and what to do/not to do next time would be great.

@Andre my patch would only help mitigate the situation in Wikibase.

our magic words are very important but imho not so essential to take the site down vs. having wikidata links appear where they shouldn't, etc.

Not sure logging the warning is sufficient action though, and the problem with missing magic words will occur again with magic words outside of wikibase. These sort of exceptions and outages have happened before and guarantee such incident *will* happen again with some other extension or magic word in core.

Open to other suggestions :)

I mentioned this on the patch, but why exactly would magic words not be found?

Change 75997 abandoned by Aude:
(bug 52038) catch exception when magic word not found and warn

Reason:
nice try, but problems with missing magic words then appear in other places.

https://gerrit.wikimedia.org/r/75997

note, when i try catching the exception, then errors occur elsewhere.

The better option would be to have a script or some sort of check that the mediawiki codebase is fine, not throwing such exceptions, etc. before scapping.

I'm not quite sure how we'd do that ....

@Tyler it happens with localisation cache build is incomplete or something and then is scapped.