This is pending DBA review. Per the Datacenter-Switchover guidelines we were given via wikitech-l, we're adviced not to deploy any extension or service that adds tables or does schecha changes unless DBA says it is okay to do so. The translate extension needs some tables to operate. Granted, foundationwiki is not that large but I'm not sure hence asking before is a good practice to avoid breaking stuff. Thanks.

@MarcoAurelio thanks for asking!
The tables can be created as those will be empty, it is not a dangerous operation, whoever, the extension shouldn't be enabled until 10th Oct, once we are back in eqiad. The failover back to eqiad is scheduled for 10th Oct at 14:00 UTC.

Thanks again!

(See https://phabricator.wikimedia.org/T46871 for previous historical discussions.)

Change 735396 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] foundationwiki: Enable Translate extension

https://gerrit.wikimedia.org/r/735396

Change 735396 merged by jenkins-bot:

[operations/mediawiki-config@master] foundationwiki: Enable Translate extension

https://gerrit.wikimedia.org/r/735396

Mentioned in SAL (#wikimedia-operations) [2021-10-28T16:24:10Z] <urbanecm> foundationwiki: Create DB tables for translate extension (T205349)

Mentioned in SAL (#wikimedia-operations) [2021-10-28T16:31:03Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 78783a7624ddf4e5bfac1e255f466d3c3e36016d: foundationwiki: Enable Translate extension (T205349) (duration: 01m 25s)

Change 735415 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Revert \"foundationwiki: Enable Translate extension\"

https://gerrit.wikimedia.org/r/735415

Change 735415 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert \"foundationwiki: Enable Translate extension\"

https://gerrit.wikimedia.org/r/735415

Mentioned in SAL (#wikimedia-operations) [2021-10-28T16:44:24Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: REVERT: 78783a7624ddf4e5bfac1e255f466d3c3e36016d: foundationwiki: Enable Translate extension (T205349) (duration: 01m 04s)

So, I tried to enable Translate at foundationwiki earlier today. Unfortunately, it did not work:

2021-10-28 16:41:17 [3b5a32c4-953f-4d01-bd68-cb2605a4c1b4] mwdebug1002 foundationwiki 1.38.0-wmf.5 exception ERROR: [3b5a32c4-953f-4d01-bd68-cb2605a4c1b4] /wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&limit=50&days=7&urlversion=2   InvalidArgumentException: Missing field: page_lang {"exception_url":"/wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&limit=50&days=7&urlversion=2","reqId":"3b5a32c4-953f-4d01-bd68-cb2605a4c1b4","caught_by":"entrypoint"}
[Exception InvalidArgumentException] (/srv/mediawiki/php-1.38.0-wmf.5/includes/cache/LinkCache.php:320) Missing field: page_lang
  #0 /srv/mediawiki/php-1.38.0-wmf.5/includes/cache/LinkCache.php(496): LinkCache->addGoodLinkObjFromRow(TitleValue, stdClass)
  #1 /srv/mediawiki/php-1.38.0-wmf.5/includes/cache/LinkCache.php(420): LinkCache->getGoodLinkRow(integer, string, array, integer)
  #2 /srv/mediawiki/php-1.38.0-wmf.5/includes/Title.php(2881): LinkCache->addLinkObj(Title)
  #3 /srv/mediawiki/php-1.38.0-wmf.5/extensions/Translate/tag/TranslatablePage.php(712): Title->getArticleID()
  #4 /srv/mediawiki/php-1.38.0-wmf.5/extensions/Translate/tag/PageTranslationHooks.php(164): TranslatablePage::isSourcePage(Title)
  #5 /srv/mediawiki/php-1.38.0-wmf.5/includes/HookContainer/HookContainer.php(338): PageTranslationHooks::fetchTranslatableTemplateAndTitle(Title, Title, boolean, NULL)
  #6 /srv/mediawiki/php-1.38.0-wmf.5/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
  #7 /srv/mediawiki/php-1.38.0-wmf.5/includes/HookContainer/HookRunner.php(973): MediaWiki\HookContainer\HookContainer->run(string, array)
  #8 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/Parser.php(3628): MediaWiki\HookContainer\HookRunner->onBeforeParserFetchTemplateRevisionRecord(Title, Title, boolean, NULL)
  #9 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/Parser.php(3564): Parser->statelessFetchTemplate(Title, Parser)
  #10 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/Parser.php(3458): Parser->fetchTemplateAndTitle(Title)
  #11 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/Parser.php(3197): Parser->getTemplateDom(Title)
  #12 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/PPFrame_Hash.php(276): Parser->braceSubstitution(array, PPFrame_Hash)
  #13 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/Parser.php(2917): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
  #14 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/Parser.php(1584): Parser->replaceVariables(string)
  #15 /srv/mediawiki/php-1.38.0-wmf.5/includes/parser/Parser.php(659): Parser->internalParse(string)
  #16 /srv/mediawiki/php-1.38.0-wmf.5/includes/cache/MessageCache.php(1312): Parser->parse(string, Title, ParserOptions, boolean)
  #17 /srv/mediawiki/php-1.38.0-wmf.5/includes/specials/SpecialRecentChanges.php(650): MessageCache->parse(string, Title, boolean, boolean, LanguageEn)
  #18 /srv/mediawiki/php-1.38.0-wmf.5/includes/specials/SpecialRecentChanges.php(532): SpecialRecentChanges->setTopText(FormOptions)
  #19 /srv/mediawiki/php-1.38.0-wmf.5/includes/specialpage/ChangesListSpecialPage.php(1652): SpecialRecentChanges->doHeader(FormOptions, integer)
  #20 /srv/mediawiki/php-1.38.0-wmf.5/includes/specialpage/ChangesListSpecialPage.php(1663): ChangesListSpecialPage->webOutputHeader(integer, FormOptions)
  #21 /srv/mediawiki/php-1.38.0-wmf.5/includes/specialpage/ChangesListSpecialPage.php(660): ChangesListSpecialPage->webOutput(Wikimedia\Rdbms\MysqliResultWrapper, FormOptions)
  #22 /srv/mediawiki/php-1.38.0-wmf.5/includes/specials/SpecialRecentChanges.php(199): ChangesListSpecialPage->execute(NULL)
  #23 /srv/mediawiki/php-1.38.0-wmf.5/includes/specialpage/SpecialPage.php(647): SpecialRecentChanges->execute(NULL)
  #24 /srv/mediawiki/php-1.38.0-wmf.5/includes/specialpage/SpecialPageFactory.php(1377): SpecialPage->run(NULL)
  #25 /srv/mediawiki/php-1.38.0-wmf.5/includes/MediaWiki.php(314): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
  #26 /srv/mediawiki/php-1.38.0-wmf.5/includes/MediaWiki.php(925): MediaWiki->performRequest()
  #27 /srv/mediawiki/php-1.38.0-wmf.5/includes/MediaWiki.php(559): MediaWiki->main()
  #28 /srv/mediawiki/php-1.38.0-wmf.5/index.php(53): MediaWiki->run()
  #29 /srv/mediawiki/php-1.38.0-wmf.5/index.php(46): wfIndexMain()
  #30 /srv/mediawiki/w/index.php(3): require(string)
  #31 {main}

I tried resetting state of LinkCache service (via MediaWikiServices::resetGlobalInstance() as well as restarting PHP), which did indeed clear LinkCache, but it didn't make Special:RecentChanges to work. I'm not sure why this happened. @Nikerabbit, could you lend me a hand on this one please?

I'd guess immediate cause is that wgPageLanguageUseDB is not set to true. It defaults to false. But this sounds like a bug in MediaWiki core. Tagging Platform Engineering.

In T205349#7467540, @Nikerabbit wrote:

I'd guess immediate cause is that wgPageLanguageUseDB is not set to true. It defaults to false. But this sounds like a bug in MediaWiki core. Tagging Platform Engineering.

That was my first guess too. However, shell.php at production tells me that variable got enabled automatically after I enabled Translate. I just checked again by livehacking the debug server. As you can see below, setting $wmgUseTranslate to true automatically caused $wgPageLanguageUseDB to be true as well (as our config tells it to).

[urbanecm@mwdebug1001 /srv/mediawiki]$ sudo -u mwdeploy vim wmf-config/InitialiseSettings.php # enable Translate locally by setting wmgUseTranslate = true
[urbanecm@mwdebug1001 /srv/mediawiki]$ mwscript shell.php foundationwiki
Psy Shell v0.10.8 (PHP 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 — cli) by Justin Hileman
>>> $wgPageLanguageUseDB
=> true
>>>
[urbanecm@mwdebug1001 /srv/mediawiki]$ scap pull # clear local changes => disable Translate again
11:00:16 Copying from deploy1002.eqiad.wmnet to mwdebug1001.eqiad.wmnet
11:00:16 Started rsync common
11:00:19 Finished rsync common (duration: 00m 03s)
11:00:19 Started scap-cdb-rebuild
11:00:20 Finished scap-cdb-rebuild (duration: 00m 00s)
11:00:20 Checking if php-fpm restart needed
[urbanecm@mwdebug1001 /srv/mediawiki]$ mwscript shell.php foundationwiki
Psy Shell v0.10.8 (PHP 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 — cli) by Justin Hileman
>>> $wgPageLanguageUseDB
=> false
>>> ^D
Exit:  Ctrl+D
[urbanecm@mwdebug1001 /srv/mediawiki]$

So, unless this means "variable is sometimes false and sometimes true" (I think that'd affect other Translate wikis, too?), this looks to be foundationwiki-specific, and not related to $wgPageLanguageUseDB.

So...I spent some time debugging this today, and I figured out that if I run: \MediaWiki\MediaWikiServices::getInstance()->getLinkCache()->invalidateTitle(...) with certain titles [1], RecentChanges load just fine, even with translate.

This means something's not invalidating persistent caching in LinkCache when it should be.

[1] Titles to use were identified by adding var_dump($row) to LinkCache.php:496 and invalidating whichever titles it yielded.

Change 735720 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/core@master] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/735720

Tried to fix the missing invalidation. No guarantees on patch's correctness, but it appears to work when I tested it at mwdebug1001.

In T205349#7469808, @Urbanecm_WMF wrote:

Tried to fix the missing invalidation. No guarantees on patch's correctness, but it appears to work when I tested it at mwdebug1001.

jenkins didn't like it though :(. @daniel, your advice will be appreciated.

I'm not sure I fully understand the issue... Is it that LinkCache throws if it finds cached rows that are missing page_lang? And this is triggered when enabling $wgPageLanguageUseDB on a wiki that previsouly didn't have it set? I'd expect such a change to invalidate caches... But in any case, Linkcache shouldn't fail hard, it should just warn in such a case. I'll have a look.

That's the correct interpretation of the issue I'm trying to solve @daniel.
FTR, I'm trying to make the change at foundationwiki, which was recently
connected to CentralAuth (that's an extraordinary change, and I don't think
anyone tried to do it since SUL finalisation happened). I don't think the
CA bit plays a significant role here, but just in case it does, I'm noting
it. I'm also happy to try to enable that option in some other wiki to see
if the same thing happens. Let me know what your investigation shows.

In T205349#7471067, @Urbanecm_WMF wrote:

I'm also happy to try to enable that option in some other wiki to see
if the same thing happens. Let me know what your investigation shows.

Trying to use rows from the cache that were written before $wgPageLanguageUseDB was enabled, that will clearly cause this issue. Your patch looks like it will fix this issue. Tests are just failing for the edge case of something trying to use LinkCache with an invalid title. I commented on the patch.

Change 735975 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[mediawiki/core@master] Gracefully handle old/broken rows found in cache

https://gerrit.wikimedia.org/r/735975

Change 735720 merged by jenkins-bot:

[mediawiki/core@master] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/735720

Change 736065 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/core@wmf/1.38.0-wmf.7] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/736065

Change 736246 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/core@wmf/1.38.0-wmf.6] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/736246

Change 736248 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Revert \"Revert \"foundationwiki: Enable Translate extension\"\"

https://gerrit.wikimedia.org/r/736248

Change 736246 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.6] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/736246

Change 736065 merged by jenkins-bot:

[mediawiki/core@wmf/1.38.0-wmf.7] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/736065

Change 736248 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert \"Revert \"foundationwiki: Enable Translate extension\"\"

https://gerrit.wikimedia.org/r/736248

Mentioned in SAL (#wikimedia-operations) [2021-11-02T17:22:47Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.6/includes/cache/LinkCache.php: rMW1e78aeabd682: LinkCache: Try invalidating cache before throwing (T205349) (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2021-11-02T17:24:13Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: e3227703a662ecda744bb159f39b128ed289c76d: Revert "Revert "foundationwiki: Enable Translate extension"" (T205349) (duration: 00m 55s)

Change 768069 had a related patch set uploaded (by RhinosF1; author: Urbanecm):

[mediawiki/core@REL1_37] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/768069

Change 768069 merged by jenkins-bot:

[mediawiki/core@REL1_37] LinkCache: Try invalidating cache before throwing

https://gerrit.wikimedia.org/r/768069