Page MenuHomePhabricator

jenkis CI wikibase-repo-docker failing with new quibble version
Closed, ResolvedPublic

Description

We are seeing build failures on some repositories, in particular wikibase where the following jobs fail with the same error

10:33:23 Purging caches...done.
10:33:38 LogicException from line 408 of /workspace/src/includes/cache/MessageCache.php: Process cache for 'en' should be set by now.
10:33:38 #0 /workspace/src/includes/cache/MessageCache.php(1112): MessageCache->load('en')
10:33:38 #1 /workspace/src/includes/cache/MessageCache.php(1041): MessageCache->getMsgFromNamespace('Seconds-abbrev', 'en')
10:33:38 #2 /workspace/src/includes/cache/MessageCache.php(1011): MessageCache->getMessageForLang(Object(LanguageEn), 'seconds-abbrev', true, Array)
10:33:38 #3 /workspace/src/includes/cache/MessageCache.php(953): MessageCache->getMessageFromFallbackChain(Object(LanguageEn), 'seconds-abbrev', true)
10:33:38 #4 /workspace/src/includes/language/Message.php(1491): MessageCache->get('seconds-abbrev', true, Object(LanguageEn))
10:33:38 #5 /workspace/src/includes/language/Message.php(968): Message->fetchMessage()
10:33:38 #6 /workspace/src/includes/language/Message.php(1071): Message->format('text')
10:33:38 #7 /workspace/src/includes/language/Language.php(4617): Message->text()
10:33:38 #8 /workspace/src/maintenance/update.php(217): Language->formatTimePeriod(1.5317060947418)
10:33:38 #9 /workspace/src/maintenance/doMaintenance.php(114): UpdateMediaWiki->execute()
10:33:38 #10 /workspace/src/maintenance/update.php(264): require_once('/workspace/src/...')
10:33:38 #11 {main}
10:33:38 INFO:quibble.commands:<<< Finish: Install MediaWiki, db=<MySQL /workspace/db/quibble-mysql-mp_s89nw/socket> vendor=True, in 18.032 s

It seems to have started with the updated Quibble image 1.4.4 merged here https://gerrit.wikimedia.org/r/c/integration/config/+/771640/

image.png (219×1 px, 17 KB)

(left failure - right last successful build)

https://integration.wikimedia.org/ci/job/wikibase-repo-docker/27074/consoleText

Event Timeline

kostajh subscribed.

Seems related to T300340: Use Memcached with Quibble. No idea why it's happening for just these jobs, though.

Change 771724 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] Revert "jjb: switch jobs to Quibble 1.4.4"

https://gerrit.wikimedia.org/r/771724

Mentioned in SAL (#wikimedia-releng) [2022-03-18T13:22:34Z] <hashar> Rolling back Quibble jobs from 1.4.4 T304147

Change 771724 merged by jenkins-bot:

[integration/config@master] Revert "jjb: switch jobs to Quibble 1.4.4"

https://gerrit.wikimedia.org/r/771724

Looks like CI is working again with the Quibble revert.

@kostajh has found the issue, the failing jobs are the one that do not start supervisord and thus do not have memcached server running. Quibble 1.4.4 configure the caches to use memcached whenever the PHP extension is loaded but since there is no memcached server the cache is empty. maintenance/update.php then complains en is not in the LocalisationCache and the build fails.

We will need to change the images entry point to invoke supervisord. Aka change them to use ENTRYPOINT ["quibble-with-apache"] and probably rename that script while at it since it invokes supervisord which now spawns Apache AND memcached.

Change 773635 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] dockerfiles: Update Quibble to 1.4.4-s1

https://gerrit.wikimedia.org/r/773635

Change 773636 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] jjb: Use quibble-with-supervisord entrypoint

https://gerrit.wikimedia.org/r/773636

Change 773637 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] dockerfiles: Update quibble-buster-php72-bundle to 1.4.4-s2

https://gerrit.wikimedia.org/r/773637

Change 773638 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/config@master] jjb: Update mwselenium jobs to use quibble 1.4.4-s2

https://gerrit.wikimedia.org/r/773638

Change 773635 merged by jenkins-bot:

[integration/config@master] dockerfiles: Update Quibble to 1.4.4-s1

https://gerrit.wikimedia.org/r/773635

Change 773637 merged by jenkins-bot:

[integration/config@master] dockerfiles: Update quibble-buster-php72-bundle to 1.4.4-s2

https://gerrit.wikimedia.org/r/773637

Change 773638 merged by jenkins-bot:

[integration/config@master] jjb: Update mwselenium jobs to use quibble 1.4.4-s2

https://gerrit.wikimedia.org/r/773638

Change 773762 had a related patch set uploaded (by Hashar; author: Kosta Harlan):

[integration/config@master] jjb: Update mwselenium jobs to use quibble 1.4.4-s3

https://gerrit.wikimedia.org/r/773762

Change 773762 merged by jenkins-bot:

[integration/config@master] jjb: Update mwselenium jobs to use quibble 1.4.4-s3

https://gerrit.wikimedia.org/r/773762

Change 773636 abandoned by Kosta Harlan:

[integration/config@master] jjb: Use quibble-with-supervisord entrypoint

Reason:

https://gerrit.wikimedia.org/r/773636

@hashar are there any other jobs to update? Is every Quibble job now using supervisord entrypoint?

Change 790370 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update api-testing jobs to Quibble 1.4.5 image

https://gerrit.wikimedia.org/r/790370

Change 790370 merged by jenkins-bot:

[integration/config@master] jjb: update api-testing jobs to Quibble 1.4.5 image

https://gerrit.wikimedia.org/r/790370

Change 790377 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update Phan jobs to Quibble 1.4.5 image

https://gerrit.wikimedia.org/r/790377

Change 790377 merged by jenkins-bot:

[integration/config@master] jjb: update Phan jobs to Quibble 1.4.5 image

https://gerrit.wikimedia.org/r/790377

Change 790621 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update wikibase jobs to Quibble 1.4.5

https://gerrit.wikimedia.org/r/790621

hashar claimed this task.

I have tested https://gerrit.wikimedia.org/r/c/integration/config/+/790621 against https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/767776 the jobs work with Quibble 1.4.5 + Memcached now that supervisord is being used ;)

Change 790621 merged by jenkins-bot:

[integration/config@master] jjb: update wikibase jobs to Quibble 1.4.5

https://gerrit.wikimedia.org/r/790621