Page MenuHomePhabricator

wmf-quibble-selenium-php72-docker jobs failing repeatedly: Undefined index: rc_logid
Closed, ResolvedPublic

Description

wmf-quibble-selenium-php72-docker is failing repeatedly, see https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/

All errors seem to have some variations of

13:06:52 mw-error.log:2021-10-20 11:02:19 bd5842a13485 wikidb: [8a9cb497451c3d495f03b830] /index.php?title=Special%3AUserLogin   PHP Notice: Undefined index: rc_logid
13:06:52 mw-error.log:#0 /workspace/src/includes/changes/RecentChange.php(490): MWExceptionHandler::handleError(integer, string, string, integer, array)
13:06:52 mw-error.log:#1 /workspace/src/extensions/Wikibase/client/includes/Changes/InjectRCRecordsJob.php(277): RecentChange->save()
13:06:52 mw-error.log:#2 /workspace/src/includes/jobqueue/JobRunner.php(373): Wikibase\Client\Changes\InjectRCRecordsJob->run()
13:06:52 mw-error.log:#3 /workspace/src/includes/jobqueue/JobRunner.php(334): JobRunner->doExecuteJob(Wikibase\Client\Changes\InjectRCRecordsJob)
13:06:52 mw-error.log:#4 /workspace/src/includes/jobqueue/JobRunner.php(247): JobRunner->executeJob(Wikibase\Client\Changes\InjectRCRecordsJob)
...

(see for instance https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/119051/console)

Event Timeline

matmarex raised the priority of this task from High to Unbreak Now!.Oct 20 2021, 1:29 PM
matmarex added a subscriber: matmarex.

This seems to be happening every time, and it's preventing patches from being merged (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/732029).

Maybe caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/731016? (I'm just guessing based on commit message and backtrace)

Error message looks very similar to T182938. Maybe it also needs a fix similar to https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/446380? I'll try to make a patch and see if that fixes it.

Change 732320 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/Wikibase@master] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/732320

(Currently waiting for Jenkins's response on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/732029 to see if the patch fixes the failure)

Nikerabbit renamed this task from wmf-quibble-selenium-php72-docker jobs failing repeatedly to wmf-quibble-selenium-php72-docker jobs failing repeatedly: Undefined index: rc_logid.Oct 20 2021, 2:39 PM

Out of curiosity, how has it been possible to get the faulty patch to pass through our testsuite? Is that due to some race condition? Since patch to mediawiki/core are tested against Wikibase a patch causing Wikibase to fail should not pass.

@Michael and I speculated that this part could be because post-request job running (which is when the error occurs) isn’t deterministic, but we’re not sure.

It’s also not clear to us why this problem didn’t show up in production – I went looking for a bit, but I’m not sure where PHP notices from jobs even get logged, I didn’t find them in logstash or in JobExecutor.log on mwlog1002.

Change 732320 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/732320

Changes seem to be merging as expected now (although there's a big backlog, not sure if it's caused by all the failures or unrelated).

Change 734589 had a related patch set uploaded (by Urbanecm; author: Bartosz Dziewoński):

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.5] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/734589

Backport to wmf.5 is in progress, in case someone wants to do last-time backport there (before replaced by wmf.6). Had to V+2 a couple of backports today thanks to this.

Change 734589 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.5] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/734589

Thank you @Urbanecm for fixing up the CI jobs!

The original patch is included in 1.38.0-wmf.6 which hence does not a backport.

Mentioned in SAL (#wikimedia-operations) [2021-10-26T13:45:05Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.5/extensions/Wikibase: 7723cf724df9ede49129443e43336e93efcd7a41: RecentChangeFactory: Add missing rc_logid value (T293885) (duration: 01m 02s)

No problem @hashar. Also deployed it, as I forgot it about it when it got merged :/.