Page MenuHomePhabricator
Create Task
Maniphest T293885

wmf-quibble-selenium-php72-docker jobs failing repeatedly: Undefined index: rc_logid
Closed, ResolvedPublic
Actions

Description

wmf-quibble-selenium-php72-docker is failing repeatedly, see https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/

All errors seem to have some variations of

13:06:52 mw-error.log:2021-10-20 11:02:19 bd5842a13485 wikidb: [8a9cb497451c3d495f03b830] /index.php?title=Special%3AUserLogin   PHP Notice: Undefined index: rc_logid
13:06:52 mw-error.log:#0 /workspace/src/includes/changes/RecentChange.php(490): MWExceptionHandler::handleError(integer, string, string, integer, array)
13:06:52 mw-error.log:#1 /workspace/src/extensions/Wikibase/client/includes/Changes/InjectRCRecordsJob.php(277): RecentChange->save()
13:06:52 mw-error.log:#2 /workspace/src/includes/jobqueue/JobRunner.php(373): Wikibase\Client\Changes\InjectRCRecordsJob->run()
13:06:52 mw-error.log:#3 /workspace/src/includes/jobqueue/JobRunner.php(334): JobRunner->doExecuteJob(Wikibase\Client\Changes\InjectRCRecordsJob)
13:06:52 mw-error.log:#4 /workspace/src/includes/jobqueue/JobRunner.php(247): JobRunner->executeJob(Wikibase\Client\Changes\InjectRCRecordsJob)
...

(see for instance https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/119051/console)

Details

SubjectRepoBranchLines +/-
RecentChangeFactory: Add missing 'rc_logid' valuemediawiki/extensions/Wikibasewmf/1.38.0-wmf.5+5 -2
RecentChangeFactory: Add missing 'rc_logid' valuemediawiki/extensions/Wikibasemaster+5 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

ihurbain created this task.Oct 20 2021, 12:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2021, 12:57 PM
Reedy triaged this task as High priority.Oct 20 2021, 12:57 PM
Reedy edited projects, added ci-test-error (WMF-deployed Build Failure), Browser-Tests; removed ci-test-error.
zeljkofilipin added a parent task: T291254: Find flaky Selenium tests.Oct 20 2021, 1:19 PM
matmarex raised the priority of this task from High to Unbreak Now!.Oct 20 2021, 1:29 PM
matmarex subscribed.

This seems to be happening every time, and it's preventing patches from being merged (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/732029).

matmarex added projects: MediaWiki-extensions-WikibaseClient, Wikidata.Oct 20 2021, 1:31 PM
Restricted Application added a project: [DEPRECATED] wdwb-tech. · View Herald TranscriptOct 20 2021, 1:31 PM
matmarex added a comment.EditedOct 20 2021, 1:31 PM

Maybe caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/731016? (I'm just guessing based on commit message and backtrace)

matmarex added subscribers: Lucas_Werkmeister_WMDE, Michael.Oct 20 2021, 1:33 PM
matmarex claimed this task.Oct 20 2021, 1:39 PM

Error message looks very similar to T182938. Maybe it also needs a fix similar to https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/446380? I'll try to make a patch and see if that fixes it.

gerritbot added a comment.Oct 20 2021, 1:44 PM

Change 732320 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/Wikibase@master] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/732320

gerritbot added a project: Patch-For-Review.Oct 20 2021, 1:44 PM
matmarex added a comment.Oct 20 2021, 1:49 PM

(Currently waiting for Jenkins's response on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/732029 to see if the patch fixes the failure)

matmarex added a comment.Oct 20 2021, 2:19 PM

Seems to fix the issue, at least for https://gerrit.wikimedia.org/r/c/mediawiki/core/+/732029.

Nikerabbit renamed this task from wmf-quibble-selenium-php72-docker jobs failing repeatedly to wmf-quibble-selenium-php72-docker jobs failing repeatedly: Undefined index: rc_logid.Oct 20 2021, 2:39 PM
hashar subscribed.Oct 20 2021, 3:32 PM

Out of curiosity, how has it been possible to get the faulty patch to pass through our testsuite? Is that due to some race condition? Since patch to mediawiki/core are tested against Wikibase a patch causing Wikibase to fail should not pass.

Lucas_Werkmeister_WMDE added a comment.Oct 20 2021, 3:37 PM

@Michael and I speculated that this part could be because post-request job running (which is when the error occurs) isn’t deterministic, but we’re not sure.

It’s also not clear to us why this problem didn’t show up in production – I went looking for a bit, but I’m not sure where PHP notices from jobs even get logged, I didn’t find them in logstash or in JobExecutor.log on mwlog1002.

gerritbot added a comment.Oct 20 2021, 4:55 PM

Change 732320 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/732320

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.6; 2021-10-26).Oct 20 2021, 5:00 PM
matmarex closed this task as Resolved.Oct 20 2021, 5:00 PM

Changes seem to be merging as expected now (although there's a big backlog, not sure if it's caused by all the failures or unrelated).

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2021, 5:10 PM
gerritbot added a comment.Oct 26 2021, 11:46 AM

Change 734589 had a related patch set uploaded (by Urbanecm; author: Bartosz Dziewoński):

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.5] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/734589

gerritbot added a project: Patch-For-Review.Oct 26 2021, 11:46 AM
Urbanecm subscribed.Oct 26 2021, 12:02 PM

Backport to wmf.5 is in progress, in case someone wants to do last-time backport there (before replaced by wmf.6). Had to V+2 a couple of backports today thanks to this.

gerritbot added a comment.Oct 26 2021, 12:29 PM

Change 734589 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@wmf/1.38.0-wmf.5] RecentChangeFactory: Add missing 'rc_logid' value

https://gerrit.wikimedia.org/r/734589

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.5; 2021-10-19); removed MW-1.38-notes (1.38.0-wmf.6; 2021-10-26).Oct 26 2021, 1:00 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 26 2021, 1:10 PM
hashar added a comment.Oct 26 2021, 1:32 PM

Thank you @Urbanecm for fixing up the CI jobs!

The original patch is included in 1.38.0-wmf.6 which hence does not a backport.

Stashbot added a comment.Oct 26 2021, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2021-10-26T13:45:05Z] <urbanecm@deploy1002> Synchronized php-1.38.0-wmf.5/extensions/Wikibase: rEWBA7723cf724df9: RecentChangeFactory: Add missing rc_logid value (T293885) (duration: 01m 02s)

Urbanecm added a comment.Oct 26 2021, 1:50 PM

No problem @hashar. Also deployed it, as I forgot it about it when it got merged :/.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits