Maniphest T207809

PHP error "CdnPurgeJob never inserted."
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

Request ID: W8@-zwpAEMUAABz2jYkAAADT

message

PHP Notice: JobQueueGroup::__destruct: 1 buffered job(s) of type(s) CdnPurgeJob never inserted.

#0 /srv/mediawiki/php-1.32.0-wmf.26/includes/jobqueue/JobQueueGroup.php(472): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 [internal function]: JobQueueGroup->__destruct()
#2 {main}

Impact

Loss of jobs we thought were queued but were not, and will likely never execute.

Notes

Comes from both API and index.php requests that did not results in a fatal exception.

Started with 1.32-wmf.26. Happened 19,000 times since last week.

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
Create JobQueueEnqueueUpdate class to call JobQueueGroup::pushLazyJobs()	Paladox	mediawiki/core	REL1_32	+68 -26
Remove obsolete pushLazyJobs() deferred update call	Aaron Schulz	mediawiki/extensions/EventBus	master	+0 -2
Create JobQueueEnqueueUpdate class to call JobQueueGroup::pushLazyJobs()	Aaron Schulz	mediawiki/core	master	+68 -26

Customize query in gerrit

Related Objects

Mentioned Here: rMW9fb67b9a41da: Make merged MergeableUpdate items always go to the end of the queue

Event Timeline

Krinkle created this task.Oct 24 2018, 12:48 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2018, 12:48 AM

Krinkle moved this task from General to HTTP Cache on the MediaWiki-libs-BagOStuff board.Oct 24 2018, 12:48 AM

I don't see any changes in the JobQueue itself in wmf.26 that could make it start happening.

Also, there's no major change in the job processing rates, so this doesn't affect all the job, but a minority of them. Will look more.

• Pchelolo added projects: Services (watching), Platform Team Legacy (Watching / External).Oct 24 2018, 3:15 PM

• Pchelolo edited projects, added: Platform Team Legacy; removed: Platform Team Legacy (Watching / External).

Krinkle moved this task from Untriaged to Jul2019/1.34.wmf.13+ on the Wikimedia-production-error board.Oct 24 2018, 5:23 PM

Change 469491 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Create JobQueueEnqueueUpdate class to handle calling JobQueueGroup::pushLazyJobs()

https://gerrit.wikimedia.org/r/469491

gerritbot added a project: Patch-For-Review.Oct 24 2018, 7:29 PM

Change 469494 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/extensions/EventBus@master] Remove obsolete pushLazyJobs() deferred update call

https://gerrit.wikimedia.org/r/469494

• CCicalese_WMF edited projects, added: Platform Team Legacy (Watching / External); removed: Platform Team Legacy.Oct 26 2018, 5:52 PM

@aaron The fix LGTM, but do we know why this started happening? I'd be nice to know what commit or task prompted it so that we can learn why it wasn't prevented by our tests and/or Jenkins.

In theory, this kind of warning can be triggered in tests and would be captured by Jenkins in a way that fail the build. That definitely worked at some point.

Krinkle assigned this task to aaron.Oct 28 2018, 8:45 PM

Krinkle triaged this task as Medium priority.

Krinkle added a project: Performance-Team.

In T207809#4701489, @Krinkle wrote:

@aaron The fix LGTM, but do we know why this started happening? I'd be nice to know what commit or task prompted it so that we can learn why it wasn't prevented by our tests and/or Jenkins.

In theory, this kind of warning can be triggered in tests and would be captured by Jenkins in a way that fail the build. That definitely worked at some point.

I mentioned on IRC that 9fb67b9a41da8714743789fa0365a6052cc86b97 moved the cdn purges after the lazy job insertion.

Change 470293 had a related patch set uploaded (by Paladox; owner: Aaron Schulz):
[mediawiki/core@REL1_32] Create JobQueueEnqueueUpdate class to call JobQueueGroup::pushLazyJobs()

https://gerrit.wikimedia.org/r/470293

Change 469491 merged by jenkins-bot:
[mediawiki/core@master] Create JobQueueEnqueueUpdate class to call JobQueueGroup::pushLazyJobs()

https://gerrit.wikimedia.org/r/469491

ReleaseTaggerBot added a project: MW-1.33-notes (1.33.0-wmf.2; 2018-10-30).Oct 29 2018, 12:01 AM

Change 469494 merged by Krinkle:
[mediawiki/extensions/EventBus@master] Remove obsolete pushLazyJobs() deferred update call

https://gerrit.wikimedia.org/r/469494

aaron closed this task as Resolved.Oct 29 2018, 9:18 PM

aaron moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.

Change 470293 merged by jenkins-bot:
[mediawiki/core@REL1_32] Create JobQueueEnqueueUpdate class to call JobQueueGroup::pushLazyJobs()

https://gerrit.wikimedia.org/r/470293

ReleaseTaggerBot added a project: MW-1.32-notes.Dec 18 2018, 4:01 AM

Krinkle moved this task from Jul2019/1.34.wmf.13+ to Resolved on the Wikimedia-production-error board.Mar 21 2019, 8:02 PM

• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM

Maintenance_bot removed a project: Patch-For-Review.Aug 28 2019, 11:41 PM

PHP error "CdnPurgeJob never inserted."Closed, ResolvedPublicPRODUCTION ERRORActions