Page MenuHomePhabricator
Create Task
Maniphest T307073

Job queue backlog time increases for the `wikibase-InjectRCRecords` and `refreshLinks` jobs
Closed, DeclinedPublic
Actions

Description

As of the 23rd of April 2022, we have been seeing some irregular increases in the job queue backlog times for the wikibase-InjectRCRecords and refreshLinks jobs.

See the following Grafana boards for the respective backlog times:

wikibase-InjectRCRecords: https://grafana.wikimedia.org/d/CbmStnlGk/jobqueue-job?orgId=1&var-job=wikibase-InjectRCRecords
refreshLinks: https://grafana.wikimedia.org/d/CbmStnlGk/jobqueue-job?orgId=1&var-job=refreshLinks&var-dc=eqiad%20prometheus%2Fk8s&from=now-7d&to=now

This has also been reflected by the age of changes in the wb_changes table:
https://grafana.wikimedia.org/d/hGFN2TH7z/edit-dispatching-via-jobs?orgId=1&from=1650567902220&to=1651135174948

Related Objects
Search...

Event Timeline

ItamarWMDE created this task.Apr 28 2022, 8:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2022, 8:33 AM
ItamarWMDE renamed this task from Job queue backlog time increses for the `wikibase-InjectRCRecords` and `refreshLinks` jobs to Job queue backlog time increases for the `wikibase-InjectRCRecords` and `refreshLinks` jobs.Apr 28 2022, 8:45 AM
ItamarWMDE added projects: Wikidata, WMF-JobQueue, [DEPRECATED] wdwb-tech, wmde-wikidata-tech.
ItamarWMDE updated the task description. (Show Details)
Maintenance_bot moved this task from Inbox to Sorted Team A on the [DEPRECATED] wdwb-tech board.Apr 28 2022, 9:29 AM
Addshore subscribed.Apr 28 2022, 9:42 AM
Michael subscribed.Apr 28 2022, 12:32 PM
Lucas_Werkmeister_WMDE subscribed.Apr 28 2022, 3:36 PM
Lucas_Werkmeister_WMDE added a comment.Apr 28 2022, 3:43 PM

constraintsRunCheck is also 12 hours backlogged at the moment (link with timestamps). I suspect this is a general job queue problem that’s just not as visible for lower-frequency or lower-runtime jobs.

Aklapper mentioned this in T307314: Lag in updating Special Pages?.May 3 2022, 7:05 AM
ItamarWMDE closed this task as Declined.Jan 18 2023, 4:36 PM

Task Review / Prio Notes:

  • This task will be closed in favor of a task to investigate and define a playbook of how to handle such spikes in data access and queued jobs [TASK TBC @Michael]
Michael added a comment.Jan 23 2023, 10:42 AM
In T307073#8535638, @ItamarWMDE wrote:

Task Review / Prio Notes:

  • This task will be closed in favor of a task to investigate and define a playbook of how to handle such spikes in data access and queued jobs [TASK TBC @Michael]

I created T327641: [TECH][WIKIDATA] Create an incident playbook/flow chart for what to do when Wikidata ChangeDispatching is lagging. It can probably be improved, but I felt done is better than perfect in this case.

ItamarWMDE closed subtask T327641: [TECH][WIKIDATA] Create an incident playbook/flow chart for what to do when Wikidata ChangeDispatching is lagging as Resolved.Oct 30 2023, 4:23 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL