Page MenuHomePhabricator
Create Task
Maniphest T327641

[TECH][WIKIDATA] Create an incident playbook/flow chart for what to do when Wikidata ChangeDispatching is lagging
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Every now and then, the time it takes from a change being made on Wikidata to it being injected to a subscribed wiki's Recent Changes lags more than 1 hour, while it is usually a matter of seconds. See the Change Dispatching Grafana Dashboard

Since that seems to be an occasionally recurring issue that is only partially under our control, it seems good to have some playbook that describes in detail steps to take to determine the cause of a given instance of high lag. The target audience would be somewhat experienced developers, but that are only slightly familiar with Wikimedia infrastructure (so that the steps are easy to follow in the heat of the moment).

https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Alerts#Change_Dispatching
https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Dispatching

Recent instances of this issue:

Open questions:

  • should that playbook / flow chart be public or not? (if it should be public, then the wiki tech articles linked above would seem to be a good place to put it)

Related Objects
Search...

Event Timeline

Michael created this task.Jan 23 2023, 10:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 23 2023, 10:41 AM
Michael mentioned this in T307073: Job queue backlog time increases for the `wikibase-InjectRCRecords` and `refreshLinks` jobs.Jan 23 2023, 10:42 AM
ItamarWMDE subscribed.EditedAug 2 2023, 2:14 PM

Prio Notes:

  • Does not affect end users / production
  • Affects monitoring
  • Does not affect development efforts
  • Affects onboarding efforts
  • Does not affect additional stakeholders
Lucas_Werkmeister_WMDE renamed this task from Create an icident playbook/flow chart for what to do when Wikidata ChangeDispatching is lagging to Create an incident playbook/flow chart for what to do when Wikidata ChangeDispatching is lagging.Aug 2 2023, 2:14 PM
ItamarWMDE renamed this task from Create an incident playbook/flow chart for what to do when Wikidata ChangeDispatching is lagging to [TECH][WIKIDATA] Create an incident playbook/flow chart for what to do when Wikidata ChangeDispatching is lagging.Aug 2 2023, 2:15 PM
ItamarWMDE moved this task from Incoming to [DOT] Prioritized on the wmde-wikidata-tech board.
ItamarWMDE added a project: Wikidata Dev Team.
ItamarWMDE moved this task from Incoming to UX Backlog on the Wikidata Dev Team board.Aug 2 2023, 2:18 PM
ItamarWMDE moved this task from UX Backlog to [DOT] Tech Backlog on the Wikidata Dev Team board.
noarave moved this task from [DOT] Tech Backlog to Unified DOT Backlog on the Wikidata Dev Team board.Aug 30 2023, 9:48 AM
karapayneWMDE set the point value for this task to 5.Sep 19 2023, 9:31 AM
karapayneWMDE moved this task from Unified DOT Backlog to Sprint-∞ on the Wikidata Dev Team board.
karapayneWMDE edited projects, added Wikidata Dev Team (Sprint-∞); removed Wikidata Dev Team.
ItamarWMDE moved this task from [DOT] Prioritized to Ongoing on the wmde-wikidata-tech board.Sep 20 2023, 2:48 PM
Michael moved this task from Parents/Waiting to Todo/Backlog on the Wikidata Dev Team (Sprint-∞) board.Sep 27 2023, 11:03 AM
Michael claimed this task.Oct 9 2023, 11:42 AM
Restricted Application added a project: User-Michael. · View Herald TranscriptOct 9 2023, 11:42 AM
Michael moved this task from Sprint-∞ to Wikidata.org Slice on the Wikidata Dev Team board.Oct 9 2023, 11:43 AM
Michael edited projects, added Wikidata Dev Team (Wikidata.org Slice); removed Wikidata Dev Team (Sprint-∞).
Michael added a comment.EditedOct 9 2023, 11:46 AM

Suggested subtasks (with ideally a lead time <2 days)

  1. research examples of structures/outlines of such documents and settle on one => T348443
  2. write it up on Wikitech and link it in the right places (this might be split into multiple tasks, depending on the outcome of the above) (To be created after the above is more or less done)
Michael removed Michael as the assignee of this task.Oct 9 2023, 11:55 AM
Michael moved this task from In Task Breakdown to Parent Task on the Wikidata Dev Team (Wikidata.org Slice) board.
Michael closed subtask T348443: Investigate incident runbook formats as Resolved.Oct 18 2023, 8:54 AM
Michael closed subtask T349173: Create runbook page on Wikitech and link it to Wikidata Dispatch documentation there as Resolved.Oct 20 2023, 2:54 PM
Michael updated the task description. (Show Details)Oct 20 2023, 3:48 PM
Lucas_Werkmeister_WMDE closed subtask T349174: Add notes on how to triage Change Dispatching alerts as Resolved.Oct 23 2023, 2:57 PM
Michael closed subtask T349175: Add insights from previous instances about fixing an ongoing incident as Resolved.Oct 24 2023, 3:04 PM
Michael moved this task from Parent Task to Ready for Tech Verification on the Wikidata Dev Team (Wikidata.org Slice) board.Oct 24 2023, 3:19 PM

This story is now ready to be verified. Here is the runbook: https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Runbooks/Change_dispatching/Alert

ItamarWMDE closed this task as Resolved.Oct 30 2023, 4:23 PM
ItamarWMDE claimed this task.
ItamarWMDE moved this task from Ready for Tech Verification to Done on the Wikidata Dev Team (Wikidata.org Slice) board.

Read through it and looks good to me. Thank you.

Restricted Application added a project: User-ItamarWMDE. · View Herald TranscriptOct 30 2023, 4:23 PM
ItamarWMDE closed subtask T350005: Create general advice for responding to Wikidata incidents / alerts as Resolved.Nov 7 2023, 9:35 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits