Page MenuHomePhabricator

Create WDQS meltdown runbook
Open, MediumPublic

Description

In order to quickly respond to WDQS outages, we need a comprehensive guide on what actions to take. Some of these actions involve PII and security-sensitive discussions, so this will be a private Google doc.

Creating this ticket to:

  • Read Slack, Wikitech, IRC etc and add non-sensitive info to existing, public-facing runbooks.
  • Add sensitive information to the private document
  • Link existing documents in the private doc (such as SRE docs on DDoS/requestctl usage)
  • Review with stakeholders (WMDE, Wikidata Platform Team, DPE SRE)

Event Timeline

bking triaged this task as Medium priority.
bking removed bking as the assignee of this task.May 12 2026, 2:16 PM
bking updated the task description. (Show Details)
Gehel subscribed.

We expect Wikidata Platform Team to do the work, we'll help as needed.

I categorized all existing runbooks I've found during ongoing WDQS docs work on Wikitech: https://w.wiki/UBtP. The WDQS landing page on Wikitech has a bunch of information that would likely be relevant for runbooks, but also will need to be completely rewritten and updated as the team migrates to the new backend (T434107).

Also this page exists:
https://www.wikidata.org/wiki/Wikidata:SPARQL_query_service/WDQS_backend_update/Blazegraph_failure_playbook