- Deployed mitigation (via reverting to old checkpoint as well as https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/764864, deployed in 0.3.104)
- root cause still unknown
- leaving wdqs@codfw depooled through the night to buy more time to investigate
What's left to do?
- Draft lightweight incident report (see form at https://wikitech.wikimedia.org/wiki/Incident_status; must be logged in to wikitech to see the form)
- T302340: Identify root cause of failure (currently we only have a mitigation in place)
- (after fixing root cause) re-enable traffic to codfw && revert patch for puppet metrics change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/764830/