codfw wdqs updater failures
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RKemper
	Feb 22 2022, 7:56 PM

Description

Current status

Current status:

Deployed mitigation (via reverting to old checkpoint as well as https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/764864, deployed in 0.3.104)
root cause still unknown
- leaving wdqs@codfw depooled through the night to buy more time to investigate

What's left to do?

Draft lightweight incident report (see form at https://wikitech.wikimedia.org/wiki/Incident_status; must be logged in to wikitech to see the form)
T302340: Identify root cause of failure (currently we only have a mitigation in place)
- (after fixing root cause) re-enable traffic to codfw && revert patch for puppet metrics change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/764830/

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Gehel	T301147 The WDQS streaming updater went unstable for several hours (2022-02-06T23:00:00 - 2022-02-07T06:20:00)
		Resolved		RKemper	T302340 codfw wdqs updater failures

Event Timeline

RKemper created this task.Feb 22 2022, 7:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 22 2022, 7:56 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-22T19:58:08Z] <ryankemper> T302340 scap deploy -v --environment wcqs 'Deploy 0.3.104 to WCQS'

RKemper updated the task description. (Show Details)Feb 22 2022, 7:58 PM

RKemper added a subscriber: Tarrow.

Mentioned in SAL (#wikimedia-operations) [2022-02-22T20:06:01Z] <ryankemper> T302340 [WCQS] Forgot to fetch & rebase deploy1002:/srv/deployment/wdqs/wdqs before deploy, so 0.3.104 did not actually deploy (still on 0.3.103). Re-rolling deploy...