Maniphest T397719

Investigate WDQS & WCQS codfw 24h+ lag
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	bking
	Jun 24 2025, 1:33 PM

Tags

Referenced Files

None

Subscribers

Description

Beginning at 1100 UTC on 2025-06-23, the rdf streaming updater for WDQS and WCQS failed. Lag continued to grow until we intervened around 1745 UTC on 2025-06-24 .

Creating this ticket to:

Identify root cause (it was the k8s upgrade in T397148 )
Create runbook for this failure scenario
Identify and implement any changes that would make this less likely to happen and/or easier to fix in the future (such as updating the Flink App dashboard)

Details

Other Assignee: RKemper

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	wcqs: restore from checkpoint	operations/deployment-charts	master	+2 -2
	wcqs: restore from checkpoint	operations/deployment-charts	master	+1 -1
	rdf-streaming-updater: point to last valid checkpoint for restore	operations/deployment-charts	master	+2 -2

Customize query in gerrit

Related Objects

Mentioned In: T404605: Plan flink-app recovery process (upcoming wikikube eqiad upgrade)
Mentioned Here: T397148: Update wikikube codfw to kubernetes 1.31

Event Timeline

bking created this task.Jun 24 2025, 1:33 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 24 2025, 1:33 PM

Maintenance_bot added a project: Wikidata.Jun 24 2025, 2:29 PM

Change #1163434 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] rdf-streaming-updater: point to last valid checkpoint for restore

https://gerrit.wikimedia.org/r/1163434

gerritbot added a project: Patch-For-Review.Jun 24 2025, 5:38 PM

Change #1163434 merged by Bking:

[operations/deployment-charts@master] rdf-streaming-updater: point to last valid checkpoint for restore

https://gerrit.wikimedia.org/r/1163434

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2025, 6:30 PM

RKemper renamed this task from Investigate WDQS codfw 24h+ lag to Investigate WDQS & WCQS codfw 24h+ lag.Jun 24 2025, 9:26 PM

Change #1163475 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/deployment-charts@master] wcqs: restore from checkpoint

https://gerrit.wikimedia.org/r/1163475

gerritbot added a project: Patch-For-Review.Jun 24 2025, 9:26 PM

Change #1163475 merged by jenkins-bot:

[operations/deployment-charts@master] wcqs: restore from checkpoint

https://gerrit.wikimedia.org/r/1163475

Change #1163478 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):

[operations/deployment-charts@master] wcqs: restore from checkpoint

https://gerrit.wikimedia.org/r/1163478

Change #1163478 merged by jenkins-bot:

[operations/deployment-charts@master] wcqs: restore from checkpoint

https://gerrit.wikimedia.org/r/1163478

Maintenance_bot removed a project: Patch-For-Review.Jun 24 2025, 10:30 PM

bking changed the task status from Open to In Progress.Jun 25 2025, 3:38 PM

bking claimed this task.

bking triaged this task as High priority.

bking updated Other Assignee, added: RKemper.

bking updated the task description. (Show Details)

bking updated the task description. (Show Details)Jun 27 2025, 9:29 PM

pfischer moved this task from Incoming to Operations/SRE on the Wikidata-Query-Service board.Jun 30 2025, 2:42 PM

Gehel edited projects, added Data-Platform-SRE (2025.09.05 - 2025.09.26); removed Data-Platform-SRE.Sep 9 2025, 2:17 PM

Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.

The AC for this ticket have been completed, so I am closing it out. Feel free to reopen if I missed something.

Gehel added a project: Essential-Work.Sep 11 2025, 12:11 PM

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.05 - 2025.09.26) board.Sep 12 2025, 2:34 PM

bking mentioned this in T404605: Plan flink-app recovery process (upcoming wikikube eqiad upgrade).Sep 15 2025, 3:59 PM