wdqs2*** lagged for more than one day
Closed, ResolvedPublic5 Estimated Story Points
Actions

Assigned To

Authored By

	Bugreporter
	May 7 2023, 3:27 PM

Description

Incident report: https://wikitech.wikimedia.org/wiki/Incidents/2023-05-05_wdqs_not_updating_in_codfw

See https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m

Events:

2023-05-04T10:00: the streaming updater flink job stopped to function in codfw for both WDQS and WCQS
- user impact starts: stale results are seen when using WDQS from a region that hits CODFW
- reason is likely https://issues.apache.org/jira/browse/FLINK-22597
2023-05-05T16:22: the problem is reported by Bovlb via https://www.wikidata.org/wiki/Wikidata:Report_a_technical_problem/WDQS_and_Search
2023-05-05T19:00: the flink jobmanager container is manually restarted and the jobs resume but the WDQS one is very unstable (k8s is heavily throttling cpu usage and taskmanager mem usage grows quickly)
- (assumptions) because the job was backfilling 1day of data it required more resources than usual, though this is not the first time that a backfill happens (e.g. k8s cluster upgrades went well)
- (assumptions) because the job was resource constrained rocksdb resource compaction did not happen in a timely manner
2023-05-05T21:00: the job fails again
2023-05-06T10:00: the job resumes (unknown reasons)
2023-05-06T19:00: the job fails again
- Seeing jvm OutOfMemoryError
- The checkpoint it tries to recover from is abnormally large (6G instead of 1.5G usually), assumption is that rocksdb compaction did not occur properly
2023-05-07T17:27: this ticket is created as UBN
2023-05-08T16:00: wdqs in CODFW is depooled
- user impact ends
2023-05-09T14:00: increasing taskmanager memory from 1.9G to 2.5G did not help
2023-05-09T14:00: starting the job from yarn using across 12 containers with 5G did help
- the job recovered and started to produce reasonable checkpoint sizes
2023-05-10T00:00: lag is back to normal on all wdqs servers
2023-05-10T10:30: the job is resumed from k8s@codfw

Remaining actions:

Repool WDQS in codfw

AC:

WDQS codfw cluster is pooled and running with up to date data
Incident report is created
issue is communicated on wikidata mailing list

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	rdf-streaming-updater: Increase task manager memory alloc	operations/deployment-charts	master	+5 -5
	flink-session-cluster: enable rocksdb metrics and increase jvm heap	operations/deployment-charts	master	+6 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	bking	T336134 wdqs2*** lagged for more than one day
Open	None	T336574 Review alerting around Wikidata Query Service update pipeline
Open	None	T270614 Automatically depool wdqs servers that are "lagged"
Resolved	hoo	T238751 Only generate maxlag from pooled query service servers.
Open	None	T239392 Applications and scripts need to be able to understand the pooled status of servers in our load balancers.
Resolved	hoo	T315423 Revive and merge patch to update maxlag calculation
Resolved	bking	T336577 Update WDQS Runbook following update lag incident
Resolved	bking	T337801 WDQS: Document procedure for switching between Kubernetes and Yarn Streaming Updater

Event Timeline

Bugreporter created this task.May 7 2023, 3:27 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2023, 3:27 PM

Bugreporter triaged this task as Unbreak Now! priority.May 7 2023, 3:27 PM

Gehel moved this task from Incoming to Current work on the Wikidata-Query-Service board.May 8 2023, 1:32 PM

Gehel added a project: Discovery-Search (Current work).

Gehel updated the task description. (Show Details)May 8 2023, 3:39 PM

Gehel set the point value for this task to 5.

Gehel moved this task from Incoming to Ready for Dev -- SRE/Ops on the Discovery-Search (Current work) board.

bking subscribed.May 8 2023, 3:41 PM

bking claimed this task.May 8 2023, 7:11 PM

Icinga downtime and Alertmanager silence (ID=d5700f27-224a-464e-878a-4a5780b823f5) set by bking@cumin1001 for 4:00:00 on 19 host(s) and their services with reason: rebooting to help with lag

wdqs[2004-2022].codfw.wmnet

Icinga downtime and Alertmanager silence (ID=8eda2109-041e-49e8-9896-a24bb9b2645e) set by bking@cumin1001 for 4:00:00 on 1 host(s) and their services with reason: rebooting to help with lag

wdqs1004.eqiad.wmnet

Icinga downtime and Alertmanager silence (ID=40727833-628f-4d74-8179-b284df022778) set by bking@cumin1001 for 4:00:00 on 1 host(s) and their services with reason: rebooting to help with lag