Wikidata dispatching slow and maxlag high on Wikidata due to db1101 replication lag
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Bugreporter
	May 16 2020, 8:03 PM

Description

For about 20 hours Wikidata dispatch lag has been being higher than 10 minutes. Further investigation is needed.

See https://grafana.wikimedia.org/d/000000156/wikidata-dispatch?orgId=1&from=now-2d&to=now&refresh=1m

Details

	Subject	Repo	Branch	Lines +/-
	Observability for SqlChangeDispatchCoordinator wait for replication	mediawiki/extensions/Wikibase	master	+7 -1

Customize query in gerrit

Related Objects

Mentioned In: T367278: Migrate mysql icinga alerts to alert manager - pt-heartbeat + scaffolding
T288056: Rename the "databases-testing" Icinga contact group
T253746: high dispatch lag in Wikidata (27 May 2020)
T253120: Create prometheus alert to detect lag spikes
T247728: Events set to SLAVESIDE_DISABLED when upgrading from 10.1 to 10.4
Mentioned Here: T247728: Events set to SLAVESIDE_DISABLED when upgrading from 10.1 to 10.4

Event Timeline

Bugreporter created this task.May 16 2020, 8:03 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 16 2020, 8:03 PM

Bugreporter triaged this task as Unbreak Now! priority.May 16 2020, 8:04 PM

Bugreporter updated the task description. (Show Details)

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 16 2020, 8:04 PM

Addshore subscribed.May 16 2020, 8:04 PM

Addshore added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).May 16 2020, 8:12 PM

Maintenance_bot moved this task from incoming to in progress on the Wikidata board.May 16 2020, 8:15 PM

Pasleim subscribed.May 16 2020, 8:46 PM

The timing lines up with this db lag on db1101

Dispatching is slowing down, as the db server is lagged and the dispatch process has a waitForReplication call in the code.
Specifically this is in SqlChangeDispatchCoordinator::releaseClient

This db server probably needs to be depooled.

Addshore renamed this task from Fix Wikidata dispatch to Wikidata dispatching slow and maxlag high on Wikidata due to db1101 replication lag.May 16 2020, 9:16 PM

Addshore added subscribers: Lydia_Pintscher, Ladsgroup, jcrespo, Marostegui.

Change 596824 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] SqlChangeDispatchCoordinator statsd time the wait for replication

https://gerrit.wikimedia.org/r/596824

gerritbot added a project: Patch-For-Review.May 16 2020, 9:20 PM

I contact @Marostegui

Addshore added subscribers: Lea_Lacroix_WMDE, darthmon_wmde, WMDE-leszek.May 16 2020, 9:23 PM

Marking as resolved as the impact on wikidata is now gone

10:32 PM <marostegui> This host is lagging just a few seconds but all the time
10:32 PM <marostegui> Due to:
10:32 PM <marostegui> yes
10:33 PM <jynus> Query | 95861
10:33 PM <marostegui> that
10:33 PM <marostegui> I am going to kill all that
10:33 PM <marostegui> the query killer is disabled due to a 10.4 bug

Restricted Application added a project: User-Addshore. · View Herald TranscriptMay 16 2020, 9:43 PM

{P11212}

Some more details.
There were a few long running queries

| 669154401 | wikiuser        | 10.64.0.59:40112   | wikidatawiki       | Query     |   95780 | Sending data                                   | SELECT /* SpecialRecentChanges::doMainQuery  */  rc_id,rc_timestamp,rc_namespace,rc_title,rc_

Those queries were running for hours and as the query killer wasn't enabled due to a mariadb 10.4 bug (T247728), they were causing 3-5 seconds of continued lag.

Change 596824 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Observability for SqlChangeDispatchCoordinator wait for replication

https://gerrit.wikimedia.org/r/596824

Maintenance_bot removed a project: Patch-For-Review.May 17 2020, 1:10 AM

ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.34; 2020-05-26).May 17 2020, 2:00 AM

Marostegui mentioned this in T247728: Events set to SLAVESIDE_DISABLED when upgrading from 10.1 to 10.4.May 18 2020, 7:47 AM

• Tonina_Zhelyazkova_WMDE subscribed.May 18 2020, 9:54 AM

Marostegui mentioned this in T253120: Create prometheus alert to detect lag spikes.May 19 2020, 1:07 PM

Addshore mentioned this in T253746: high dispatch lag in Wikidata (27 May 2020).May 28 2020, 8:40 AM

jcrespo mentioned this in T288056: Rename the "databases-testing" Icinga contact group.Aug 4 2021, 1:02 PM

jcrespo mentioned this in T367278: Migrate mysql icinga alerts to alert manager - pt-heartbeat + scaffolding.Jun 19 2024, 8:41 PM

	F31822639: image.png
	May 16 2020, 9:05 PM

	F31822631: image.png
	May 16 2020, 9:03 PM

Wikidata dispatching slow and maxlag high on Wikidata due to db1101 replication lagClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Wikidata dispatching slow and maxlag high on Wikidata due to db1101 replication lag
Closed, ResolvedPublic
Actions