Page MenuHomePhabricator

Investigate 1AM enwiki slave lag spikes
Closed, ResolvedPublic

Description

See https://tendril.wikimedia.org/host/view/db1072.eqiad.wmnet/3306 (lag goes from 1-2sec to 20sec).

Tempted to at least partly blame the misc::maintenance::purge_abusefilter job. It runs at 1AM daily and uses 'LIMIT' with UPDATE, which we don't support so the LIMIT is ignored...patch incoming.
Taking that issue to task https://phabricator.wikimedia.org/T95382?workflow=create.

Event Timeline

aaron created this task.Apr 8 2015, 2:19 AM
aaron claimed this task.
aaron raised the priority of this task from to Needs Triage.
aaron updated the task description. (Show Details)
aaron added a subscriber: aaron.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 8 2015, 2:19 AM

Change 202657 had a related patch set uploaded (by Aaron Schulz):
Fixed broken batching in –PurgeOldLogIPData

https://gerrit.wikimedia.org/r/202657

aaron updated the task description. (Show Details)Apr 8 2015, 2:31 AM
aaron set Security to None.

Change 202640 had a related patch set uploaded (by Aaron Schulz):
Log huge write queries in CLI scripts

https://gerrit.wikimedia.org/r/202640

Change 202763 had a related patch set uploaded (by Aaron Schulz):
Use wfWaitForSlaves in upload stash cleanup script

https://gerrit.wikimedia.org/r/202763

Change 202792 had a related patch set uploaded (by Aaron Schulz):
Fixed broken batching in –PurgeOldLogIPData

https://gerrit.wikimedia.org/r/202792

Change 202640 merged by jenkins-bot:
Log huge write queries in CLI scripts

https://gerrit.wikimedia.org/r/202640

Change 202763 merged by jenkins-bot:
Use wfWaitForSlaves in upload stash cleanup script

https://gerrit.wikimedia.org/r/202763

Change 202657 merged by jenkins-bot:
Fixed broken batching in –PurgeOldLogIPData

https://gerrit.wikimedia.org/r/202657

Change 202792 merged by jenkins-bot:
Fixed broken batching in –PurgeOldLogIPData

https://gerrit.wikimedia.org/r/202792

aaron moved this task from Backlog to Doing on the Availability board.Apr 8 2015, 10:55 PM

Change 203204 had a related patch set uploaded (by Aaron Schulz):
Fixed broken batching in –PurgeOldLogIPData

https://gerrit.wikimedia.org/r/203204

Change 203204 merged by jenkins-bot:
Fixed broken batching in –PurgeOldLogIPData

https://gerrit.wikimedia.org/r/203204

Change 203615 had a related patch set uploaded (by Aaron Schulz):
Log huge write queries in CLI scripts

https://gerrit.wikimedia.org/r/203615

Change 203615 merged by Aaron Schulz:
Log huge write queries in CLI scripts

https://gerrit.wikimedia.org/r/203615

aaron closed this task as Resolved.Apr 13 2015, 11:11 PM