Page MenuHomePhabricator
Create Task
Maniphest T149210

Set cron script to dump MediaWiki DB lag times into statsd
Closed, ResolvedPublic
Actions

Description

Similar to getJobQueueLengths.php, we could modify getLagTimes.php and have a chron call it with a --report option. This would be good for showing:
a) MW perceived lag times
b) Not counting depooled servers (which often have technically high lag that doesn't matter)

Details

SubjectRepoBranchLines +/-
Set cron script to dump MediaWiki DB lag times into statsdoperations/puppetproduction+16 -0
Send integer ms to DB lag time guage instead of secondsmediawiki/coremaster+27 -20
Set cron script to dump MediaWiki DB lag times into statsdoperations/puppetproduction+16 -0
Improve getLagTimes.php output and add statsD flagmediawiki/coremaster+80 -7
Customize query in gerrit

Related Objects

Event Timeline

aaron created this task.Oct 26 2016, 4:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 26 2016, 4:46 PM
• jcrespo subscribed.Oct 26 2016, 4:52 PM
ori subscribed.Oct 26 2016, 5:16 PM

Depooled servers will continue to show up in Graphite and Grafana (particularly when using wildcards to select metrics), because the metric doesn't get deleted when you stop reporting values. For that reason, it might be useful to have a metric representing the average and maximum lag of all pooled servers.

gerritbot subscribed.Oct 26 2016, 10:21 PM

Change 318215 had a related patch set uploaded (by Aaron Schulz):
Improve getLagTimes.php output and add statsD flag

https://gerrit.wikimedia.org/r/318215

gerritbot added a project: Patch-For-Review.Oct 26 2016, 10:21 PM
• Gilles moved this task from Inbox, needs triage to Doing (old) on the Performance-Team board.Oct 27 2016, 8:11 PM
gerritbot added a comment.Nov 4 2016, 10:00 PM

Change 318215 merged by jenkins-bot:
Improve getLagTimes.php output and add statsD flag

https://gerrit.wikimedia.org/r/318215

ReleaseTaggerBot added projects: MW-1.29-release (WMF-deploy-2016-11-08_(1.29.0-wmf.2)), MW-1.29-release-notes.Nov 4 2016, 11:00 PM
Krinkle triaged this task as Medium priority.Nov 16 2016, 10:27 PM
Krinkle removed projects: MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2016-11-08_(1.29.0-wmf.2)), Patch-For-Review.
gerritbot added a comment.Dec 15 2016, 11:20 PM

Change 327667 had a related patch set uploaded (by Aaron Schulz):
Set cron script to dump MediaWiki DB lag times into statsd

https://gerrit.wikimedia.org/r/327667

gerritbot added a project: Patch-For-Review.Dec 15 2016, 11:20 PM
aaron moved this task from Doing (old) to Blocked (old) on the Performance-Team board.Dec 22 2016, 9:21 PM
• jcrespo added a comment.Feb 14 2017, 2:57 PM

I have unblocked this, but please have a look at my comments on gerrit.

gerritbot added a comment.Mar 22 2017, 5:23 PM

Change 327667 merged by Jcrespo:
[operations/puppet@production] Set cron script to dump MediaWiki DB lag times into statsd

https://gerrit.wikimedia.org/r/327667

• jcrespo added a comment.Mar 22 2017, 5:29 PM

This has been deployed to terbium. Please keep me updated on how this is used, I am interested on graphs or anything created.

• jcrespo added a comment.Mar 22 2017, 6:18 PM

Sadly, I had to revert the script (https://gerrit.wikimedia.org/r/344174) because I am almost sure this is not the intended results:

Screenshot from 2017-03-22 18:53:28.png (974×284 px, 39 KB)

Please followup with me to see how to deploy it again.

gerritbot added a comment.Mar 23 2017, 6:55 PM

Change 344472 had a related patch set uploaded (by Aaron Schulz):
[mediawiki/core@master] Send integer ms to DB lag time guage instead of seconds

https://gerrit.wikimedia.org/r/344472

gerritbot added a comment.Apr 5 2017, 1:09 AM

Change 344472 merged by jenkins-bot:
[mediawiki/core@master] Send integer ms to DB lag time guage instead of seconds

https://gerrit.wikimedia.org/r/344472

ReleaseTaggerBot added projects: MW-1.29-release (WMF-deploy-2017-04-11_(1.29.0-wmf.20)), MW-1.29-release-notes.Apr 5 2017, 2:00 AM
aaron added a comment.Apr 20 2017, 7:08 PM

Looks the puppet change can be attempted again.

• jcrespo added a comment.Apr 20 2017, 7:12 PM

No problem, but unless you have a specific reason to want it sooner (a specific test or issue to debug), I would wait until DC failback.

aaron added a comment.Apr 20 2017, 7:17 PM

Of course, it's lower priority, I'm just making note since I keep forgetting about this.

Krinkle removed projects: MW-1.29-release-notes, MW-1.29-release (WMF-deploy-2017-04-11_(1.29.0-wmf.20)), Patch-For-Review.Apr 20 2017, 7:32 PM
• jcrespo added a comment.Apr 21 2017, 8:21 PM

I just got a request from wikiuser checking the lag on a depooled server (hours ago): db2062, could this be related in some way to this script, checking incorrectly non-pooled servers. It is not icinga, it comes from mediawiki.

aaron added a comment.May 10 2017, 7:14 PM

Doubtful, the script just uses LoadBalancer::getLagTimes(), which is what the main code does for getConnection() lag checks.

gerritbot added a comment.May 17 2017, 6:46 PM

Change 354138 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[operations/puppet@production] Set cron script to dump MediaWiki DB lag times into statsd

https://gerrit.wikimedia.org/r/354138

gerritbot added a project: Patch-For-Review.May 17 2017, 6:46 PM
gerritbot added a comment.Jun 8 2017, 4:12 PM

Change 354138 merged by Filippo Giunchedi:
[operations/puppet@production] Set cron script to dump MediaWiki DB lag times into statsd

https://gerrit.wikimedia.org/r/354138

aaron closed this task as Resolved.Jun 10 2017, 12:10 PM

Graphs can be seen at https://grafana-admin.wikimedia.org/dashboard/db/mediawiki-mysql-loadbalancer?orgId=1&from=now-24h&to=now

Krinkle mentioned this in T172559: Ensure getLagTimes.php is working properly.Aug 4 2017, 8:06 PM
Dzahn mentioned this in T195392: Switch cronjobs on maintenance hosts to PHP7.Aug 7 2019, 2:16 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits