Page MenuHomePhabricator

Frontend Save Timing broken (<1 metrics per minute)
Closed, ResolvedPublic

Description

https://grafana.wikimedia.org/dashboard/db/save-timing

Screen Shot 2017-03-24 at 21.36.16.png (914×2 px, 344 KB)

Screen Shot 2017-03-24 at 21.39.15.png (906×1 px, 180 KB)

https://grafana.wikimedia.org/dashboard/db/performance-metrics

Screen Shot 2017-03-24 at 21.36.25.png (520×708 px, 39 KB)

Drop started straight at 2017-03-23 21:18:00 UTC

2017-03-23

  • 20:16 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.17

Event Timeline

Most likely I broke something with 8957895a6e9e that somehow only affects Save Timing.

Yup. It used to be that navtiming is sampled but save timing is not sampled. After my change, both are sampled. The sampling is meant to apply to page views, not edits. This changed reduce Save Timing rate from 250/minute to 0 most minutes, and 1 or 2 in others.
https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?var-schema=SaveTiming

I'll fix this on Monday.

Krinkle renamed this task from Frontend Save Timing broken (or regressed by +18.000s) to Frontend Save Timing broken (<1 metrics per minute).Mar 25 2017, 9:35 PM
Krinkle claimed this task.

Change 345051 had a related patch set uploaded (by Krinkle):
[mediawiki/extensions/NavigationTiming@master] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345051

Krinkle triaged this task as Unbreak Now! priority.Mar 27 2017, 9:42 PM
Krinkle added a project: NavigationTiming.

Change 345078 had a related patch set uploaded (by Krinkle):
[mediawiki/extensions/NavigationTiming@wmf/1.29.0-wmf.17] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345078

Change 345051 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@master] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345051

Change 345078 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@wmf/1.29.0-wmf.17] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345078

Mentioned in SAL (#wikimedia-operations) [2017-03-28T23:22:05Z] <thcipriani@tin> Synchronized php-1.29.0-wmf.17/extensions/NavigationTiming/modules/ext.navigationTiming.js: SWAT: [[gerrit:345078|ext.NavigationTiming: Restore unsampled Save Timing]] T161368 (duration: 00m 45s)