Description

https://grafana.wikimedia.org/dashboard/db/save-timing

Screen Shot 2017-03-24 at 21.36.16.png (914×2 px, 344 KB)

Screen Shot 2017-03-24 at 21.39.15.png (906×1 px, 180 KB)

https://grafana.wikimedia.org/dashboard/db/performance-metrics

Screen Shot 2017-03-24 at 21.36.25.png (520×708 px, 39 KB)

Drop started straight at 2017-03-23 21:18:00 UTC

https://wikitech.wikimedia.org/wiki/Server_Admin_Log

2017-03-23

20:16 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.17

Details

	Project	Branch	Lines +/-	Subject
	mediawiki/extensions/NavigationTiming	wmf/1.29.0-wmf.17	+15 -13	ext.NavigationTiming: Restore unsampled Save Timing
	mediawiki/extensions/NavigationTiming	master	+15 -13	ext.NavigationTiming: Restore unsampled Save Timing

Customize query in gerrit

Related Objects

Mentioned Here: rENTI54d8b4232384: ext.navigationTiming: Move hoisted functions before statements
rENTI8957895a6e9e: ext.navigationTiming: Only load eventlogging when needed
rENTIa7e536949034: build: Enable ESLint for tests/ and fix violations

Event Timeline

Krinkle created this task.Mar 25 2017, 4:37 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 25 2017, 4:37 AM

https://github.com/wikimedia/mediawiki-extensions-NavigationTiming/commits/wmf/1.29.0-wmf.17:

Most likely I broke something with 8957895a6e9e that somehow only affects Save Timing.

In T161368#3130149, @Krinkle wrote:>

Most likely I broke something with 8957895a6e9e that somehow only affects Save Timing.

Yup. It used to be that navtiming is sampled but save timing is not sampled. After my change, both are sampled. The sampling is meant to apply to page views, not edits. This changed reduce Save Timing rate from 250/minute to 0 most minutes, and 1 or 2 in others.
https://grafana.wikimedia.org/dashboard/db/eventlogging-schema?var-schema=SaveTiming

I'll fix this on Monday.

Krinkle renamed this task from Frontend Save Timing broken (or regressed by +18.000s) to Frontend Save Timing broken (<1 metrics per minute).Mar 25 2017, 9:35 PM

Krinkle claimed this task.

Change 345051 had a related patch set uploaded (by Krinkle):
[mediawiki/extensions/NavigationTiming@master] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345051

gerritbot added a project: Patch-For-Review.Mar 27 2017, 9:42 PM

Krinkle triaged this task as Unbreak Now! priority.Mar 27 2017, 9:42 PM

Krinkle added a project: NavigationTiming.

Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptMar 27 2017, 9:42 PM

Krinkle moved this task from Inbox to Doing (old) on the Performance-Team board.Mar 27 2017, 9:42 PM

Change 345078 had a related patch set uploaded (by Krinkle):
[mediawiki/extensions/NavigationTiming@wmf/1.29.0-wmf.17] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345078

Change 345051 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@master] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345051

ReleaseTaggerBot added a project: MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)).Mar 28 2017, 1:00 AM

Change 345078 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@wmf/1.29.0-wmf.17] ext.NavigationTiming: Restore unsampled Save Timing

https://gerrit.wikimedia.org/r/345078

Mentioned in SAL (#wikimedia-operations) [2017-03-28T23:22:05Z] <thcipriani@tin> Synchronized php-1.29.0-wmf.17/extensions/NavigationTiming/modules/ext.navigationTiming.js: SWAT: [[gerrit:345078|ext.NavigationTiming: Restore unsampled Save Timing]] T161368 (duration: 00m 45s)

Recovered.

Screen Shot 2017-03-28 at 16.38.51.png (596×1 px, 46 KB)

Screen Shot 2017-03-28 at 16.39.55.png (902×1 px, 117 KB)

Krinkle closed this task as Resolved.Mar 28 2017, 11:40 PM

ReleaseTaggerBot edited projects, added MW-1.29-release (WMF-deploy-2017-03-21_(1.29.0-wmf.17)); removed MW-1.29-release (WMF-deploy-2017-03-28_(1.29.0-wmf.18)).Mar 29 2017, 12:00 AM

	F7063902: Screen Shot 2017-03-28 at 16.38.51.png
	Mar 28 2017, 11:40 PM

	F7063949: Screen Shot 2017-03-28 at 16.39.55.png
	Mar 28 2017, 11:40 PM

	F6948332: Screen Shot 2017-03-24 at 21.39.15.png
	Mar 25 2017, 4:40 AM

	F6948282: Screen Shot 2017-03-24 at 21.36.25.png
	Mar 25 2017, 4:37 AM

	F6948284: Screen Shot 2017-03-24 at 21.36.16.png
	Mar 25 2017, 4:37 AM

Frontend Save Timing broken (<1 metrics per minute)Closed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Frontend Save Timing broken (<1 metrics per minute)
Closed, ResolvedPublic
Actions