It seems like median/p75/p95 are not changed on the fly,
see that they keep the same value for hours (https://grafana.wikimedia.org/dashboard/db/navigation-timing-alerts?refresh=5m&orgId=1&panelId=23&fullscreen&from=1505436334194&to=1505598247554):

The metrics on that graph are 24h moving averages, it would makes sense that they don't move instantly.

However, something is very wrong with that graph (at least over that period), as median p75 and p95 are identical, with or without the moving average. The explanation is seen when zooming out, it seems like the data mostly stopped and the brokenness is just some artifact:

And the report rate confirms it, it tanked:

The timing corresponds with @Krinkle's deployment.

Maybe the ingestion point looks at the old EL schema? I'm going to check EL SQL data to see if it's working there or not.

SELECT COUNT(*) FROM `NavigationTiming_16305090` WHERE timestamp >= 20170912000000 AND timestamp < 20170913000000

361199

SELECT COUNT(*) FROM `NavigationTiming_17216284` WHERE timestamp >= 20170915000000 AND timestamp < 20170916000000

288187

Looks like the same order of magnitude, difference is probably due to weekday/weekend. This would suggest that Grafana/graphite is just looking at the old schema, which is still getting a tiny trickle of clients with outdated cached client side code referencing the old schema version.

Code that needs updating for the new metric:

https://github.com/wikimedia/puppet/blob/fb10b98a3e17d15201be8252b708f4c94241f9eb/modules/role/files/mariadb/eventlogging_purging_whitelist.tsv

https://github.com/wikimedia/puppet/blob/1132c140fb392043d671f0df12b97da7c4c75bb5/modules/coal/files/coal

https://github.com/wikimedia/puppet/blob/c1f24c8ccfbed9f0bee0c46a87afe0042b6f234c/modules/webperf/files/navtiming.py

I'm not familiar with how EL kafka consumption works, it's possible that something would have needed to be restarted in that setup to pick up the new schema?

The ideas was that we didn't need to change coal at at the same moment (pickup that new metrics) but we missed something, I'm not familiar how it exactly works.

Coal is fine, it's the EL/kafka intake that seems to be stuck on the old schema.

In T176105#3614326, @Gilles wrote:

Coal is fine, it's the EL/kafka intake that seems to be stuck on the old schema.

This is odd. We very intentionally only filter by schema name in our pipeline, not by revision, so that we always consider older versions without needing to fragment the handling at each layer. In addition, we aim that each version is backwards and forwards compatible (we only add optional fields and never tighten restrictions on existing fields).

Reporting of navtiming.fetchStart in Graphite went from 70/min to ~ 1/min:

Reporting of navtiming.mediaWikiLoadComplete went from 200/min to 30/min:
https://grafana.wikimedia.org/dashboard/db/navigation-timing?refresh=5m&panelId=12&fullscreen&orgId=1&var-metric=mediaWikiLoadComplete

Statistics from Kafka production directly, also impacted, albeit not nearly as much. Went from around 300/min to 250/min.

https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&orgId=1&from=now-7d&to=now&var-server=hafnium&var-network=eth0

Change 378737 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/NavigationTiming@master] Unbreak navTiming reporting

https://gerrit.wikimedia.org/r/378737

Change 378740 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/extensions/NavigationTiming@wmf/1.30.0-wmf.18] Unbreak navTiming reporting

https://gerrit.wikimedia.org/r/378740

Change 378737 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@master] Unbreak navTiming reporting

https://gerrit.wikimedia.org/r/378737

Change 378740 merged by jenkins-bot:
[mediawiki/extensions/NavigationTiming@wmf/1.30.0-wmf.18] Unbreak navTiming reporting

https://gerrit.wikimedia.org/r/378740

Mentioned in SAL (#wikimedia-operations) [2017-09-18T21:18:47Z] <krinkle@tin> Synchronized php-1.30.0-wmf.18/extensions/NavigationTiming/modules/ext.navigationTiming.js: Unbreak - T176105 (duration: 00m 48s)