| Jdlrobson | |
| Aug 11 2021, 7:59 PM |
| F34590907: Screen Shot 2021-08-11 at 1.35.29 PM.png | |
| Aug 11 2021, 8:35 PM |
VirtualPageViews is part of a core metric but stopped logging with last train's deploy due to a rebase error in T267211.
https://grafana.wikimedia.org/d/000000566/overview?viewPanel=6&orgId=1
This should be fixed by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Popups/+/710724 and should be immediately backported.
Note: Alerts have been setup to avoid this happening in future.
Change 710724 had a related patch set uploaded (by Jdlrobson; author: Phuedx):
[mediawiki/extensions/Popups@wmf/1.37.0-wmf.18] virtualPageView: Log VirtualPageView events to Event Platform
Change 710723 had a related patch set uploaded (by Jdlrobson; author: Phuedx):
[mediawiki/extensions/Popups@wmf/1.37.0-wmf.17] virtualPageView: Log VirtualPageView events to Event Platform
Change 710724 merged by jenkins-bot:
[mediawiki/extensions/Popups@wmf/1.37.0-wmf.18] virtualPageView: Log VirtualPageView events to Event Platform
Change 710723 merged by jenkins-bot:
[mediawiki/extensions/Popups@wmf/1.37.0-wmf.17] virtualPageView: Log VirtualPageView events to Event Platform
Mentioned in SAL (#wikimedia-operations) [2021-08-11T20:20:20Z] <mholloway-shell@deploy1002> Synchronized php-1.37.0-wmf.18/extensions/Popups: Log VirtualPageView events to Event Platform (T288655) (duration: 01m 09s)
Mentioned in SAL (#wikimedia-operations) [2021-08-11T20:23:26Z] <mholloway-shell@deploy1002> Synchronized php-1.37.0-wmf.17/extensions/Popups: Log VirtualPageView events to Event Platform (T288655) (duration: 01m 06s)
Big thanks to @Mholloway @jlinehan @phuedx and @Milimetric for identifying the issue
The graph has recovered after the backport:
.
I'll check in ticket T267211 after a few days to make sure we are seeing normal levels.
I've documented this in https://www.mediawiki.org/wiki/Reading/Web/Notable_incidents and set up an email alert to avoid us going so long without noticing such an issue again.