- At 16:00 Max prepares for SWAT but sees errors in fatalmonitor and investigates:
- PHP Warning: Creating default object from empty value in /srv/mediawiki/wmf-config/CommonSettings.php on line 686
- PHP Notice: Undefined variable: wgContactConfig in /srv/mediawiki/wmf-config/CommonSettings.php on line 968
- Max sees no such errors in Logstash.
|operations/puppet||production||+69 -0||Include hhvm fatals and exceptions in scap canary checks|
- Mentioned In
- T121597: Implement MediaWiki pre-promote checks
T154646: Include fatal log rate check in scap canary test
rOPUP8be18ee5fcaf: Add the fatalmonitor query to logstash_checker
rOPUP9d0e5fead54a: Add the fatalmonitor query to logstash_checker
- Mentioned Here
- T154646: Include fatal log rate check in scap canary test
T121597: Implement MediaWiki pre-promote checks
What about 20170104-MonologSpi? That incident had an unconditional fatal error on all page views. I'm surprised scap's canary/logstash checker didn't catch this. T121597 wouldn't expose end-users to these basic errors in the first place, but at least now that we have the canary check it should have automatically aborted and reversed the deployment.
Is it because of the same reason as the discrepancy https://gerrit.wikimedia.org/r/304327 will fix? Or is there something else that needs to be added?
EDIT: Looks like T154646 will cover this.