Change Details

The 2018-02-26 WikibaseQualityConstraints incident (documentation to follow once wikitech is no longer read-only) resulted in a strong server load spike on replica servers within minutes of deployment of the relevant config change, but incident response only started almost three hours later. There must be something to improve here, but I’m afraid I’m not familiar enough with the operations environment to say what exactly. @jcrespo said on IRC: > a prometheus alert would be nice

The 2018-02-26 WikibaseQualityConstraints incident (https://wikitech.wikimedia.org/wiki/Incident_documentation/20180226-WikibaseQualityConstraints) resulted in a strong server load spike on replica servers within minutes of deployment of the relevant config change, but incident response only started almost three hours later. There must be something to improve here, but I’m afraid I’m not familiar enough with the operations environment to say what exactly. @jcrespo said on IRC: > a prometheus alert would be nice

The 2018-02-26 WikibaseQualityConstraints incident (documentation to follow once wikitech is no longer read-onlyhttps://wikitech.wikimedia.org/wiki/Incident_documentation/20180226-WikibaseQualityConstraints) resulted in a strong server load spike on replica servers within minutes of deployment of the relevant config change, but incident response only started almost three hours later. There must be something to improve here, but I’m afraid I’m not familiar enough with the operations environment to say what exactly. @jcrespo said on IRC: > a prometheus alert would be nice