Page MenuHomePhabricator

Scap's canary check should stop us from deploying config changes that cause floods of "undefined variable" errors
Closed, ResolvedPublic

Description

@Niharika and myself SWATted a patch that removed a setting in InitialiseSettings.php while it was still being used in CommonSettings.php. This caused hhvm.log to be flooded with:

Apr 13 23:54:07 mw1223:  #012Notice: Undefined variable: wmgRelatedArticlesFooterBlacklistedSkins in /srv/mediawiki/wmf-config/CommonSettings.php on line 2878
Apr 13 23:54:07 mw1223:  [Thu Apr 13 23:54:06 2017] [hphp] [22226:7f010bfff700:119034:000001] [] \nNotice: Undefined variable: wmgRelatedArticlesFooterBlacklistedSkins in /srv/mediawiki/wmf-config/CommonSettings.php on line 2878

at a frequency of >10 lines per second. The canary checks should have seen dozens if not hundreds of these during the 20s canary phase, and should have prevented us from deploying this patch.

Details

Related Gerrit Patches:

Event Timeline

Catrope created this task.Apr 14 2017, 12:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 14 2017, 12:15 AM
thcipriani triaged this task as Medium priority.Apr 14 2017, 12:20 AM
thcipriani moved this task from Needs triage to Debt on the Scap board.
thcipriani added subscribers: Krinkle, thcipriani.

canary check missed this as it ignores NOTICE log level messages in logstash https://github.com/wikimedia/puppet/blob/production/modules/service/files/logstash_checker.py#L143-L150

@Krinkle proposed we move only exclude DEBUG/INFO-level messages and include the mediawiki error channel as well (paraphrasing IRC)

Makes the dashboard look like: https://logstash.wikimedia.org/goto/0ec6676a577b05f7a0f946393428a33d for the time period of this deploy which would have caught the issue.

Change 348475 had a related patch set uploaded (by Thcipriani):
[operations/puppet@production] Scap: canaries should include INFO-level messages

https://gerrit.wikimedia.org/r/348475

Change 348475 merged by Filippo Giunchedi:
[operations/puppet@production] Scap: canaries should include INFO-level messages

https://gerrit.wikimedia.org/r/348475

thcipriani closed this task as Resolved.May 9 2017, 9:27 PM
thcipriani claimed this task.
thcipriani added a subscriber: fgiunchedi.

This change was deployed and is live now. Thanks for the swat @fgiunchedi !

Addshore added a subscriber: Addshore.