Page MenuHomePhabricator

Newline in scap sync-file message: no notification, no SAL entry
Open, MediumPublic

Description

I accidentally had a newline in a sync-file message, and as a result my sync didn't appear in the SAL nor on IRC (synced today, 2019-03-04):

gilles@deploy1001:/srv/mediawiki-staging$ scap sync-file wmf-config/InitialiseSettings.php "T209857 
> Increase CPU benchmark sampling factor"
           ___ ____
         ⎛   ⎛ ,----
          \  //==--'
     _//|,.·//==--'    ____________________________
    _OO≣=-  ︶ ᴹw ⎞_§ ______  ___\ ___\ ,\__ \/ __ \
   (∞)_, )  (     |  ______/__  \/ /__ / /_/ / /_/ /
     ¨--¨|| |- (  / ______\____/ \___/ \__^_/  .__/
         ««_/  «_/ jgs/bd808                /_/

15:17:36 Checking for new runtime errors locally
15:17:37 Copying from deploy1001.eqiad.wmnet to deploy1001.eqiad.wmnet
15:17:37 Started rsync common
15:17:37 Finished rsync common (duration: 00m 00s)
15:17:37 Started cache_git_info
15:17:44 Finished cache_git_info (duration: 00m 06s)
15:17:44 Started sync-masters
sync-masters: 100% (ok: 1; fail: 0; left: 0)                                    
15:17:52 Finished sync-masters (duration: 00m 08s)
15:17:52 Started sync-pull-masters
sync-pull-masters: 100% (ok: 1; fail: 0; left: 0)                               
15:17:54 Finished sync-pull-masters (duration: 00m 01s)
15:17:54 Started sync-check-canaries
check-canaries: 100% (ok: 11; fail: 0; left: 0)                                 
15:17:55 Finished Canaries Synced (duration: 00m 01s)
15:17:56 Executing check 'Check endpoints for mw1279.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1276.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1261.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1264.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mwdebug1002.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mwdebug1001.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1263.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1262.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1278.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1277.eqiad.wmnet'
15:17:56 Executing check 'Check endpoints for mw1265.eqiad.wmnet'
15:17:58 Finished Canary Endpoint Check Complete (duration: 00m 02s)
15:17:58 Waiting for canary traffic...
15:18:15 Executing check 'Logstash Error rate for mw1279.eqiad.wmnet'
15:18:15 Executing check 'Logstash Error rate for mw1276.eqiad.wmnet'
15:18:15 Executing check 'Logstash Error rate for mw1261.eqiad.wmnet'
15:18:15 Executing check 'Logstash Error rate for mw1264.eqiad.wmnet'
15:18:16 Executing check 'Logstash Error rate for mwdebug1002.eqiad.wmnet'
15:18:16 Executing check 'Logstash Error rate for mwdebug1001.eqiad.wmnet'
15:18:16 Executing check 'Logstash Error rate for mw1263.eqiad.wmnet'
15:18:16 Executing check 'Logstash Error rate for mw1262.eqiad.wmnet'
15:18:16 Executing check 'Logstash Error rate for mw1278.eqiad.wmnet'
15:18:16 Executing check 'Logstash Error rate for mw1277.eqiad.wmnet'
15:18:16 Executing check 'Logstash Error rate for mw1265.eqiad.wmnet'
15:18:16 Finished sync-check-canaries (duration: 00m 21s)
15:18:16 Started sync-proxies
sync-proxies: 100% (ok: 9; fail: 0; left: 0)                                    
15:18:18 Finished sync-proxies (duration: 00m 02s)
15:18:18 Started sync-apaches
sync-apaches: 100% (ok: 263; fail: 0; left: 0)                                  
15:18:23 Finished sync-apaches (duration: 00m 05s)
15:18:25 Synchronized wmf-config/InitialiseSettings.php: T209857 
Increase CPU benchmark sampling factor (duration: 00m 48s)

Event Timeline

thcipriani triaged this task as Medium priority.Mar 12 2019, 2:22 PM
thcipriani moved this task from Needs triage to Debt on the Scap board.
thcipriani subscribed.

I do see the message in its entirety in logstash

Synchronized wmf-config/InitialiseSettings.php: T209857
Increase CPU benchmark sampling factor (duration: 00m 48s)

so that would probably narrow this down to code in the error handler emit method (https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/master/scap/log.py#126)

or something strange with https://wikitech.wikimedia.org/wiki/Logmsgbot