Last week I called out some of the logspam that lowered my confidence about deployments to great effect! All of the tasks I pointed to saw some kind of movement which was awesome.

RelEng is going to make a habit of calling out logspam on the train blocker task. Not to block the train, but to hopefully make meaningful progress towards having more confidence that a spike in the error logs is actually indicative of a problem.

A few huge log spammers this week are:

• T157646: wfShellExec errors end up in HHVM log – These errors are easily the noisiest recently. They take various forms, some relate to PDF handling and start with "SyntaxError".
• T157392: Warning: Cannot modify header information - headers already sent in /srv/mediawiki/php-1.29.0-wmf.10/includes/WebResponse.php on line 45 – Called this out last week, but there's limited information to go on here. The latest iteration of this error seems to have started with the release of 1.29.0-wmf.10. Some debug logging has shifted this warning to line 48 in wmf.11/12.
• T125735: Warning: timed out after 0.2 seconds when connecting to rdb1001.eqiad.wmnet [110]: Connection timed out – This message is still one of the loudest in logstash. The task saw a bit of movement last week, but there are some unanswered questions and a few proposed solutions. I don't have the familiarity with the underlying code to weigh-in on the Right™ answers there.

Change 337876 had a related patch set uploaded (by Thcipriani):
Revert "Revert "Group0 to 1.29.0-wmf.12""

https://gerrit.wikimedia.org/r/337876

Change 337876 merged by jenkins-bot:
Revert "Revert "Group0 to 1.29.0-wmf.12""

https://gerrit.wikimedia.org/r/337876

Mentioned in SAL (#wikimedia-operations) [2017-02-15T17:17:35Z] <thcipriani@tin> rebuilt wikiversions.php and synchronized wikiversions files: Group0 to 1.29.0-wmf.12 T155527