Puppet failed on tools puppetmaster on 28/2 due to apache failing with - causing puppet failures across tools
Feb 28 06:49:53 tools-puppetmaster-02 apache2[22628]: Starting web server: apache2 failed! Feb 28 06:49:53 tools-puppetmaster-02 apache2[22628]: The apache2 configtest failed. ... (warning). Feb 28 06:49:53 tools-puppetmaster-02 apache2[22628]: Output of config test was: Feb 28 06:49:53 tools-puppetmaster-02 apache2[22628]: AH00526: Syntax error on line 8 of /etc/apache2/sites-enabled/50-puppetmaster-wikimedia-org.conf: Feb 28 06:49:53 tools-puppetmaster-02 apache2[22628]: Invalid command 'SSLOpenSSLConfCmd', perhaps misspelled or defined by a module not included in the server configuration Feb 28 06:49:53 tools-puppetmaster-02 apache2[22628]: Action 'configtest' failed. Feb 28 06:49:53 tools-puppetmaster-02 apache2[22628]: The Apache error log may have more information. Feb 28 06:49:53 tools-puppetmaster-02 systemd[1]: apache2.service: control process exited, code=exited status=1 Feb 28 06:49:53 tools-puppetmaster-02 systemd[1]: Failed to start LSB: Apache2 web server. Feb 28 06:49:53 tools-puppetmaster-02 systemd[1]: Unit apache2.service entered failed state.
_joe_ and moritzm helped fix it - and this was the underlying issue (from moritzm irc)
1:46 AM hi labs team, so apache on tools-puppetmaster-02 failed to start and that was caused by the use of unattended-upgrades in labs: 1:46 AM the standard apache package in Debian is built against openssl 1.0.1 1:49 AM but custom diffi hellman parameters can only be configured with apache is linked against openssl 1.0.2 (as required by "logjam"), so we're using a custom rebuild of apache on apt.wikimedia.org 1:49 AM so if Debian issues a new apache package it gets rebuilt internally and pushed to apt.wikimedia.org 1:51 AM but since labs uses unattended upgrades, the stock Debian version gets installed in the time window between Debian release and rebuild (DSA for apache happened Sunday evening and I pushed the rebuild yesterday at around 11am) 1:51 AM and since the native Debian version is not built against 1.0.2, it failed to start with an error since it doesn't provide SSLOpenSSLConfCmd 1:52 AM I suggest to blacklist apache from unattended-upgrades on at least the puppet masters using Unattended-Upgrade::Package-Blacklist: http://askubuntu.com/questions/193773/can-i-configure-unattended-upgrades-to-not-upgrade-packages-that-require-a-reboo 1:53 AM this is only needed for jessie, on trusty/precise the stock apache package is used 1:53 AM and stretch as well