When running puppet agent for the first time on the doh* hosts after https://gerrit.wikimedia.org/r/q/Id438985ffe720dc630f0e43eed8bda4a47c9196c, the auditd service failed to start with the following message:
Created symlink /etc/systemd/system/multi-user.target.wants/auditd.service -> /lib/systemd/system/auditd.service. Job for auditd.service failed because a timeout was exceeded. See "systemctl status auditd.service" and "journalctl -xe" for details. invoke-rc.d: initscript auditd, action "start" failed. * auditd.service - Security Auditing Service Loaded: loaded (/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed
The failure happened on some hosts but not all of them, within the same change. It seems like this is a bug in auditd: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962451 where it timeouts when started after an install and then doing it again fixes it; this matches the behaviour we observed. A possible fix is available at https://github.com/linux-audit/audit-userspace/commit/ee6608eca034494fc2597b2990852adec236e486.
We should observe if this happens again after subsequent restarts and then consider backporting the patch to our auditd build.