Systemd started looping a restart of the update-etcd-mw-config-lastindex.service service, that is triggered by a systemd timer. The root cause is not clear but that filled the root partition due to syslog spam.
Jul 11 09:32:32 tegmen systemd[1]: Starting Update the etcd last modified index for MediaWiki config... Jul 11 09:32:32 tegmen systemd[1]: update-etcd-mw-config-lastindex.service failed to run 'start' task: Resource temporarily unavailable Jul 11 09:32:32 tegmen systemd[1]: Failed to start Update the etcd last modified index for MediaWiki config. Jul 11 09:32:32 tegmen systemd[1]: Unit update-etcd-mw-config-lastindex.service entered failed state. Jul 11 09:32:32 tegmen systemd[1]: Starting Update the etcd last modified index for MediaWiki config... Jul 11 09:32:32 tegmen systemd[1]: update-etcd-mw-config-lastindex.service failed to run 'start' task: Resource temporarily unavailable Jul 11 09:32:32 tegmen systemd[1]: Failed to start Update the etcd last modified index for MediaWiki config.
So far one of the proposal has been to try to move the timer to an on calendar event. Unsure if that might help. As of today we don't have a way to repro it, it has happened only once.
The Max concurrent service checks message logged has been fixed already, so not anymore an issue, and kinda unrelated to the real root cause here.
Original description
tegmen went down (T199318) and when I logged in to see what might have caused the crash I noticed that the root partition was full, there's a 32G syslog.1 file (daemon.log is also 30G) which is mostly filled with log lines like
Jul 11 07:06:54 tegmen icinga: #011Max concurrent service checks (6000) has been reached. Nudging (...)
For some reason this didn't occur on einsteinium. In T1242 the limit was raised from 3200 to 6000, maybe with increased checks we need to bump it again?