The prune_old_srv_syslog_directories.service fails when it attempts to delete non-empty directories.
The journalctl -u prune_old_srv_syslog_directories.service output is /usr/bin/find: cannot delete ‘/srv/syslog/.linux.dhcp.DictModel’: Directory not empty.
Looking inside of that directory there's a log file from November 30 2023 which is old enough for the service to delete the file and directory.
Output of ls -la /srv/syslog/.linux.dhcp.DictModel/:
total 13832 drwxr-x--- 2 root ops 4096 Nov 27 19:42 . drwxr-xr-x 1 root root 13926400 Apr 12 00:00 .. -rw-r----- 1 root ops 219051 Nov 30 11:13 syslog.log
This is the output of of the /lib/systemd/system/prune_old_srv_syslog_directories.service unit:
[Unit] Description=clean up logs from old hosts Documentation=https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [Service] Type=oneshot User=root ExecStart=/usr/bin/find /srv/syslog/ -mtime +135 -delete
I don't think this is a permissions issue as the unit is ran as the root user. I think the issue has to do with a missing argument for the find command to delete non-empty directories.
I think that either forcing the delete or a recursive delete of the directories may also work.
From the find man pages: The -delete action will fail to remove a directory unless it is empty. This explains why the service is failing.