From: Cron Daemon <root@an-test-client1001.eqiad.wmnet> To: root@an-test-client1001.eqiad.wmnet Subject: Cron <root@an-test-client1001> for user in $(ls /srv/home); do rm -rf /srv/home/$user/.local/share/Trash/*; done Date: Sun, 11 Jul 2021 00:00:01 +0000 X-Cron-Env: <SHELL=/bin/sh> X-Cron-Env: <HOME=/root> X-Cron-Env: <PATH=/usr/bin:/bin> X-Cron-Env: <LOGNAME=root> Message-Id: <E1m2Mt7-0006Yy-63@an-test-client1001.eqiad.wmnet> Date: Sun, 11 Jul 2021 00:00:01 +0000 ls: cannot access '/srv/home': No such file or directory
Description
Description
Details
Details
Related Changes in Gerrit:
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| analytics: Migrate clean_jupyter_user_local_trash to systemd timer | operations/puppet | production | +9 -0 |
Event Timeline
Comment Actions
Is it coming from puppet? It should be migrated to systemd timer if that's the case: T273673: replace all puppet crons with systemd timers
Comment Actions
Yes, https://gerrit.wikimedia.org/g/operations/puppet/+/4a3bf542618f4550dfbe450452ddc9e6294ed1d3/modules/profile/manifests/analytics/jupyterhub.pp#61 is the cron
But I'm not sure migrating it to a timer fixes the underlying issue, which is that sometimes(?) /srv/home is missing.
Comment Actions
Actually it's not sometimes, it's always missing. We've been getting this since the end of June at least, which is when I last cleaned out my root@ folder.
Comment Actions
Nvm, it would. Even though the subshell is failing, it still exits with status code 0, so it's just the stderr that's triggering the email.
Comment Actions
Change 708183 had a related patch set uploaded (by Legoktm; author: Legoktm):
[operations/puppet@production] analytics: Migrate clean_jupyter_user_local_trash to systemd timer
Comment Actions
I think it's just this single host an-test-client1001 that's sending this daily logspam now, isn't it?
To me, the issue looks like we should just be setting up /home as a symlink to /srv/home on this box, like we do on all of the other analytics-client (stat100[4-8]) boxes.
btullis@an-test-client1001:~$ ls -ld /home drwxr-xr-x 240 root root 4096 Aug 3 15:26 /home
btullis@stat1008:~$ ls -ld /home lrwxrwxrwx 1 root root 9 Mar 12 2020 /home -> /srv/home
I've been searching through puppet, but I can't seem to find where this symlink is configured. Once I find it, we can add an-test-client1001.eqiad.wmnet to the list and rebuild it, or point /home to /srv/home manually.
Comment Actions
@BTullis we have been doing it manually for the stat100x boxes so far, nothing on puppet!
Comment Actions
OK, in that case I've done the following to clear this bit of cron spam temporarily.
btullis@an-test-client1001:~$ ls -l /srv/home ls: cannot access '/srv/home': No such file or directory btullis@an-test-client1001:~$ sudo mkdir /srv/home btullis@an-test-client1001:~$ ls -ld /srv/home drwxr-xr-x 2 root root 4096 Aug 25 11:09 /srv/home
We could decide to move /home to /srv/home at another time, but this should stop the emails I think.
Comment Actions
Change 708183 abandoned by Legoktm:
[operations/puppet@production] analytics: Migrate clean_jupyter_user_local_trash to systemd timer
Reason:
Comment Actions
I think that this is resolved, isn't it? I'm not seeing any regular cronspam from an-test hosts. Please feel free to reopen if I have missed something.