Hi, I have a tool bot on toolforge mdbot and when I'm trying to save edits on crontab I got this error message
crontab: crontabs/tmp.YdH9kW: No space left on device /usr/local/bin/crontab: unable to execute remote crontab command
I have checked df -h and df -i and every thing looks normal, barley used tmp files, so what's the problem in this case?
Me and @Andrew removed some stuff, please try again. Note this is writing files on tools-sgecron-01 rather than whatever bastion you are logged on to, so a simple df won't show anything.
No we just removed some stuff under the system's /var/log. It looks like /var/log/syslog for example had filled up with collector-runner exceptions, it had managed to generate over 9 million lines in 36 hours, like this:
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: 2020-11-29 17:42:11,517 Exception trying to validate / load tool grantmetrics
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: Traceback (most recent call last):
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: File "/usr/lib/python3/dist-packages/tools/manifest/webservicemonitor.py", line 39, in from_name
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: user_info = pwd.getpwnam(username)
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: KeyError: 'getpwnam(): name not found: tools.grantmetrics'
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: During handling of the above exception, another exception occurred:
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: Traceback (most recent call last):
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: File "/usr/lib/python3/dist-packages/tools/manifest/webservicemonitor.py", line 146, in collect
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: tool = Tool.from_name(toolname)
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: File "/usr/lib/python3/dist-packages/tools/manifest/webservicemonitor.py", line 42, in from_name
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: raise Tool.InvalidToolException("No tool with name %s" % (name,))
Nov 29 17:42:11 tools-sgecron-01 collector-runner[9414]: tools.manifest.webservicemonitor.Tool.InvalidToolException: No tool with name grantmetricsIt's still generating more so this will happen again at some point. It has 3.7G to burn through first though.