Page MenuHomePhabricator

IAbot is writing loads of text to Toolforge NFS at a high rate
Closed, ResolvedPublic

Description

I truncated the /data/project/iabot/Workers/Worker2.out file at nearly a TB. However, it is still writing at the same speed, which may be why the NFS is experiencing high iowait at that moment. Please fix the verbosity or something.

Event Timeline

It's still growing. After I truncated it to zero, it went back over 30 GB really fast. In less than a minute, it gained 4GB.

Mentioned in SAL (#wikimedia-cloud) [2021-08-05T22:30:18Z] <bstorm> chmodded the file Worker2.out to read only to stop the bleeding T288300

Mentioned in SAL (#wikimedia-cloud) [2021-08-05T22:33:53Z] <bstorm> chowned the file Worker2.out to cyberpower678's account so take might work to revert things T288300

Bstorm triaged this task as High priority.Sep 15 2021, 11:35 PM

Currently, iabot is using over 500 GB of space on Toolforge and is the largest consumer of NFS in the tools project right now. Almost all of that is these log files:

bstorm@tools-sgebastion-08:/data/project/iabot/Workers$ du -sh *
109G	Worker1.out
79G	Worker2.out
101G	Worker3.out
99G	Worker4.out
140G	Worker5.out

There has to be something that can be done to curb the verbosity of the logging or rotate the logs. I'm going to truncate them all since the NFS shares are paging us for running low on space.

bstorm@tools-sgebastion-08:/data/project/iabot/Workers$ sudo tail Worker1.out
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...
RATE LIMIT: Sleeping for  second(s)...

for instance

Mentioned in SAL (#wikimedia-cloud) [2021-09-15T23:43:41Z] <bstorm> truncated massive files Worker[1-5].out T288300 T288276

They are not! I see they are currently staying at 0. That seems like the problem is resolved at this time.

bstorm@tools-sgebastion-08:/data/project/iabot/Workers$ du -sh *
0	Worker1.out
0	Worker2.out
0	Worker3.out
0	Worker4.out
0	Worker5.out