ruthenium runs parsoid tests, specifically "visualdiff" tests.
They fill up /srv/visualdiff/png with lots of small files , several GB every once in a while.
This made ruthenium run low on disk, as everything was on the 40GB / mount.
We fixed that for now by mounting the un-used /dev/mapper/ruthenium-vg-tank with over 377GB free space.
But we still need proper logrotate for this added to the puppet role.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| parsoid-testing: add logrotate | operations/puppet | production | +19 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Invalid | None | T161920 logrotate for visualdiff tests on Parsoid test host (scandium) | |||
| Resolved | jcrespo | T164915 ruthenium is going to run out of space on /srv and stop working |
Mentioned in SAL (#wikimedia-operations) [2017-03-31T18:27:19Z] <mutante> ruthenium mounting /dev/mapper/ruthenium--vg-tank into /srv/visualdiff/pngs | deleted "mysql" and "dumps" data that was on previously unmounted partition , subbu checked that wasn't needed anymore, we still need logrotate (T161920)
correction: logrotate for /var/log/parsoid !
desired rules:
<subbu> rotate after log size hits 500mb; gzip after rotation; delete all files older than 24 hours.
and since we have more space:
<subbu> alright, so, you can change logrotate rules to: logrotate after 24 hours; gzip; delete all files older than 1 week ;-)
Change 345888 had a related patch set uploaded (by Dzahn):
[operations/puppet@production] parsoid-testing: add logrotate
Change 345888 merged by Dzahn:
[operations/puppet@production] parsoid-testing: add logrotate
Mentioned in SAL (#wikimedia-operations) [2017-03-31T23:11:56Z] <mutante> ruthenium: logrotate --force /etc/logrotate.d/parsoid (note this is existing file "parsoid" not new file "parsoid_testing") (T161920)
@ssastry Right after adding our new logrotate snippet for parsoid-testing on ruthenium, i noticed there is also an existing logrotate config for parsoid.
/etc/logrotate.d/parsoid
# logrotate(8) config for parsoid
/srv/log/parsoid/*.log {
daily
copytruncate
missingok
compress
delaycompress
notifempty
rotate 15
size 256M
}vs. our new /etc/logrotate.d/parsoid_testing
#####################################################################
### THIS FILE IS MANAGED BY PUPPET
### puppet:///modules/parsoid/parsoid_testing_logrotate
#####################################################################
/srv/log/parsoid/main.log {
daily
compress
delaycompress
missingok
notifempty
rotate 7
}Having both of them handle the same log wouldn't be right, so i reverted the change that added the new config file.
What i did instead was run logrotate --force /etc/logrotate.d/parsoid to manually trigger the rotation using the existing config file.
The result was that we now see main.log main.log.1 main.log.2.gz syslog.log syslog.log.1 in /srv/log/parsoid/
Yes, there are 56G in scandium:/srv/visualdiff/pngs but that is just over 50% so not an acute issue here.
@ssastry Do you ever manually delete pngs from /srv/visualdiff? Should we have a cron to delete old files or do you think this is a non-issue now?
Since we aren't going to be running parsoid-vd and parsoid-vd-client on scanidum, that whole dir can be removed and unmounted (iirc, that is an external volume that is mounted onto scandium).
Thanks! Reimaging scandium right now as part of T257906.
We can close this ticket then as not needed anymore.