Page MenuHomePhabricator

logrotate for visualdiff tests on Parsoid test host (scandium)
Open, MediumPublic

Description

ruthenium runs parsoid tests, specifically "visualdiff" tests.

They fill up /srv/visualdiff/png with lots of small files , several GB every once in a while.

This made ruthenium run low on disk, as everything was on the 40GB / mount.

We fixed that for now by mounting the un-used /dev/mapper/ruthenium-vg-tank with over 377GB free space.

But we still need proper logrotate for this added to the puppet role.

Details

Related Gerrit Patches:
operations/puppet : productionparsoid-testing: add logrotate

Event Timeline

Dzahn created this task.Mar 31 2017, 6:27 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 31 2017, 6:27 PM

Mentioned in SAL (#wikimedia-operations) [2017-03-31T18:27:19Z] <mutante> ruthenium mounting /dev/mapper/ruthenium--vg-tank into /srv/visualdiff/pngs | deleted "mysql" and "dumps" data that was on previously unmounted partition , subbu checked that wasn't needed anymore, we still need logrotate (T161920)

Dzahn added a comment.Mar 31 2017, 6:34 PM

correction: logrotate for /var/log/parsoid !

desired rules:

<subbu> rotate after log size hits 500mb; gzip after rotation; delete all files older than 24 hours.

and since we have more space:

<subbu> alright, so, you can change logrotate rules to: logrotate after 24 hours; gzip; delete all files older than 1 week ;-)

Change 345888 had a related patch set uploaded (by Dzahn):
[operations/puppet@production] parsoid-testing: add logrotate

https://gerrit.wikimedia.org/r/345888

Change 345888 merged by Dzahn:
[operations/puppet@production] parsoid-testing: add logrotate

https://gerrit.wikimedia.org/r/345888

Mentioned in SAL (#wikimedia-operations) [2017-03-31T23:11:56Z] <mutante> ruthenium: logrotate --force /etc/logrotate.d/parsoid (note this is existing file "parsoid" not new file "parsoid_testing") (T161920)

Dzahn triaged this task as Medium priority.Mar 31 2017, 11:13 PM

@ssastry Right after adding our new logrotate snippet for parsoid-testing on ruthenium, i noticed there is also an existing logrotate config for parsoid.

/etc/logrotate.d/parsoid

# logrotate(8) config for parsoid

/srv/log/parsoid/*.log {
    daily
    copytruncate
    missingok
    compress
    delaycompress
    notifempty
    rotate 15
    size 256M
}

vs. our new /etc/logrotate.d/parsoid_testing

#####################################################################
### THIS FILE IS MANAGED BY PUPPET
### puppet:///modules/parsoid/parsoid_testing_logrotate
#####################################################################

/srv/log/parsoid/main.log {
    daily
    compress
    delaycompress
    missingok
    notifempty
    rotate 7
}

Having both of them handle the same log wouldn't be right, so i reverted the change that added the new config file.

What i did instead was run logrotate --force /etc/logrotate.d/parsoid to manually trigger the rotation using the existing config file.

The result was that we now see main.log main.log.1 main.log.2.gz syslog.log syslog.log.1 in /srv/log/parsoid/

Dzahn added a comment.Jan 3 2019, 9:01 PM

T201366 is about replacing ruthenium with scandium (and jessie -> stretch upgrade)

MoritzMuehlenhoff renamed this task from logrotate for ruthenium to logrotate for visualdiff tests on Parsoid test host (scandium).Jul 5 2019, 12:46 PM