Today one of the compiler workers ran out of disk space, and the only indicator was failed job output.
To address this I can think of a few actions:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T222072 compiler1002.puppet-diffs.eqiad.wmflabs disk is full | |||
| Resolved | jbond | T222075 Prevent puppet catalog compiler workers from running out of disk space |
This happened again the other day and made me mail the SRE list. Then I added docs how to cleanup here:
https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs/Documentation#Out_of_disk_space
Change 830957 had a related patch set uploaded (by RLazarus; author: RLazarus):
[operations/puppet@production] pcc: Warn before compiling all nodes by default
Change 830957 merged by RLazarus:
[operations/puppet@production] pcc: Warn before compiling all nodes by default
Change 852280 had a related patch set uploaded (by Jbond; author: John Bond):
[operations/software/puppet-compiler@master] worker: store catalogs as gziped file
Change 852280 merged by jenkins-bot:
[operations/software/puppet-compiler@master] worker: store catalogs as gziped file
I have no preformed the following actions
Along with this we have for some time had cron jobs to delete old reports as such im going to close this but please reopen if there are still issues or further improvements please re-open with an update