centrallog hosts in eqiad and codfw are routinely running out of space. These hosts have had storage added, which in turn has been eaten up. The vast majority of logs on theses hosts come from swift hosts, and the volume is probably not particularly useful.
We should do one, some or all of the following:
- evaluate the retention of these logs (or all logs, if necessary - ideally not) - currently glob-based/per-host retention isn't something we can easily do so this would require a bit of work
- sample these logs
- filter the logs before sending (we already attempt to do some different filtering (1, 2) , which gives some precedent even if this isn't the same problem
