Page MenuHomePhabricator

thanos-be hosts filing up root filesystem with logs
Open, Needs TriagePublic

Description

There's a whole lot of swift access logs on thanos-be hosts, to the point that they are filling up the root filesystem (e.g. thanos-be2004).

Most of the activity seems to be from tegola and specifically file requests from object-server

root@thanos-be2004:/srv/log/swift# fgrep -c tegola /var/log/swift/server.log
20429551
root@thanos-be2004:/srv/log/swift# wc -l /var/log/swift/server.log
34142226 /var/log/swift/server.log

Event Timeline

Mentioned in SAL (#wikimedia-operations) [2021-12-17T16:02:32Z] <godog> root@thanos-be2004:/srv/log/swift# rm server.log.1 - T297959

Change 748150 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] swift: temp ban for tegola access logs

https://gerrit.wikimedia.org/r/748150

Change 748150 merged by Filippo Giunchedi:

[operations/puppet@production] swift: temp ban for tegola access logs

https://gerrit.wikimedia.org/r/748150

Mentioned in SAL (#wikimedia-operations) [2021-12-17T16:33:39Z] <godog> remove /var/log/swift/server.log.1 from thanos-be* - T297959

I've bandaided the immediate issue, leaving the task open since we haven't addressed the high volume of logs

Change 809966 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] swift: heavier banhammer for tegola object-server 'access logs'

https://gerrit.wikimedia.org/r/809966

Change 809966 merged by Filippo Giunchedi:

[operations/puppet@production] swift: heavier banhammer for tegola object-server 'access logs'

https://gerrit.wikimedia.org/r/809966

Change 810276 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] swift: turn off uwsgi request logging

https://gerrit.wikimedia.org/r/810276

Change 810276 merged by Filippo Giunchedi:

[operations/puppet@production] swift: turn off uwsgi request logging

https://gerrit.wikimedia.org/r/810276

Mentioned in SAL (#wikimedia-operations) [2022-07-11T07:52:21Z] <godog> roll-restart swift-account swift-container across swift/thanos bullseye hosts - T297959