There's a whole lot of swift access logs on thanos-be hosts, to the point that they are filling up the root filesystem (e.g. thanos-be2004).
Most of the activity seems to be from tegola and specifically file requests from object-server
root@thanos-be2004:/srv/log/swift# fgrep -c tegola /var/log/swift/server.log 20429551 root@thanos-be2004:/srv/log/swift# wc -l /var/log/swift/server.log 34142226 /var/log/swift/server.log
Mentioned in SAL (#wikimedia-operations) [2021-12-17T16:02:32Z] <godog> root@thanos-be2004:/srv/log/swift# rm server.log.1 - T297959
Change 748150 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] swift: temp ban for tegola access logs
Change 748150 merged by Filippo Giunchedi:
[operations/puppet@production] swift: temp ban for tegola access logs
Mentioned in SAL (#wikimedia-operations) [2021-12-17T16:33:39Z] <godog> remove /var/log/swift/server.log.1 from thanos-be* - T297959
I've bandaided the immediate issue, leaving the task open since we haven't addressed the high volume of logs
Change 809966 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] swift: heavier banhammer for tegola object-server 'access logs'
Change 809966 merged by Filippo Giunchedi:
[operations/puppet@production] swift: heavier banhammer for tegola object-server 'access logs'
Change 810276 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):
[operations/puppet@production] swift: turn off uwsgi request logging
Change 810276 merged by Filippo Giunchedi:
[operations/puppet@production] swift: turn off uwsgi request logging
Mentioned in SAL (#wikimedia-operations) [2022-07-11T07:52:21Z] <godog> roll-restart swift-account swift-container across swift/thanos bullseye hosts - T297959