The vast majority of the space is taken up by /srv/airflow-analytics

Within there are 73.9 GB of logs.

The recent scheduler logs are all about 1.4 GB is size and don't seem to be rotated very frequently.

I'll see what I can delete.

There is a logrotate fragment for airflow_analytics, but I don't think it's catching the files that are in their own subdirectories per name;.

# logrotate(8) config for airflow_analytics_clean_logs

/var/log/airflow_analytics_clean_logs/*.log {
    daily
    copytruncate
    missingok
    compress
    delaycompress
    notifempty
    rotate 15
    size 256M
}

Mentioned in SAL (#wikimedia-analytics) [2023-06-13T19:03:42Z] <btullis> freeing up space in /srv on an-launcher1002 with btullis@an-launcher1002:/srv/airflow-analytics/logs/scheduler$ find -maxdepth 1 -type d -mtime +15 -print0 | xargs -0 sudo rm -rf for T339002

We should check log rotation settings on airflow instances, to make sure they are correct and working as we expect.