Just cleaned up some more since we hit the warning threshold again. Here are the commands/stats:

root@fran2001:/var/lib/dagster# du -sh dbt_target/
24G	dbt_target/
root@fran2001:/var/lib/dagster# find /var/lib/dagster/dbt_target/ -maxdepth 1 -type d -name my_dbt_assets-\* -mtime +29 | wc -l
440
root@fran2001:/var/lib/dagster# find /var/lib/dagster/dbt_target/ -maxdepth 1 -type d -name my_dbt_assets-\* -mtime +29 | xargs -I{} rm -rf {} 
root@fran2001:/var/lib/dagster# du -sh dbt_target/
21G	dbt_target/

I just added a cron job to run the above find/rm command every 6 hours. It'll keep the space under control to some degree but is really just a band aid while a more permanent fix is chosen. The cron job only runs on the active analytics host.

Don't know if we are doing more job runs or not, but the cron job for cleaning up /var/lib/dagster/dbt_target/ is already moot. Given that it's back up over 24G, I'm adjusting the cleanup days down to 25. That will currently have us sitting around 76% utilization.

Reached that level by trimming back one day at a time until we got well clear of the 80% threshold.

Current level:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md2 57323624 41171036 13208284 76% /

We are going to look at shifting dagster runtime files off of the root partition into /srv during the maintenance window time tomorrow.

At a minimum we need to:

• determine the new location
• stop dagster
• copy the tree from /var/lib/dagster to it's new location
• probably want to add a symlink to the new location in case folks have it hardcoded in local files
• update dagster_home in the dagster manifest or call it differently in the analytics role manifest
• update any aide exclusions to include the new location in the analytics role manifest
• start dagster
• verify it all works

Pushed up a proposed branch T392747_move_dagster to move the active dir to /srv/dagster_data. This has the added effect of moving the dagster logs/artifacts out of the dagster user's home directory. Testing will be done in VB land to make sure dagster still functions as desired with this change.