Post Gravy 'tinkering' we now get the same Adyen audit report twice - each of these is 50MB
-rw-r--r-- 1 jenkins www-data 55164053 Feb 27 02:08 settlement_detail_report_batch_1152.csv -rw-r--r-- 1 jenkins www-data 55164053 Feb 27 08:26 settlement_detail_report_2026-02-26.csv
If I gzip they go down to 9MB
I'm a bit scared not to keep both of them cos I'm never quite sure when Gravy will cause unexpected change & wasting disk seems less bad than missing out on files (e.g if we stopped receiving the one we had decided we could delete). However I think we could treat one as 'ignored' which seems easier to recover from
There is also a 3rd copy of one of these in gravy/ignored which is the same as the 2 above but with slightly different column names.
Note these are currently weekly files & the peak size is about 5 times this
Easiest option is just to gzip files when they are moved to 'completed' - I already did similar here - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1247663