The hdfs path /wmf/data/discovery/transfer_to_es is populated by the transfer_to_es job that can be run hourly and thus might create many folders and files. The oldest snapshot is 20200105.
It might make sense to have an automated cleanup process for this dataset, the retention data is yet to be defined (60 days?).
AC:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Script to cleanup transfer_to_es folder in hdfs | repos/search-platform/discolytics!6 | ebernhardson | work/ebernhardson/cleanup-transfer-to-es | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | EBernhardson | T323616 Cleanup the /wmf/data/discovery/transfer_to_es folder in hdfs | |||
| Resolved | bking | T327970 Create airflow v2 instance and supporting repos for search platform | |||
| Resolved | EBernhardson | T331580 Fix permissions in hdfs://analytics-hadoop/wmf/data/discovery |
ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/6
Script to cleanup transfer_to_es folder in hdfs
I have this written up and have run it manually, so the existing data has been cleaned up. For automated runs i'm delaying until we have the new airflow instance up and running, with the intent of scheduling it in airflow 2
ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/6
Script to cleanup transfer_to_es folder in hdfs
Running this weekly is part of https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/241 done for T329870