⚓ T329870 Migrate drop_old_data_daily.py from airflow 1 to airflow 2

	Subject	Repo	Branch	Lines +/-
	Deploy analytics-refinery to search airflow instance	operations/puppet	production	+1 -0

Status	Assigned	Task
Resolved	Gehel	T322905 [EPIC] Upgrade Search Platform spark jobs to spark 3
Resolved	Gehel	T318414 [Tracking] Migrate Search Airflow jobs to Airflow 2 and use shared supporting code from the data engineering Airflow
Resolved	EBernhardson	T329870 Migrate drop_old_data_daily.py from airflow 1 to airflow 2

EBernhardson renamed this task from Migrate drop_old_data_daily.py from airflow 2 to airflow 3 to Migrate drop_old_data_daily.py from airflow 1 to airflow 2.Feb 16 2023, 6:45 PM

EBernhardson created this task.

EBernhardson claimed this task.Feb 21 2023, 5:04 PM

EBernhardson moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.

Change 890906 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/puppet@production] Deploy analytics-refinery to search airflow instance

https://gerrit.wikimedia.org/r/890906

gerritbot added a project: Patch-For-Review.Feb 21 2023, 9:23 PM

ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/7

Import refinery-drop-mediawiki-snapshots

Once the patch to discolytics is merged i can update the work/ebernhardson/drop-old-data-daily branch in airflow-dags with the appropriate conda environment version and create a merge request for that as well. I did a test run from stat1005 with the airflow 2 dev env and this seems to work as expected. It also needs the puppet patch merged prior to live deployment on the airflow instance.

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/7

Import refinery-drop-mediawiki-snapshots

Change 890906 merged by Ottomata:

[operations/puppet@production] Deploy analytics-refinery to search airflow instance

https://gerrit.wikimedia.org/r/890906

Maintenance_bot removed a project: Patch-For-Review.Feb 22 2023, 7:30 PM

ebernhardson opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/241

Port over search drop_old_data_daily dag

gerritbot added a project: Patch-For-Review.Feb 23 2023, 12:18 AM

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/9

drop_date_directories: Support bare dates

EBernhardson mentioned this in T323616: Cleanup the /wmf/data/discovery/transfer_to_es folder in hdfs.Feb 27 2023, 4:18 PM