Maniphest T392698

Refine to Hive with Airflow – Post-Migration Cleanup
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Antoine_Quhen
	Apr 25 2025, 3:26 PM

Tags

Referenced Files

None

Subscribers

Description

Following the migration, perform the following cleanup tasks after the transition period (~3 weeks, including the first day of a month):

Stop the old Refine systemd jobs (refine_event & refine_eventlogging_legacy) + the refine_monitor jobs.
Drop event_systemd db including hdfs data
~~Remove the additional generation of Canary events now made redundant by the new pipeline.~~ We don't change it now.
Drop diff task in refine_to_hive_hourly (+hdfs dependencies)

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	analytics: Refine remove systemd job	operations/puppet	production	+2 -2

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	analytics-test: Refine remove diff	repos/data-engineering/airflow-dags!1568	aqu	T369845_analytics_test_refine_remove_diff	main
	analytics-test: Refine - Remove diff with legacy partition	repos/data-engineering/airflow-dags!1559	aqu	T369845_refine_analytics_test_post_migration_fixes	main

Customize query in GitLab

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T307500 Airflow Hackathon (May 2022)
		Open		None	T307505 Refine jobs should be scheduled by Airflow
		Resolved		Antoine_Quhen	T369845 [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment
		Resolved		Antoine_Quhen	T392698 Refine to Hive with Airflow – Post-Migration Cleanup

Event Timeline

Antoine_Quhen created this task.Apr 25 2025, 3:26 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 25 2025, 3:26 PM

Antoine_Quhen added a parent task: T369845: [Refine Refactoring] Refine jobs should be scheduled by Airflow: deployment.Apr 25 2025, 3:26 PM

Peachey88 added a project: Data-Engineering.Apr 25 2025, 10:51 PM

Ahoelzl moved this task from Incoming (new tickets) to Q4 2025 April 1st - June 30th on the Data-Engineering board.Apr 29 2025, 5:37 PM

Ahoelzl edited projects, added Data-Engineering (Q4 2025 April 1st - June 30th); removed Data-Engineering.

Ahoelzl assigned this task to Antoine_Quhen.Jul 15 2025, 12:00 AM

Ahoelzl removed Antoine_Quhen as the assignee of this task.

Ahoelzl assigned this task to Antoine_Quhen.Jul 15 2025, 12:03 AM

Antoine_Quhen changed the task status from Open to In Progress.Jul 18 2025, 1:01 PM

Antoine_Quhen updated the task description. (Show Details)

Antoine_Quhen moved this task from Next Up to In progress on the Data-Engineering (Q4 2025 April 1st - June 30th) board.

Ahoelzl edited projects, added Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed Data-Engineering (Q4 2025 April 1st - June 30th).Jul 25 2025, 10:56 PM

Ahoelzl moved this task from Next Up to In progress on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.

aqu updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1568

analytics-test: Refine remove diff

aqu updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1559

analytics-test: Refine - Remove diff with legacy partition

aqu closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1559

analytics-test: Refine - Remove diff with legacy partition

Change #1180149 had a related patch set uploaded (by Aqu; author: Aqu):

[operations/puppet@production] analytics: Refine remove systemd job

https://gerrit.wikimedia.org/r/1180149

gerritbot added a project: Patch-For-Review.Aug 19 2025, 2:03 PM

Antoine_Quhen updated the task description. (Show Details)Aug 19 2025, 3:22 PM

aqu merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1568

analytics-test: Refine remove diff

Antoine_Quhen moved this task from In progress to In Review on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 21 2025, 1:53 PM

Change #1180149 merged by Stevemunene:

[operations/puppet@production] analytics: Refine remove systemd job

https://gerrit.wikimedia.org/r/1180149

Antoine_Quhen moved this task from In Review to Done on the Data-Engineering (Q1 FY25/26 July 1st - September 30th) board.Aug 22 2025, 12:09 PM

Ahoelzl closed this task as Resolved.Sep 24 2025, 9:37 PM