Page MenuHomePhabricator

Refine to Hive with Airflow – Switch Over day scripts
Closed, ResolvedPublic

Description

We’re implementing the all-at-once switchover strategy as outlined in T369845 to transition from the event_alt database to the primary event database.

At the end of the switch the new Refine process will write in event and the legacy refine process will write in event_alt

Steps:

  1. Pause the refine systemd jobs on an-launcher1002.
  2. Update systemd jobs to write to the event_alt database via a Puppet patch.
  3. Update Airflow refine DAG to write to the event database and diff against event_alt (requires Airflow patch).
  4. Manually rerun the refine job for the hour during which the cutover happens.

This ticket is focused on the scripts we need to run the dday of the migration on prod.

  1. After a monitoring period (~1 week) of successful Airflow-based refining:
    • Ensure systemd timers for refine are absent.
    • Clean up corresponding Puppet code.
    • Drop the event_alt Hive database, including its tables and HDFS files.

Event Timeline

Antoine_Quhen renamed this task from Refine to Hive with Airflow – Switch Over plan to Refine to Hive with Airflow – Switch Over day scripts.Apr 28 2025, 2:43 PM
Antoine_Quhen updated the task description. (Show Details)