Page MenuHomePhabricator

User Migration from Run dev instances to airflow devenv.
Closed, ResolvedPublic

Description

We have been using run_dev_instance.sh to spin up airflow instances for development purposes for a while, but with the recent move to kubernetes we aim to switch to Airflow devenvs in Kubernetes.
The run_dev_instance.sh instances had some limits in using and setting up and the airflowdevenvs address this and offer a much smoother UX. The main limiation however was the fact that there's only one process to do the job of the worker and the scheduler’s job.

These are the steps we shall follow to migrate the users to the new airflowdevevns and create awareness of the same.

Steps

  1. Pre-Migration Planning
  2. Identify all active users running run_dev_instance.sh
  3. Audit current user roles and permissions to map them to equivalents.
  1. User Migration
  2. Assign appropriate roles and permissions
  3. Migrate relevant user-specific data (if applicable)
  4. Perform verification testing
  5. Send instructions to users
  1. Training and Enablement
  2. Create and distribute user training materials (PDFs, videos, quick-start guides)
  3. Schedule live training sessions or webinars
  4. Offer drop-in Q&A office hours during transition period
  5. support/helpdesk via data-platform-sre slack and irc channels for user questions
  1. Post-Migration Support
  2. Monitor user activity and feedback in airflow devenvs
  3. Log and resolve user-reported issues
  4. Conduct a satisfaction survey two weeks post-migration
  5. Provide follow-up sessions as needed

Success criteria

  • All users migrated to airflow devenvs
  • All relevant user data transferred
  • Training materials (guide, video, FAQ)
  • Completion report with feedback
  • Point the run_dev_instance.sh documentation to airflow devenvs/
  • Sunset the rundev instances

We are done with the migration and post migration reports and support shall be ongoing, we are ready to sunset the rundev instances on stat machines.
Important links
Airflow Devenvs
Airflow Developer guide
Airflow Developer guide on kubernetes

Event Timeline

BTullis triaged this task as Medium priority.Sep 23 2025, 9:26 AM
BTullis moved this task from Incoming to Epics on the Data-Platform-SRE board.

All users have migrated away from run_dev_instance.sh and are using the new airflow dev envs.

Change #1202050 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] stat: Remove the airflow package from stat hosts

https://gerrit.wikimedia.org/r/1202050

Change #1202050 merged by Gehel:

[operations/puppet@production] stat: Remove the airflow package from stat hosts

https://gerrit.wikimedia.org/r/1202050

I think we're good on this, but a review from someone more familiar would be welcomed.