We have been using run_dev_instance.sh to spin up airflow instances for development purposes for a while, but with the recent move to kubernetes we aim to switch to Airflow devenvs in Kubernetes.
The run_dev_instance.sh instances had some limits in using and setting up and the airflowdevenvs address this and offer a much smoother UX. The main limiation however was the fact that there's only one process to do the job of the worker and the scheduler’s job.
These are the steps we shall follow to migrate the users to the new airflowdevevns and create awareness of the same.
Steps
- Pre-Migration Planning
- Identify all active users running run_dev_instance.sh
- Audit current user roles and permissions to map them to equivalents.
- User Migration
- Assign appropriate roles and permissions
- Migrate relevant user-specific data (if applicable)
- Perform verification testing
- Send instructions to users
- Training and Enablement
- Create and distribute user training materials (PDFs, videos, quick-start guides)
- Schedule live training sessions or webinars
- Offer drop-in Q&A office hours during transition period
- support/helpdesk via data-platform-sre slack and irc channels for user questions
- Post-Migration Support
- Monitor user activity and feedback in airflow devenvs
- Log and resolve user-reported issues
- Conduct a satisfaction survey two weeks post-migration
- Provide follow-up sessions as needed
Success criteria
- All users migrated to airflow devenvs
- All relevant user data transferred
- Training materials (guide, video, FAQ)
- Completion report with feedback
- Point the run_dev_instance.sh documentation to airflow devenvs/
- Sunset the rundev instances
We are done with the migration and post migration reports and support shall be ongoing, we are ready to sunset the rundev instances on stat machines.
Important links
Airflow Devenvs
Airflow Developer guide
Airflow Developer guide on kubernetes