Change Details

We have been using `run_dev_instance.sh` to spin up airflow instances for development purposes for a while, but with the recent move to kubernetes we aim to switch to Airflow devenvs in Kubernetes. The `run_dev_instance.sh` instances had some limits in using and setting up and the airflowdevenvs address this and offer a much smoother UX. The main limiation however was the fact that there's only one process to do the job of the worker and the scheduler’s job. These are the steps we shall follow to migrate the users to the new airflowdevevns and create awareness of the same. Steps 1. Pre-Migration Planning [] Identify all active users running `run_dev_instance.sh` [] Audit current user roles and permissions to map them to equivalents. 2. User Migration [] Assign appropriate roles and permissions [] Migrate relevant user-specific data (if applicable) [] Perform verification testing [] Send instructions to users 3. Training and Enablement [] Create and distribute user training materials (PDFs, videos, quick-start guides) [] Schedule live training sessions or webinars [] Offer drop-in Q&A office hours during transition period [] support/helpdesk via data-platform-sre slack and irc channels for user questions 4. Post-Migration Support [] Monitor user activity and feedback in airflow devenvs [] Log and resolve user-reported issues [] Conduct a satisfaction survey two weeks post-migration [] Provide follow-up sessions as needed Success criteria [] All users migrated to airflow devenvs [] All relevant user data transferred [] Training materials (guide, video, FAQ) [] Completion report with feedback [] Pint the `run_dev_instance.sh` documentation to airflow devenvs/ Important links [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv | Airflow Devenvs ]] [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide | Airflow Developer guide ]] [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide#On_Kubernetes | Airflow Developer guide on kubernetes ]]

We have been using `run_dev_instance.sh` to spin up airflow instances for development purposes for a while, but with the recent move to kubernetes we aim to switch to Airflow devenvs in Kubernetes. The `run_dev_instance.sh` instances had some limits in using and setting up and the airflowdevenvs address this and offer a much smoother UX. The main limiation however was the fact that there's only one process to do the job of the worker and the scheduler’s job. These are the steps we shall follow to migrate the users to the new airflowdevevns and create awareness of the same. Steps 1. Pre-Migration Planning [x] Identify all active users running `run_dev_instance.sh` [x] Audit current user roles and permissions to map them to equivalents. 2. User Migration [x] Assign appropriate roles and permissions [x] Migrate relevant user-specific data (if applicable) [x] Perform verification testing [x] Send instructions to users 3. Training and Enablement [] Create and distribute user training materials (PDFs, videos, quick-start guides) [] Schedule live training sessions or webinars [] Offer drop-in Q&A office hours during transition period [] support/helpdesk via data-platform-sre slack and irc channels for user questions 4. Post-Migration Support [] Monitor user activity and feedback in airflow devenvs [] Log and resolve user-reported issues [] Conduct a satisfaction survey two weeks post-migration [] Provide follow-up sessions as needed Success criteria [] All users migrated to airflow devenvs [] All relevant user data transferred [] Training materials (guide, video, FAQ) [] Completion report with feedback [] Pint the `run_dev_instance.sh` documentation to airflow devenvs/ Important links [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv | Airflow Devenvs ]] [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide | Airflow Developer guide ]] [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide#On_Kubernetes | Airflow Developer guide on kubernetes ]]

We have been using `run_dev_instance.sh` to spin up airflow instances for development purposes for a while, but with the recent move to kubernetes we aim to switch to Airflow devenvs in Kubernetes. The `run_dev_instance.sh` instances had some limits in using and setting up and the airflowdevenvs address this and offer a much smoother UX. The main limiation however was the fact that there's only one process to do the job of the worker and the scheduler’s job. These are the steps we shall follow to migrate the users to the new airflowdevevns and create awareness of the same. Steps 1. Pre-Migration Planning [x] Identify all active users running `run_dev_instance.sh` [x] Audit current user roles and permissions to map them to equivalents. 2. User Migration [x] Assign appropriate roles and permissions [x] Migrate relevant user-specific data (if applicable) [x] Perform verification testing [x] Send instructions to users 3. Training and Enablement [] Create and distribute user training materials (PDFs, videos, quick-start guides) [] Schedule live training sessions or webinars [] Offer drop-in Q&A office hours during transition period [] support/helpdesk via data-platform-sre slack and irc channels for user questions 4. Post-Migration Support [] Monitor user activity and feedback in airflow devenvs [] Log and resolve user-reported issues [] Conduct a satisfaction survey two weeks post-migration [] Provide follow-up sessions as needed Success criteria [] All users migrated to airflow devenvs [] All relevant user data transferred [] Training materials (guide, video, FAQ) [] Completion report with feedback [] Pint the `run_dev_instance.sh` documentation to airflow devenvs/ Important links [[ https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv | Airflow Devenvs ]] [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide | Airflow Developer guide ]] [[ https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide#On_Kubernetes | Airflow Developer guide on kubernetes ]]