When working on writing/modifying an [Airflow](https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow) DAG, it has become customary to rely on [development environments](https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide#Development_instance) (also called airflow devenvs) to perform end-to-end testing of the patch being worked on.
These instances can be created and deleted at will, and will be configured as close as the original instance. For example, when creating an airflow devenv when working on a DAG under the [test_k8s](https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/tree/main/test_k8s?ref_type=heads) root folder, the [configuration](https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/903f9010bd134e5210bd12b24280c4c5989814d4/helmfile.d/dse-k8s-services/airflow-test-k8s/values-production.yaml) for the airflow-test-k8s instance will be used, in order for the devenv instance to behave as close as the "real" one.
One recurring issue that we've seen is the time it takes to [serialize](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/dag-serialization.html) all dags defined under that root folder, especially for instances with [[ https://thanos.wikimedia.org/graph?g0.expr=avg(airflow_dagbag_size)%20by%20(kubernetes_namespace)&g0.tab=0&g0.stacked=0&g0.range_input=1h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.engine=prometheus&g0.analyze=0&g0.tenant) | a lot of DAGs ]] defined.
To ensure the fastest feedback loop possible, we'd need to introduce a flag to the `airflow-devenv create` [command](https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv/-/blob/main/airflow_devenv/cli.py?ref_type=heads#L70) that would specify the path of the DAG currently being worked on, which would then be injected as a helm value, and used by the [airflow chart](https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/903f9010bd134e5210bd12b24280c4c5989814d4/charts/airflow/Chart.yaml) to restrict the set of files being pulled by `gitsync` when performing a [sparse checkout](https://git-scm.com/docs/git-sparse-checkout).
Ideally, we could also infer this value at runtime by listing the files being modified in the branch being passed to `airflow-devenv create --branch` and automatically do The Right Thing ™ . If we're going this way, we may even think about inferring the value of the `airflow-devenv create --dags-folder` argument the same way.
**Definition of done:**
- [x] A new `airflow-devenv` version is deployed where a user is given the option to specify the DAGs they'd like to be serialized
- [ ] A demo is communicated broadly (either pre-recorded or live) to the DPE population
- [x] Instructions are added to the `airflow-devenv` [README](https://gitlab.wikimedia.org/repos/data-engineering/airflow-devenv/-/blob/main/README.md) and [wiki](https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Developer_guide) as to how to test changes in the CLI before releasing them as a new deb version
**Additional reading:**
- https://wikitech.wikimedia.org/wiki/Kubernetes
- https://wikitech.wikimedia.org/wiki/Helmfile
- https://airflow.apache.org/docs/
- https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes