Goal: To have CI/CD workflow available on Airflow's shared infrastructure.
User Story: As as Data Engineering, Pipeline Engineer or DevOps Engineer I want an automated way to quickly test, build and deploy my data pipelines to our shared infrastructure so that I can understand and quickly fix errors, know that deployment will be safe and iterate quickly on data projects.
Done Means:
- Have an automated CI/CD process implemented that includes:
- Style Guide Enforcement Tests
- Test for DAG Import Errors
- Run Any Required Unit Tests
- Test DAGs and Artifacts on input and reference data [Stretch]
- Clear and understandable reporting of errors [Stretch]
- A way to deploy a DAG to a production environment once tests have passed [Stretch]
- Have Documentation that includes:
- Developer/Admin-focused documentation on Wikitech
- User-focused documentation on Wikitech