This task is to add Python Linter to the Airflow CI to ensure consistent formatting/styling and remove need to flag violations in code reviews.
I've a MR with:
- It adds linting to the wmf_airflow_common & analytics_test folders.
- It proposes some Python code autoformatting tools (black & isort). They are also used for lining.
- It adds Dockerfile linting in CI.
- It marks some unit tests as slow and not being run locally. (HTTP get on a server)
- The Gitlab CI pipelines are using the trusted workers provided by Releng. Then we can stop maintaining our custom Docker worker.
- It's using Kokkuri to launch buildctl to build container images.
- The built environment is cached in a container image. That makes the running time shorter for us for linting & unit tests:
- each lint check now runs in 6 seconds in place of 6minutes
- pytest now runs in 50s in place of 7minutes
- It uses lib mamba to resolve Conda dependencies.
- Bump from buster to bullseye for the building image.
- Fix Python version to ~3.10.
- Moves most of the project configuration from setup.cfg to pyproject.toml.
- Should we merge those CI pipeline changes to make it the standard in workflow-utils?
- Should we update conda-analytics CI?