+1

+1

+11

@lbowmaker now that we are on the other side of the Oozie migration can we prioritize for the next sprint?

@odimitrijevic sure! I added to this current sprint (it was already estimated). If someone has time this sprint they can work on it or it will rollover to next.

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/417

Gitlab linting in CI

aqu opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/418

Format the code following implementation of new linting rules

I've a MR with:

• It adds linting to the wmf_airflow_common & analytics_test folders.
• It proposes some Python code autoformatting tools (black & isort). They are also used for lining.
• It adds Dockerfile linting in CI.
• It marks some unit tests as slow and not being run locally. (HTTP get on a server)
• The Gitlab CI pipelines are using the trusted workers provided by Releng. Then we can stop maintaining our custom Docker worker.
• It's using Kokkuri to launch buildctl to build container images.
• The built environment is cached in a container image. That makes the running time shorter for us for linting & unit tests:
  • each lint check now runs in 6 seconds in place of 6minutes
  • pytest now runs in 50s in place of 7minutes
• It uses lib mamba to resolve Conda dependencies.
• Bump from buster to bullseye for the building image.
• Fix Python version to ~3.10.
• Moves most of the project configuration from setup.cfg to pyproject.toml.

Remaining questions:

• Should we merge those CI pipeline changes to make it the standard in workflow-utils?
• Should we update conda-analytics CI?

Documentation added here: https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow/Developer_guide