Page MenuHomePhabricator

Add Python Linter Checks to CI
Closed, ResolvedPublic3 Estimated Story Points


This task is to add Python Linter to the Airflow CI to ensure consistent formatting/styling and remove need to flag violations in code reviews.

Event Timeline

EChetty set the point value for this task to 3.

@lbowmaker now that we are on the other side of the Oozie migration can we prioritize for the next sprint?

lbowmaker edited projects, added Data Pipelines (Sprint 14); removed Data Pipelines.

@odimitrijevic sure! I added to this current sprint (it was already estimated). If someone has time this sprint they can work on it or it will rollover to next.

I've a MR with:

  • It adds linting to the wmf_airflow_common & analytics_test folders.
  • It proposes some Python code autoformatting tools (black & isort). They are also used for lining.
  • It adds Dockerfile linting in CI.
  • It marks some unit tests as slow and not being run locally. (HTTP get on a server)
  • The Gitlab CI pipelines are using the trusted workers provided by Releng. Then we can stop maintaining our custom Docker worker.
  • It's using Kokkuri to launch buildctl to build container images.
  • The built environment is cached in a container image. That makes the running time shorter for us for linting & unit tests:
    • each lint check now runs in 6 seconds in place of 6minutes
    • pytest now runs in 50s in place of 7minutes
  • It uses lib mamba to resolve Conda dependencies.
  • Bump from buster to bullseye for the building image.
  • Fix Python version to ~3.10.
  • Moves most of the project configuration from setup.cfg to pyproject.toml.

Remaining questions:

  • Should we merge those CI pipeline changes to make it the standard in workflow-utils?
  • Should we update conda-analytics CI?