It takes too much time to get feedback from the GitLab CI pipelines. Developers are waiting for CI confirmations to merge or deploy. Also, not everybody set up a development environment to run the test suite locally. Thus they rely on the CI.
Example: data-engineering/airflow-dags is a python project, yet running the test suite (including linting) is taking something like 8 minutes. https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/pipelines
The CI is also critical for repositories in charge of building artifacts (e.g. https://gitlab.wikimedia.org/repos/data-engineering/conda-base-env ), where iteration over the CI is key.
- Use custom Docker images. Currently, we are using WMF Debian. The ideal image may have:
- already installed apt packages, conda
- already set up a conda environment (Especially useful for big pip packages like Pyspark)
- Pin dependencies https://phabricator.wikimedia.org/T309046
- Tweak pytest configuration? (tests suit is running in 4s locally, and in 1.5min in CI)