This should improve considerably the amount of things downloaded during CI checks and reduce the runs, specially for golanglintci that seem to be extra heavy on the network (and extra slow).
Description
Description
Event Timeline
Comment Actions
dcaro opened https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/32
precommit: add code to open MRs to update pre-commit
Comment Actions
dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/32
precommit: add code to open MRs to update pre-commit
Comment Actions
Hmm, I'm not 100% convinced of the current approach, it will still need to download stuff when running golangci-lint (~1G for builds-api):
dcaro@urcuchillay$ podman run --rm --volume $PWD:/src --userns=keep-id -ti d4bad7b7cc324fbfee2776b391f50b9842bef118eb570e356470d90248958217 bash dcaro@93f26ec766a1:/$ cd src dcaro@93f26ec766a1:/src$ du -hs /.cache 1007M /.cache dcaro@93f26ec766a1:/src$ pre-commit run -a trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...............................................................Passed check that scripts with shebangs are executable..........................Passed check that executables have shebangs.....................................Passed check for merge conflicts................................................Passed golangci-lint-full.......................................................Passed shellcheck...............................................................Passed dcaro@93f26ec766a1:/src$ du -hs /.cache 2.2G /.cache
I'll go with this as it cuts that in half, but still seems like too much, will investigate other ways to improve it too.
Comment Actions
dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/31
py3.11-bookworm-tox: add pre-commit caching
Comment Actions
The first run was way faster than expected (<2min):
https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/jobs/240230
Nice, let's see how it works during regular runs.
Comment Actions
Note also T362314: [cicd,infra] pre-cache all the pre-commit hooks, that is impacting the times to build things.
Anyhow, this has been alleviated in two ways:
- Creating a ci runner image that pre-caches the pre-commit tools
- Adding a go cache that is shared between the runners (so after the first run, it has to download only new stuff)
That should help though getting blocked out of github would still prevent the builds.