pywikibot/core pre-commit cache has a very large cache
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Fri, Apr 3, 9:24 AM

Description

I have noticed a build of the Jenkins job pywikibot-core-tox that spent a lot of time syncing the cache:

00:00:01.240 Syncing...
00:13:01.139 
00:13:01.139 Done

That is 13 minutes to restore pre-commit caches.

The duration has kept growing, here over a year (Grafana):

pywikibot-core-tox_runtime.png (565×1 px, 57 KB)

The namespace is CASTOR_NAMESPACE="pywikibot-core/master/pywikibot-core-tox", looking at the Castor instance:

ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo du -m -d1 /srv/castor/pywikibot-core/master/pywikibot-core-tox
1182	/srv/castor/pywikibot-core/master/pywikibot-core-tox/pip
60	/srv/castor/pywikibot-core/master/pywikibot-core-tox/virtualenv
11446	/srv/castor/pywikibot-core/master/pywikibot-core-tox/pre-commit
12688	/srv/castor/pywikibot-core/master/pywikibot-core-tox

pre-commit stores caches there that keeps accumulating. Some of the cache directories take 189MB, they a couple pyenv:

94	./py_env-python3.10
95	./py_env-python3.13

I don't think we should store them, or at least they should not be accumulating them? For the tox/pip part, we only save and restore the pip cache, the venv are reinstalled from scratch on each build.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
tests: run garbage collection after lint tests	pywikibot/i18n	master	+1 -0
tests: run garbage collection after lint tests	pywikibot/bots/xqbot	master	+1 -0
pre-commit: clean unused cached repos	pywikibot/core	master	+1 -0

Customize query in gerrit

Related Objects

Mentioned In: T422266: doctest fails due to server maxlag
Mentioned Here: T403266: CASTOR sync for pywikibot-core-tox can take over 5 minutes in some cases

Event Timeline

hashar created this task.Fri, Apr 3, 9:24 AM

Restricted Application added subscribers: pywikibot-bugs-list, Aklapper. · View Herald TranscriptFri, Apr 3, 9:24 AM

hashar updated the task description. (Show Details)Fri, Apr 3, 9:25 AM

Mentioned in SAL (#wikimedia-releng) [2026-04-03T09:26:58Z] <hashar> integration: nuked pywikibot/core pre-commit cache # T422242

I have removed the whole cache since restoring it took 13 minutes. But we still need to address the root cause.

hashar updated the task description. (Show Details)Fri, Apr 3, 9:32 AM

hashar attached a referenced file: F74891436: pywikibot-core-tox_runtime.png. (Show Details)

Xqt subscribed.Fri, Apr 3, 1:26 PM

See also: T422242, T403266

I remember about that T403266 I assumed it was simply that the builds were serialized.

I know nothing about pre-commit but their doc at https://pre-commit.com/#managing-ci-caches states:

Managing CI Caches

pre-commit by default places its repository store in ~/.cache/pre-commit -- this can be configured in two ways:

PRE_COMMIT_HOME: if set, pre-commit will use that location instead.
XDG_CACHE_HOME: if set, pre-commit will use $XDG_CACHE_HOME/pre-commit following the XDG Base Directory Specification.

And since the CI image set XDG_CACHE_HOME it writes cache there and they end up accumulating.

The question is why there were so many repoXXXXX entries there.

At https://pre-commit.com/#pre-commit-gc there is:

pre-commit gc [options]

Clean unused cached repos.

pre-commit keeps a cache of installed hook repositories which grows over time. This command can be run periodically to clean out unused repos from the cache directory.

Maybe that is the command that should be run after the lint?

Change #1267859 had a related patch set uploaded (by Hashar; author: Hashar):

[pywikibot/core@master] pre-commit: clean unused cached repos

https://gerrit.wikimedia.org/r/1267859

gerritbot added a project: Patch-For-Review.Fri, Apr 3, 2:34 PM

Change #1267859 merged by Xqt:

[pywikibot/core@master] pre-commit: clean unused cached repos

https://gerrit.wikimedia.org/r/1267859

In T422242#11785662, @hashar wrote:

Maybe that is the command that should be run after the lint?

I had a similar idea using either garbage collection or clean the pre-commit files after the tests. In the meantime, I am pretty sure that pre-commit hooks were causing this issue. I merged your patch, so we can see whether it works as expected.

Change #1267896 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/i18n@master] tests: run garbage collection after lint tests

https://gerrit.wikimedia.org/r/1267896

Change #1267897 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/bots/xqbot@master] tests: run garbage collection after lint tests

https://gerrit.wikimedia.org/r/1267897

@hashar: Thanks a lot for investigation and to finding this out. I guess this is solved, see
https://integration.wikimedia.org/ci/job/pywikibot-i18n-tox/230/console and
https://integration.wikimedia.org/ci/job/pywikibot-xqbot-tox/127/console
where some repos are removed with garbage collection.