Page MenuHomePhabricator

Waiting for the completion of castor-save-workspace-cache sometimes takes almost 4 minutes for core tests
Open, Needs TriagePublic

Description

Hi @hashar do you know if there's anything we can do about that wait time? In the logs it looks like it just waits before it can starts, or do you think maybe we are missing some logs?

Screenshot 2026-03-04 at 06.08.10.png (1×3 px, 675 KB)

https://integration.wikimedia.org/ci/job/quibble-for-mediawiki-core-browser-tests-only-vendor-mysql-php83/5830/console

I can see that those jobs when it takes time ends up in the longest running jobs.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

The caches are namespaces, the build had

CASTOR_NAMESPACE=mediawiki-core/master/quibble-for-mediawiki-core-browser-tests-only-vendor-mysql-php83

That is used as path when rsyncing to integration-castor05.integration.eqiad1.wikimedia.cloud under /srv/castor.

Here is the disk usage:

ssh integration-castor05.integration.eqiad1.wikimedia.cloud 'cd /srv/castor/mediawiki-core/master/quibble-for-mediawiki-core-browser-tests-only-vendor-mysql-php83 && du -m -d1'
89	./npm
17	./composer
2	./mesa_shader_cache
1	./fontconfig
107	.

107M is not that big.

There is otherwise a mutex on the castor-save-workspace-cache job and there is thus only one build happening at any given time. The job was probably busy saving caches from a whole lot of other builds / changes. That is similar to T403266

Here's another example. The jobs takes 1 minute and the waiting time adds 37 s

Screenshot 2026-04-16 at 09.41.53.png (188×1 px, 78 KB)