Page MenuHomePhabricator

quibble-composer-sqlite-php72-docker does not use cache for composer downloads
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/quibble-composer-sqlite-php72-docker/14/console

Everytime looking at the experimental sqlite job, I see that the composer install is downloading all the packages

Installing wikimedia/composer-merge-plugin (v1.4.1): [258.2MB/5.43s] Downloading[258.2MB/5.51s] (100%)[258.2MB/5.51s]

Only some packages are cached:

Installing guzzlehttp/promises (v1.3.1): [258.3MB/5.68s] Loading from cache[258.3MB/5.68s]

It is possible to cache all packages to reduce network traffic and save some run time?

Event Timeline

The cache is only populated for job triggered in gate-and-submit or postmerge. Since that is only in experimental pipeline, the cache is never populated.

I have manually triggered the job against mediawiki/core setting ZUUL_PROJECT=postmerge: https://integration.wikimedia.org/ci/job/quibble-composer-sqlite-php72-docker/15/console
And eventually:

20:40:10 Waiting for the completion of castor-save-workspace-cache
20:40:16 castor-save-workspace-cache #780358 completed. Result was SUCCESS

Which rsynced the content.

The next build https://integration.wikimedia.org/ci/job/quibble-composer-sqlite-php72-docker/16/console should thus be faster.

20:41:08 Defined: CASTOR_NAMESPACE="mediawiki-cor`aster/quibble-composer-sqlite-php72-docker"
20:41:08 Syncing...
20:41:08 rsync: failed to set times on "/cache/.": Operation not permitted (1)
20:41:10 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.2]
20:41:10 
20:41:10 Done

(the failures there can be both ignored)

Stuff loads from cache:

20:41:27 [258.0MB/3.63s]   - Installing wikimedia/composer-merge-plugin (v1.4.1): [258.1MB/3.63s] Loading from cache[258.1MB/3.63s] 
20:41:27 [258.1MB/3.63s]  Extracting archive[258.3MB/3.67s]   [merge-plugin] composer-merge-plugin installed
20:41:27 [258.3MB/3.68s]   - Installing cssjanus/cssjanus (v1.3.0): [258.3MB/3.68s] Loading from cache[258.3MB/3.68s] 
20:41:27 [258.3MB/3.68s]  Extracting archive[258.3MB/3.70s]   - Installing guzzlehttp/promises (v1.3.1): [258.3MB/3.70s] Loading from cache[258.3MB/3.70s] 
...
20:41:35 [411.9MB/11.50s] Memory usage: 411.88MB (peak: 460.56MB), time: 11.5s

Though eventually the cache would end up outdated and the job would need to be manually triggered with ZUUL_PROJECT=postmerge again :-\

A better system would be to populate the mediawiki/core on each postmerge, potentially by just polling the repository and run the population whenever something has been updated. And for extension on a daily basis or so when changes have been made to composer.json.

All of that is not ideal for sure :-\

A better solution: T147635: Investigate again a central cache for package managers