Background
The mwext-codehealth-master-non-voting job runs in the postmerge pipeline after every merge to a MediaWiki extension or skin.
It is non-voting and runs after the merge, so no developer is ever waiting for it, not for merging and not for a patch. Still, every successful
run kicks off a castor-save-workspace-cache build to sync its cache back to castor.
Castor has one executor, so saves run one at a time. In June it ran 90510 saves for a total of 157.3 hours, and 46.6% of the saves started back to back
(less than 250 ms gap), so about half the time there is a queue (statistics in https://releng-data.wmcloud.org/-/dashboards/ci-by-repo-and-job and T432688).
Most saves finish in under a second. The codehealth one takes almost a minute and runs approximately 60 times a day, so 2% of the saves stands
for 17% of castor's save time. That is close to an hour of castor time every day, spent on a cache that saves no developer feedback time:
| slice | runs | CI hours | median | p95 | p99 | max |
|---|---|---|---|---|---|---|
| all saves (June) | 90510 | 157.3 | 00:00 | 00:30 | 00:51 | 03:00 |
| triggered by mwext-codehealth-master-non-voting | 1779 | 27.4 | 00:47 | 01:51 | 02:24 | 03:00 |
Pruning the cache will not help long term since it will fill up again. Way forward is to stop saving the cache for this job and measure what happens.
Experiment
Stop the cache save for mwext-codehealth-master-non-voting. Nothing changes for any other job.
Implementation: in jjb, in the`mwext-codehealth-master-non-voting` publishers list, replace the teardown
publisher with docker-cleanup. teardown is castor-save-workspace-cache + docker-cleanup, so this only drops the save.
Duration
One week, then we look at the numbers and decide.
Metrics to collect (before/during)
- Number of castor-save-workspace-cache runs per day and their durations
- Queue/wait time for other jobs' cache saves on castor
- mwext-codehealth-master-non-voting build duration over the period
- Disk usage of castor-mw-ext-and-skins/master/mwext-codehealth-master-non-voting on castor (we will clear the cache, so this will probably be empty)
Risks
- The mwext master job will take longer to finish since builds will download node modules and sonar artifacts every time. Since the job is non-voting and postmerge, no developer waits on it. Acceptable for the duration of the experiment.
Acceptance criteria
- The change is deployed and mwext-codehealth-master-non-voting no longer triggers castor-save-workspace-cache
- The experiment has run for one week
- We have before/after numbers for saves per day, castor blocking time and the job's own run time, documented on this task
- Decision made and documented, keep the change or revert it
Revert plan
Revert the one-line change in integration/config and redeploy the job.