Between 18:37 UTC on 8 July and 01:24 UTC on 9 July 2026, jobs on the wmcs-tagged runners became roughly 6.5 times slower at every BuildKit step that touches the filesystem, while identical builds on the Digital Ocean Kubernetes runners were unaffected. For function-orchestrator this took the test job from a median of 4.1 minutes to 27.3, and later to the one-hour job timeout. CPU throughput is unchanged, so this looks like per-file latency in whatever backs the BuildKit snapshotter and Docker storage on that fleet.

Technical notes

The window is somewhat narrow. The last fast run was job 889807 at 8 July 18:37 UTC in 1.8 minutes; the first slow one was job 890058 at 9 July 01:24 UTC in 27.3 minutes. Nothing ran in between from our end, so I have no more precise data. The two distributions do not overlap at all: 8 July's slowest test was 6.3 minutes, 9 July's fastest was 12.1.

The split is by fleet, not by job. Pulling every test, lint, test-memcached and build-orchestrator-for-catalyst job in repos/abstract-wiki/wikifunctions/function-orchestrator from 1 June to 30 July via the API (sorry, Claude) gives 262 samples per job with no fleet mixing, 175 before the cutover and 87 after. Median durations in minutes, before then after: test on runner 1507 (Wikimedia Cloud Services) 4.1 then 27.3, a factor of 6.7; build-orchestrator-for-catalyst on runner 1507, 3.0 then 18.8, a factor of 6.2; lint on the Digital Ocean Kubernetes runners 1.1 then 1.1; test-memcached on the same, 0.6 then 0.6. The Digital Ocean jobs did not move by a measurable amount.

This is close to a controlled comparison. lint and test are the same Blubber build, both includes: [build], so identical apt-get install, identical npm install and identical build-context copy, differing only in entrypoint. The one other difference is that test carries tags: [ wmcs ] and lint does not, so lint lands on the Digital Ocean fleet via run_untagged. Same repository, same commits, same days, same layer chain, opposite outcomes. Nothing in the project can produce that split, and we confirmed the project's own inputs are flat across the window: .pipeline/blubber.yaml byte-identical, lockfile down from 894 to 878 packages, build context 877 blobs / 11.61 MiB to 887 / 11.69 MiB.

On the mechanism, comparing cold builds on both sides, amd64 only: (getent group "65533" || groupadd ...) && (getent passwd "65533" || useradd ...) went from 3.1s in job 885870 to 259.0s in job 911227, a factor of 84; a two-file copy of package.json and package-lock.json from 3.1s to 293.5s, a factor of 95; the build-context copy of roughly 15 MiB across 2650 files from 21.3s to 651.0s; and npm install of the full 795-package tree from 16.5s to 411.5s. apt-get update && apt-get install moved much less, 28.1s to 170.7s, and it is the one step dominated by downloading rather than by file operations. Individual image layers took 158.5s and 124.0s to extract.

Two observations localise this to storage rather than CPU, memory or network bandwidth. The Mocha suite inside the same container is unaffected: 951 tests in 24s on 6 July against 1037 tests in 26s on 29 July, i.e. unchanged per-test throughput. And the slowdown factor tracks how metadata-bound each operation is, close to monotonically, with useradd at 84x and apt-get at 6x.

It is also visible on a single runner, which rules out one unhealthy host: the build-context copy of the same repository took 5.5s in job 882187 on 2 July and 651.0s in job 911227 on 29 July, both on runner-1040. That step is never served from cache while the source changes, so it is directly comparable. The only caveat is that a Cloud VPS instance could in principle have been rebuilt under the same runner name in between.

Separately, and possibly part of the same event, native arm64 build capacity appears to have been lost. In job 885870 the arm64 npm install took 23.3s, faster than the amd64 one at 16.5s-scale, which QEMU emulation cannot do. By job 910963 the same step took 1081.5s, a factor of 46 and consistent with a fall back to emulation.

I will mitigate on our side by moving these jobs to the memory-optimized Digital Ocean runners and by dropping arm64 from pre-merge builds, so this isn't blocking us. However, I'm filing because the regression affects anything else still using the wmcs fleet, and because I could not find an existing task for it.