Recent incidents of buildkitd's storage volume filling up
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	dancy
	May 22 2025, 11:38 PM

Description

Several jobs have failed recently due to a buildkitd volume being full. This isn't something that we've seen regularly.

https://gitlab.wikimedia.org/repos/sre/libvmod-wmfuniq/-/jobs/517608
https://gitlab.wikimedia.org/repos/releng/scap/-/jobs/515939
https://gitlab.wikimedia.org/repos/releng/scap/-/jobs/515937

A quick way to prune all bulidkitd volumes:

for n in $(seq 0 2); do kubectl -n gitlab-runner exec buildkitd-$n -- buildctl --addr localhost:1234 prune; done

This assumes that your kubectl config is pointing to the gitlab-cloud-runner cluster.

I set up this dashboard for monitoring:
https://grafana.cloud.releng.team/d/demojg23eidq8c/buildkitd

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	.gitlab-ci.yml: Add wmcs tag to publish-build job	repos/sre/wikitech-static-docker!3	dancy	main-I5d769213f9b107da1513eaac703ea90102b250d4	main

Customize query in GitLab

Related Objects

Mentioned In: T393731: buildkit v0.21.1 released

Event Timeline

dancy created this task.May 22 2025, 11:38 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2025, 11:38 PM

dancy mentioned this in T393731: buildkit v0.21.1 released.May 22 2025, 11:40 PM

On a 7-day view, https://grafana.cloud.releng.team/d/demojg23eidq8c/buildkitd shows regular daily spikes where a buildkitd volume maxes out, starting 2025-05-19. There must be a new CI job doing something interesting/immense.

dancy added a project: GitLab (CI & Job Runners).May 22 2025, 11:45 PM

I think this is related to the publish-build job of https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/blob/main/.gitlab-ci.yml?ref_type=heads

In T395097#10859698, @dancy wrote:

I think this is related to the publish-build job of https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/blob/main/.gitlab-ci.yml?ref_type=heads

Tagging in cloud-services-team folks for awareness/comment. It makes sense that that job is eating space if it's dumping the entirety of wikitech (but, honestly, unsure what the size we should expect there).

Would isolating that job (possibly on a WMCS runner) help/make sense?

In T395097#10860917, @thcipriani wrote:

In T395097#10859698, @dancy wrote:

I think this is related to the publish-build job of https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/blob/main/.gitlab-ci.yml?ref_type=heads

Tagging in cloud-services-team folks for awareness/comment. It makes sense that that job is eating space if it's dumping the entirety of wikitech (but, honestly, unsure what the size we should expect there).

Would isolating that job (possibly on a WMCS runner) help/make sense?