buildkit accumulates some cached data on gitlab-runner hosts. During a full disk (T326378) it was noticed that buildkit uses ~3GB+ of diskspace without a cleanup or retention period.
A cleanup policy should be implemented for buildkit before the cache uses all of the disk space.
See also T326378#8509591
Buildkit GC options are sorta mentioned in https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Gitlab runners: Use gckeepstorage buildkitd setting to manage storage | operations/puppet | production | +30 -16 |
Change 881007 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):
[operations/puppet@production] Gitlab runners: Use gckeepstorage buildkitd setting to manage storage
Change 881007 merged by Dzahn:
[operations/puppet@production] Gitlab runners: Use gckeepstorage buildkitd setting to manage storage
@dancy disk usage of buildkitd container on runner-1026.gitlab-runners.eqiad1.wikimedia.cloud went down from 3.3GB to 1MB. Mostly because it was recreated (different ID).
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE 8b031f421780 docker-registry.wikimedia.org/repos/releng/buildkit:wmf-v0.10-1 "/usr/local/bin/entr…" 15 hours ago Up 15 hours buildkitd 1.07MB (virtual 217MB)
Let keep an eye on the disk usage the next weeks. Currently only 14% (5GB) are used on /var/lib/docker on runner-1026.