Page MenuHomePhabricator

Requesting additional Harbor / Toolforge Build Service disk quota for wd-shex-infer tool
Closed, ResolvedPublic

Description

In T320140 I’m making progress on migrating the wd-shex-infer tool off the Grid engine, but it seems that my image is now large enough that I can only fit one version of it in Harbor at the same time. This isn’t ideal – at the moment I’m still experimenting, so it’s fine to delete all the existing images before building new ones, but in future I’d like to be able to build and test a new image version while still keeping an older, known-working image around. Therefore, I’d like to request some additional image disk quota for the wd-shex-infer tool – and, if it’s not too much to ask, also for the lucaswerkmeister-test tool, where I’m doing my testing at the moment.

It’s perhaps worth noting that for some reason, 317.19Mi or 317.20Mi of storage are still reported by toolforge build quota even after a successful toolforge build clean (twice now, with almost the same number). I don’t know where that’s coming from.

[step-export] 2024-01-27T17:24:44.425813779Z ERROR: failed to export: failed to write image to the following tags: [tools-harbor.wmcloud.org/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest: PUT https://tools-harbor.wmcloud.org/v2/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test/blobs/uploads/422c60a3-46d0-421c-a6a5-f8037a49c395?_state=REDACTED&digest=sha256%3A7676309ce3f84c4f54204415e3a4713b4f9c20f78723b3fb0e9132acf4762201: DENIED: adding 158.6 MiB of storage resource, which when updated to current usage of 972.7 MiB will exceed the configured upper limit of 1.0 GiB.]
[step-results] 2024-01-27T17:24:44.552919216Z 2024/01/27 17:24:44 Skipping step because a previous step failed
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build clean
NOTE: This will remove all your built images to clean up space, you'll have to start a new build to create a new image before being able to restart any running webservice/job or start a new one, are you sure? [y/N]: y
Result of the cleanup: Deleted 1 artifacts from harbor repository tool-lucaswerkmeister-test/tool-lucaswerkmeister-test

If you still need more space, please contact a toolforge maintainer (https://w.wiki/6Zuu).
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build quota
Registry
===================
Storage
-----------
Limit        1.00Gi
Used         317.20Mi
Available    706.80Mi
Capacity     31%

tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build list
build_id                                            status    start_time            end_time              source_url                                                  ref    envvars    destination_image
lucaswerkmeister-test-buildpacks-pipelinerun-lrkfj  error     2024-01-27T17:10:56Z  2024-01-27T17:24:45Z  https://gitlab.wikimedia.org/toolforge-repos/wd-shex-infer  k8s    N/A        tools-harbor.wmcloud.org/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest
lucaswerkmeister-test-buildpacks-pipelinerun-nkrlx  ok        2024-01-27T16:37:37Z  2024-01-27T16:52:54Z  https://gitlab.wikimedia.org/toolforge-repos/wd-shex-infer  k8s    N/A        tools-harbor.wmcloud.org/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build delete lucaswerkmeister-test-buildpacks-pipelinerun-lrkfj lucaswerkmeister-test-buildpacks-pipelinerun-nkrlx
I'm going to delete 2 builds, continue? [y/N]: y
Deleted 2 builds
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build list
No builds found, you can start one using `toolforge build start`,run `toolforge build start --help` for more details
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build quota
Registry
===================
Storage
-----------
Limit        1.00Gi
Used         317.20Mi
Available    706.80Mi
Capacity     31%

Event Timeline

Another look at the mysterious leftover space:

tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build quota
Registry
===================
Storage
-----------
Limit        1.00Gi
Used         972.71Mi
Available    51.29Mi
Capacity     95%

tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build clean
NOTE: This will remove all your built images to clean up space, you'll have to start a new build to create a new image before being able to restart any running webservice/job or start a new one, are you sure? [y/N]: y                                                                            
Result of the cleanup: Deleted 1 artifacts from harbor repository tool-lucaswerkmeister-test/tool-lucaswerkmeister-test

If you still need more space, please contact a toolforge maintainer (https://w.wiki/6Zuu).
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build quota
Registry
===================
Storage
-----------
Limit        1.00Gi
Used         317.20Mi
Available    706.80Mi
Capacity     31%

This is after building (and then cleaning) one image, so it seems that currently my image is somewhere around 655.51 MiB big, and I don’t know what the remaining 317.20 MiB are.

It’s perhaps worth noting that for some reason, 317.19Mi or 317.20Mi of storage are still reported by toolforge build quota even after a successful toolforge build clean (twice now, with almost the same number). I don’t know where that’s coming from.

Hmm, I have not been able to reproduce this. If it happens again, would you mind opening a separate bug report?

...
[step-results] 2024-01-29T09:42:36.500163907Z Built image tools-harbor.wmcloud.org/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest@sha256:f61e4e82c66c1c5b3684ad9dc9a70dd0f3d3ce557a2dd4cdc2d3045a090514e0
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build quota
Registry
===================
Storage
-----------
Limit        1.00Gi
Used         661.30Mi
Available    362.70Mi
Capacity     65%

tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build clean
NOTE: This will remove all your built images to clean up space, you'll have to start a new build to create a new image before being able to restart any running webservice/job or start a new one, are you sure? [y/N]: y
Result of the cleanup: Deleted 1 artifacts from harbor repository tool-lucaswerkmeister-test/tool-lucaswerkmeister-test

If you still need more space, please contact a toolforge maintainer (https://w.wiki/6Zuu).
tools.lucaswerkmeister-test@tools-sgebastion-10:~$ toolforge build quota
Registry
===================
Storage
-----------
Limit        1.00Gi
Used         0B
Available    1.00Gi
Capacity     0%
Slst2020 changed the task status from Open to In Progress.Jan 29 2024, 10:56 AM
Slst2020 claimed this task.

Done – Both projects now have 2Gi harbor storage quota each.

I did notice that also at some point, it's gone now though, I'm guessing that there might be some dangling layers or similar that a cleanup job takes care of eventually, let's see if it happens more often though.