There are (as far as I can tell, haven't done thorough research yet) no docs about how much Harbor quota each tool has, if the quota is per-image, per-tool or both, if there is a process to get more, etc.
In addition, trying to push an image larger than the quota results in an unhelpful raw 413 error.
Example error:
step-export: 2023-11-13T05:41:56.835942824Z ERROR: failed to export: failed to write image to the following tags: [tools-harbor.wmcloud.org/tool-dpe-alerts-dashboard/tool-dpe-alerts-dashboard:latest: PATCH https://tools-harbor.wmcloud.org/v2/tool-dpe-alerts-dashboard/tool-dpe-alerts-dashboard/blobs/uploads/b62dd944-4fad-4ee8-b900-8409f7860d6c?_state=REDACTED: unexpected status code 413 Request Entity Too Large: <html> step-export: 2023-11-13T05:41:56.835973012Z <head><title>413 Request Entity Too Large</title></head> step-export: 2023-11-13T05:41:56.835976984Z <body> step-export: 2023-11-13T05:41:56.835979969Z <center><h1>413 Request Entity Too Large</h1></center> step-export: 2023-11-13T05:41:56.835983468Z <hr><center>nginx/1.18.0</center> step-export: 2023-11-13T05:41:56.836002364Z </body> step-export: 2023-11-13T05:41:56.836005027Z </html> step-export: 2023-11-13T05:41:56.836008032Z ] step-export: step-results: 2023-11-13T05:41:57.433667715Z 2023/11/13 05:41:57 Skipping step because a previous step failed
The current quota is 1GB per tool. From irc conversation with @tchin, this image was 1.17 GB.
I think the per-image quota should probably be increased. I tested building a few projects locally and a project with NodeJS and 0 dependencies results in a built image that's 805.58 MB. One with only VueJS as a dependency bumps it up to 858.13 MB. I'm probably not going to be the last one who needs more than 200 MB of working space :/
Some notes on the current setup:
As @taavi pointed out on irc, 1GB per project is probably on the low side. We have a number of "hello world" type of projects with a minimal amount of dependencies. These are in the 250-350 MB range. Given that users have no control over how images are built (choice of base images, multi-stage builds, etc.), asking folks to slim down their builds may be unrealistic.
If we allow 5 images per project, a more realistic limit could be ~500 MB per image = 2.5 GB per project.
Harbor does not come with any default quotas, but we have the default project quota set to 1GB. This was likely done manually, as maintain-harbor does not set any storage limits.
Yep, done manually
If we allow 5 images per project, a more realistic limit could be ~500 MB per image = 2.5 GB per project.
I think that 5 images is too much even, we should only store 2 images if the space is limiting (current and rollback, that we don't have a process yet to do).
For the docs on the current quota setup this might be T329176: [maintain-harbor,docs] Document current setup and admin procedures
For the docs on how to request more quota, we might want to do it directly from the cli here T351178: [tbs] Give a meaningful error message when a user exceeds their Harbor quota
One of the suggestions should be using the cleanup command T341067: [builds-cli,builds-api] Allow build service to cleanup images to free quota, and if that's not enough, then creating a task for us.
For the docs on the current quota setup this might be T329176: [tbs.maintain-harbor] Document current setup and admin procedures
Nope, that's just maintain-harbor, the harbor admin docs were already created https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Harbor
So should update that.
I think this task is specifically for documenting the functioning of the maintain-harbor tool, not so much for user-facing docs about getting a quota bump, what the default quota is, etc.
circling back to this, @tchin can you drop the repository URL you used here so I can attempt to reproduce this? @Slst2020 do you have the url?
Something is going on here other than just the quota limit being exceeded. This is related to some nginx proxy server (I think we only have one that handles harbor traffics).
out of quota error response directly from harbor itself looks like this:
[step-export] 2024-01-31T02:30:24.298291285Z 192.168.188.128/tool-minikube-user/tool-minikube-user:latest - PUT http://192.168.188.128/v2/tool-minikube-user/tool-minikube-user/blobs/uploads/73207165-265c-43b4-8a0a-68c5e9a86a51?_state=REDACTED&digest=sha256%3A62aa624a48ae8657521d340a8a360cfd57e9949aec354ed92594451a5c1ed300: DENIED: adding 2.4 MiB of storage resource, which when updated to current usage of 414.0 MiB will exceed the configured upper limit of 415.0 MiB. [step-export] 2024-01-31T02:30:24.312467187Z ERROR: failed to export: failed to write image to the following tags: [192.168.188.128/tool-minikube-user/tool-minikube-user:latest: PUT http://192.168.188.128/v2/tool-minikube-user/tool-minikube-user/blobs/uploads/73207165-265c-43b4-8a0a-68c5e9a86a51?_state=REDACTED&digest=sha256%3A62aa624a48ae8657521d340a8a360cfd57e9949aec354ed92594451a5c1ed300: DENIED: adding 2.4 MiB of storage resource, which when updated to current usage of 414.0 MiB will exceed the configured upper limit of 415.0 MiB.] [step-export] 2024-01-31T02:30:24.317625401Z export step failed
The request probably didn't get to harbor itself before being DENIED