[builds-cli,builds-api] Allow build service to cleanup images to free quota
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	taavi
	Jul 4 2023, 1:15 PM

Description

Add a command to the cli toolforge build cleanup-images or similar that will remove all the harbor images for a tool to free up space.

This should make sure to prompt the user for confirmation, saying something like:

This will remove all the images you built, you will have to build a new one to start/restart a new job or webservice. Continue? [y/N]
y
...

All images cleaned up, now you must build a new one before starting a job or webservice, using a command like:
    toolforge build start <url-to-git-repo>

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		Raymond_Ndibe	T351092 [harbor,docs] Improve Harbor quota handling and docs
		Resolved		dcaro	T341067 [builds-cli,builds-api] Allow build service to cleanup images to free quota

Event Timeline

taavi created this task.Jul 4 2023, 1:15 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2023, 1:15 PM

I think we should make the system in a way that the user does not have to know or care if there's images behind it (similat to what do or heroku do).

It would be nice if we delete the backing image when deleting a build, but I think that just a cleanup of the images might be enough.

This might mean also that the user does not have to care much about builds, making something like toolforge deploy or toolforge app deploy the trigger to build + deploy the app, leaving the build as just as step in the pipeline and any details of how it's implemented (using container images or anything else) a curiosity for the user but not a need nor a concern.

I'm still trying to get a clearer idea of how that abstraction should be in detail, but I see clearly that caring about the images themselves should be a very low level activity, and not the default path for most users.

Hmm. One option might be:

toolforge $SOMETHING list, displays list of images from harbor
toolforge $SOMETHING create bar --git-url="foo", starts a new build
toolforge $SOMETHING show bar, displays data about the harbor image storage use and builds for that image
toolforge $SOMETHING logs bar [build-id]
toolforge $SOMETHING update bar [--git-url=baz], starts a new build and defaults to the URL from the previous build
toolforge $SOMETHING delete bar, deletes an image from harbor and related builds

The UX would be something like:

:# create new thing
$ toolforge $SOMETHING create myproject --git-url https://gitlab.wikimedia.org/toolforge-repos/my-project.git
:# wait for it to complete
$ toolforge $SOMETHING show myproject
$ toolforge $SOMETHING logs myproject
:# use it
$ toolforge webservice buildservice start --image=mytool/myproject
$ toolforge jobs run --continuous --image=mytool/myproject --command "my-worker" worker
:# after updating the git repo
$ toolforge $SOMETHING update myproject
:# if it's no longer needed
$ toolforge $SOMETHING delete myproject

I think that overlaps with the webservice command, it's somehow similar to the 'app' command that was mentioned before, I guess that'd be the $SOMETHING there.

My point is that I think "images" is the wrong abstraction, I know it's what k8s uses, but the users should not need to know it (same as they should not need to know k8s).

I would propose a UX like:

# show all the 'apps' that you have in the tool account, ex. backend, frontend, worker, ...
$ toolforge app list 

# create a new app, this should probably try to build it and deploy it right away, setup the CI/CD for it and such
$ toolforge app create [--repo-url url/to/repo (maybe default to gitlab)] [--name customname (default: web or similar)]

# get the app logs and related, probably from all the services involved (build logs if a build is running, run logs, start logs, ....)
$ toolforge app logs

# trigger a new build + deploy for the specific app
$ toolforge app deploy [--refspec] [--repo-url] [... other potential options, like --webservice, --continuous, --one-off ...]

# similar to what you mentoined
$ toolforge app delete

Once that is there, the build command becomes not needed for most of the users, and controlling which image is being deployed with, not necessary, you just care about the git repository.

We can add extra info, like the commit hash the app was deployed with and such.

I agree that the quota usage in harbor should be improved, but I don't think the users should know about it.

dcaro moved this task from Backlog to To be Discussed on the Toolforge Build Service board.Sep 13 2023, 11:55 AM

Would be interesting to have though an "escape" command to remove all the images, something like:

$ toolforge build remove-all --yes-im-sure
Do this only if you are out of quota and want to build a new image to be deployed right after.
This will remove all the build and associated images, this will prevent your webservice from starting again until you build it, are you sure?

Just in case a user has an image that takes >50% the quota and can't rebuild the new one with the old one still in the repo.

In T341067#9163276, @dcaro wrote:

My point is that I think "images" is the wrong abstraction, I know it's what k8s uses, but the users should not need to know it (same as they should not need to know k8s).

Agree that the end-user should not be exposed to the concept of "images". In terms of UX and CLI semantics, we should probably think one step ahead of what is possible today. What if we enable an "App" to have multiple "Components", all part of the same "Tool". For instance, a frontend exposed to the web and a backend that is exposed only to the frontend. Would that change the way we think about this?

dcaro edited projects, added Toolforge Build Service (Iteration 20); removed Toolforge Build Service.Sep 13 2023, 2:28 PM

I think that we are in the same page here, your "component" there would be the "app" I mentioned.

As in we have tool accounts, that can create many "code running thingies"(apps/components/whatever), some are one-off runs, some periodical, some continuous, some expose some ports internally and only one cat expose an external port (webservice).

dcaro edited projects, added Toolforge (Toolforge iteration 00); removed Toolforge Build Service (Iteration 20).Sep 26 2023, 2:23 PM

fnegri subscribed.Oct 10 2023, 1:34 PM

dcaro edited projects, added Toolforge (Toolforge iteration 01); removed Toolforge (Toolforge iteration 00).Oct 10 2023, 1:59 PM

dcaro renamed this task from Allow listing and managing images of a tool to [builds-cli,builds-api] Allow build service to cleanup images to free quota.Oct 25 2023, 12:40 PM

dcaro updated the task description. (Show Details)

Hmm, how does the updated task description mesh with the wish expressed earlier in this discussion to not expose users to the "image" abstraction?

More generally, we might be reaching the limits of "winging it" in terms of UI/UX. I know you are working on consolidating our roadmap @dcaro. Maybe this decision needs to wait until we collectively have a clearer vision of the way forward for new toolforge cli/api features.

In T341067#9279590, @Slst2020 wrote:

Hmm, how does the updated task description mesh with the wish expressed earlier in this discussion to not expose users to the "image" abstraction?

Replace cleanup-images with just cleanup, garbage-collect or other generic term that does not include images in it.

We then can use that command to free up also any other build-related resources the user might be exceeding (though builds themselves do auto-clean, so I guess that there's no other resources as of right now).

In T341067#9279639, @dcaro wrote:

In T341067#9279590, @Slst2020 wrote:

Hmm, how does the updated task description mesh with the wish expressed earlier in this discussion to not expose users to the "image" abstraction?

Replace cleanup-images with just cleanup, garbage-collect or other generic term that does not include images in it.

We then can use that command to free up also any other build-related resources the user might be exceeding (though builds themselves do auto-clean, so I guess that there's no other resources as of right now).

Extra point if it return a message like "XMB freed!", that is always very rewarding :)

dcaro edited projects, added Toolforge (Toolforge iteration 02); removed Toolforge (Toolforge iteration 01).Oct 25 2023, 1:53 PM

Slst2020 added a parent task: T351092: [harbor,docs] Improve Harbor quota handling and docs.Nov 14 2023, 8:40 AM

Slst2020 added a comment.Dec 4 2023, 2:35 PM

This comment was removed by Slst2020.

dcaro mentioned this in T351092: [harbor,docs] Improve Harbor quota handling and docs.Dec 5 2023, 10:21 AM

dcaro mentioned this in T351178: [tbs] Give a meaningful error message when a user exceeds their Harbor quota.

dcaro changed the task status from Open to In Progress.Dec 12 2023, 1:15 PM

dcaro claimed this task.

dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 02) board.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/66

add cleanup

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/37

cleanup: add subcommand to free some quota

dcaro moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 02) board.Dec 14 2023, 4:12 PM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/66

clean: add new endpoint to cleanup space

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/157

builds-api: bump to 0.0.118-20231215130655-6b2c1521

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-15T13:17:49Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T341067)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-15T13:18:20Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T341067)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-15T13:32:51Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T341067)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-15T13:33:27Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T341067)