Page MenuHomePhabricator
Create Task
Maniphest T337025

Unknown error when trying to delete a Toolforge build
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):

  • Start a Toolforge build (toolforge build start SOME_REPO)
  • Wait for it to finish
  • Get its ID from toolforge build list
  • Try to delete it with toolforge build delete ITS_ID

What happens?:

…
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/api/v1/namespaces/image-build/pods?labelSelector=tekton.dev%2FpipelineRun%3Dlucaswerkmeister-wmde-test-buildpacks-pipelinerun-cwp4j

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/toolforge", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/toolforge_cli/cli.py", line 809, in main
    toolforge()
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/toolforge_cli/cli.py", line 440, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/toolforge_cli/cli.py", line 738, in build_delete
    k8s_client.delete_objects(kind="pods", selector=f"tekton.dev/pipelineRun={run['metadata']['name']}")
  File "/usr/lib/python3/dist-packages/toolforge_cli/k8sclient.py", line 220, in delete_objects
    version=K8sAPIClient.KIND_TO_VERSION[kind],
  File "/usr/lib/python3/dist-packages/toolforge_cli/k8sclient.py", line 55, in _inner
    raise K8sError(toolforge_build.ERROR_STRINGS["UNKNOWN_ERROR"]) from error
toolforge_cli.k8sclient.K8sError: An unknown error occured while trying to perform this operation.
If the problem persists, please contact us or open a bug:
see https://phabricator.wikimedia.org/T324822

What should have happened instead?:
The build should have been deleted.

Software version (skip for WMF-hosted wikis like Wikipedia):
Toolforge CLI, version 0.3.0

Other information (browser name/version, screenshots, etc.):
I tried this in the lucaswerkmeister-wmde-test tool with the lucaswerkmeister-wmde-test-buildpacks-pipelinerun-cwp4j build. (It’s now gone, but I don’t know if that’s because the delete actually worked or if it was cleaned up automatically after a while – I got distracted right after getting the error.)

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
build: use the api to delete buildsrepos/cloud/toolforge/toolforge-cli!5dcaromove_delete_to_apimain
Customize query in GitLab

Related Objects

Event Timeline

Lucas_Werkmeister_WMDE created this task.May 19 2023, 11:33 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2023, 11:33 AM
dcaro edited projects, added Toolforge Build Service (Beta release); removed Toolforge Build Service.May 19 2023, 11:34 AM
dcaro subscribed.

Will look into it :), thanks!

Lucas_Werkmeister_WMDE added a comment.May 19 2023, 1:47 PM

I just tried it with another build, and the behavior indeed seems to be that I get an error, but the build is still deleted successfully.

Lucas_Werkmeister_WMDE added a comment.May 19 2023, 1:49 PM

Full output of toolforge build delete lucaswerkmeister-wmde-test-buildpacks-pipelinerun-hsz44 (the one in the task description is slightly truncated because of mosh):

I'm going to delete 1 runs, continue? [y/N]: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/toolforge_cli/k8sclient.py", line 47, in _inner
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/toolforge_cli/k8sclient.py", line 186, in _delete
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/api/v1/namespaces/image-build/pods?labelSelector=tekton.dev%2FpipelineRun%3Dlucaswerkmeister-wmde-test-buildpacks-pipelinerun-hsz44

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/toolforge", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/toolforge_cli/cli.py", line 809, in main
    toolforge()
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/toolforge_cli/cli.py", line 440, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/toolforge_cli/cli.py", line 738, in build_delete
    k8s_client.delete_objects(kind="pods", selector=f"tekton.dev/pipelineRun={run['metadata']['name']}")
  File "/usr/lib/python3/dist-packages/toolforge_cli/k8sclient.py", line 220, in delete_objects
    version=K8sAPIClient.KIND_TO_VERSION[kind],
  File "/usr/lib/python3/dist-packages/toolforge_cli/k8sclient.py", line 55, in _inner
    raise K8sError(toolforge_build.ERROR_STRINGS["UNKNOWN_ERROR"]) from error
toolforge_cli.k8sclient.K8sError: An unknown error occured while trying to perform this operation.
If the problem persists, please contact us or open a bug:
see https://phabricator.wikimedia.org/T324822
Lucas_Werkmeister_WMDE added a comment.May 19 2023, 1:49 PM

There’s a few more leftover builds in lucaswerkmeister-wmde-test, feel free to use those for testing :)

Lucas_Werkmeister_WMDE mentioned this in T337040: Play with Toolforge build service / buildpacks.May 19 2023, 2:07 PM
Raymond_Ndibe added a project: User-Raymond_Ndibe.May 24 2023, 12:57 AM
dcaro changed the task status from Open to In Progress.Jun 16 2023, 11:40 AM
dcaro claimed this task.
dcaro moved this task from Backlog to Doing on the Toolforge Build Service (Beta release) board.
dcaro added a comment.Jun 16 2023, 1:55 PM

We have implemented autoclean for builds, that is triggered when you start a new one, let me know if that help for now while getting the fix for the delete subcommand out.

CodeReviewBot added a comment.Jun 16 2023, 3:08 PM

dcaro updated https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/16

delete: add endpoint

CodeReviewBot added a comment.Jun 21 2023, 9:58 AM

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/16

delete: add endpoint

Stashbot added a comment.Jun 21 2023, 11:48 AM

Mentioned in SAL (#wikimedia-cloud) [2023-06-21T11:48:50Z] <dcaro> deploy bulids-api 0.2.0 (T337025)

Stashbot added a comment.Jun 21 2023, 11:57 AM

Mentioned in SAL (#wikimedia-cloud) [2023-06-21T11:57:25Z] <dcaro> deploy bulids-api 0.2.0 (T337025)

CodeReviewBot added a comment.Jun 21 2023, 1:00 PM

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/5

build: use the api to delete builds

CodeReviewBot added a project: Patch-For-Review.Jun 21 2023, 1:00 PM
CodeReviewBot added a comment.Jun 23 2023, 12:22 AM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/5

build: use the api to delete builds

Stashbot added a comment.Jun 23 2023, 3:42 PM

Mentioned in SAL (#wikimedia-cloud) [2023-06-23T15:42:43Z] <dcaro> deploy builds-api 0.3.2 (T337025)

dcaro added a comment.Jun 29 2023, 10:18 AM

This has been deployed, you should be able to delete builds if you want now :)

dcaro closed this task as Resolved.Jun 29 2023, 10:18 AM
dcaro moved this task from Doing to Done on the Toolforge Build Service (Beta release) board.
Lucas_Werkmeister_WMDE added a comment.Jun 29 2023, 10:45 AM

Being able to delete more than one build at once would be quite useful IMHO, especially if it’s going to ask me for confirmation (and also because the command seems quite slow to start up). But I can confirm it works.

tools.lucaswerkmeister-wmde-test@tools-sgebastion-10:~$ toolforge build delete lucaswerkmeister-wmde-test-buildpacks-pipelinerun-xz8kd
I'm going to delete the build lucaswerkmeister-wmde-test-buildpacks-pipelinerun-xz8kd, continue? (there's no way back) [y/N]: y
Deleted build lucaswerkmeister-wmde-test-buildpacks-pipelinerun-xz8kd.
tools.lucaswerkmeister-wmde-test@tools-sgebastion-10:~$ toolforge build delete lucaswerkmeister-wmde-test-buildpacks-pipelinerun-tjf72 lucaswerkmeister-wmde-test-buildpacks-pipelinerun-8q9b2 lucaswerkmeister-wmde-test-buildpacks-pipelinerun-bkvrh lucaswerkmeister-wmde-test-buildpacks-pipelinerun-vfj58 
Usage: toolforge build delete [OPTIONS] BUILD_NAME
Try "toolforge build delete --help" for help.

Error: Got unexpected extra arguments (lucaswerkmeister-wmde-test-buildpacks-pipelinerun-8q9b2 lucaswerkmeister-wmde-test-buildpacks-pipelinerun-bkvrh lucaswerkmeister-wmde-test-buildpacks-pipelinerun-vfj58)
Raymond_Ndibe mentioned this in rCTTCe4b107ccade4: build: use the api to delete builds.Oct 3 2025, 9:41 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits