Page MenuHomePhabricator

[builds-api] catch harbor timeout when creating repository
Closed, ResolvedPublic

Description

Per the builds-api logic, when creating a new build (start action), the harbor project (repository) for the tool is created.

However, if there is network connectivity problems to the harbor API for whatever reason, the error reported to the user is weird.

In this particular example, harbor was not listening in the expected IP address, and caused this user-visible error.

user@laptop:~$ toolforge-build --debug start https://gitlab.wikimedia.org/toolforge-repos/wm-lol
DEBUG:toolforge_weld.config:Unable to find config file /etc/toolforge/builds.yaml, skipping
DEBUG:toolforge_weld.config:Unable to find config file /etc/toolforge/common.yaml, skipping
DEBUG:toolforge_weld.config:Unable to find config file /home/arturo/.toolforge.yaml, skipping
DEBUG:toolforge_weld.config:Unable to find config file /home/arturo/.config/toolforge.yaml, skipping
DEBUG:toolforge_weld.config:Updating config from /home/arturo/.toolforge-lima-kilo/chroot/etc/toolforge/toolforge.yaml
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): localhost:30003
An unknown error occured while trying to perform this operation.
If the problem persists, please contact us or open a bug:
see https://phabricator.wikimedia.org/T324822
Original error: HTTPSConnectionPool(host='localhost', port=30003): Read timed out. (read timeout=10)

On server side, the logs were:

[..]
{"level":"debug","msg":"Authenticating for /v1/build","time":"2023-09-08T10:41:46Z"}
{"level":"debug","msg":"Got user tf-test, error %!s(\u003cnil\u003e)","time":"2023-09-08T10:41:46Z"}
{"level":"debug","msg":"Got user tf-test, error %!s(\u003cnil\u003e)","time":"2023-09-08T10:41:46Z"}
{"level":"debug","msg":"Attempting to create harbor project tool-tf-test","time":"2023-09-08T10:41:46Z"}
{"level":"error","msg":"Failed to create harbor project for tool tf-test: Post \"http://192.168.1.236/api/v2.0/projects\": dial tcp 192.168.1.236:80: i/o timeout","time":"2023-09-08T10:42:16Z"}
[..]

My suggestion would be to catch this error (because it can happen in real toolforge) and report something more meaningful to the user,
something like timeout connecting to harbor, please contact a Toolforge admin [..]

Details

ReferenceSource BranchDest BranchAuthorTitle
repos/cloud/toolforge/toolforge-deploy!128bump_builds-apimainraymond-ndibebuilds-api: bump to 0.0.104-20231113143657-f8b48e05
repos/cloud/toolforge/builds-api!55handle_harbor_request_timeoutsmainraymond-ndibe[build.start]: handle harbor timeout error
Customize query in GitLab

Event Timeline

This error which is handled more gracefully:

user@laptop:~$ toolforge-build --debug start https://gitlab.wikimedia.org/toolforge-repos/wm-lol
DEBUG:toolforge_weld.config:Unable to find config file /etc/toolforge/builds.yaml, skipping
DEBUG:toolforge_weld.config:Unable to find config file /etc/toolforge/common.yaml, skipping
DEBUG:toolforge_weld.config:Unable to find config file /home/arturo/.toolforge.yaml, skipping
DEBUG:toolforge_weld.config:Unable to find config file /home/arturo/.config/toolforge.yaml, skipping
DEBUG:toolforge_weld.config:Updating config from /home/arturo/.toolforge-lima-kilo/chroot/etc/toolforge/toolforge.yaml
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): localhost:30003
DEBUG:urllib3.connectionpool:https://localhost:30003 "POST /builds/v1/build HTTP/1.1" 500 156
Failed to create harbor project for tool tf-test: Post "http://localhost/api/v2.0/projects": dial tcp [::1]:80: connect: connection refused

But maybe should include a reference to how to contact Toolforge admins.

Raymond_Ndibe changed the task status from Open to In Progress.Oct 24 2023, 5:50 PM
Raymond_Ndibe reopened this task as Open.
Raymond_Ndibe moved this task from In Review to Done on the Toolforge (Toolforge iteration 02) board.