Once we've got all services on zuul100{1,2} VMs. And we have nodepool talking to our worker setup in WMCS. We should be able to run a simple test on these VMs.
In the prototype phase, we started with a Hello World job for the test/gerrit-ping project.
Seems like a fine initial test for the VMs to confirm everything is set up correctly.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dzahn | T395938 puppetize setup of new zuul VMs | |||
| In Progress | dduvall | T405120 Test new zuul test VMs |
Change #1283967 had a related patch set uploaded (by Dduvall; author: Dduvall):
[integration/config@master] zuul: Echo job for basic testing of Zuul 14
Change #1283967 merged by jenkins-bot:
[integration/config@master] zuul: Echo job for basic testing of Zuul 14
Change #1284000 had a related patch set uploaded (by Dduvall; author: Dduvall):
[integration/config@master] zuul: Add missing run playbook to echo job
Change #1284000 merged by jenkins-bot:
[integration/config@master] zuul: Add missing run playbook to echo job
An initial test in the test/gerrit-ping repo has surfaced a couple of issues.
First, node creation via the kubernetes based provider is failing, seeming due to service account perms. From the patchset comment:
Build failed.
https://zuul.wikimedia.org/t/wikimedia/buildset/15fb63323838447081d51892400083c3echo https://zuul.wikimedia.org/t/wikimedia/build/30357eea008d49e980534be8a203ce3f : NODE_FAILURE Node(set) request 10b432f6e3ba4a61b3b5cffa5b17e9a6 failed in 0s
From zuul-scheduler and zuul-launcher logs:
May 06 22:42:07 zuul1001 docker[1046299]: 2026-05-06 22:42:07,846 INFO zuul.GerritConnection: [e: 13a2c6097375453a847d0a3e6ba454e7] Updating <Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>
May 06 22:42:08 zuul1001 docker[1046299]: 2026-05-06 22:42:08,572 INFO zuul.Pipeline.wikimedia.check: [e: 13a2c6097375453a847d0a3e6ba454e7] Adding [<Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>] to queue <ChangeQueue check: > in <Pipeline check>
May 06 22:42:08 zuul1001 docker[1046299]: 2026-05-06 22:42:08,596 INFO zuul.Pipeline.wikimedia.check: [e: 13a2c6097375453a847d0a3e6ba454e7] Reporting enqueue, action [] item <QueueItem f40e5182044b49e18334dfb0e5c4ce91 live for [<Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>] in check>
May 06 22:42:10 zuul1001 docker[1046299]: 2026-05-06 22:42:10,009 INFO zuul.ConfigLoader: [e: 13a2c6097375453a847d0a3e6ba454e7] Loading configuration dynamically from test/gerrit-ping/.zuul.yaml@master
May 06 22:42:10 zuul1001 docker[1046299]: 2026-05-06 22:42:10,066 INFO zuul.Pipeline.wikimedia.check: [e: 13a2c6097375453a847d0a3e6ba454e7] Scheduling global repo state for item <QueueItem f40e5182044b49e18334dfb0e5c4ce91 live for [<Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>] in check>
May 06 22:42:11 zuul1001 docker[1046299]: 2026-05-06 22:42:11,193 INFO zuul.Pipeline.wikimedia.check: Received global repo state for <QueueItem f40e5182044b49e18334dfb0e5c4ce91 live for [<Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>] in check>
May 06 22:42:11 zuul1001 docker[1046299]: 2026-05-06 22:42:11,237 INFO zuul.LauncherClient: [e: 13a2c6097375453a847d0a3e6ba454e7] Submitted nodeset request <NodesetRequest uuid=69a5b0c9453a46558c170f84669b587f, state=requested, labels=['trixie'], path=/zuul/nodeset/requests/69a5b0c9453a46558c170f84669b587f>
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: [e: 13a2c6097375453a847d0a3e6ba454e7] [req: 69a5b0c9453a46558c170f84669b587f] Marking node <KubernetesProviderNode uuid=46db03236ff94465bf7df02706ae1e67, label=trixie, state=building, provider=gerrit.wikimedia.org%2Fintegration%2Fconfig/untrusted> as failed
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: Traceback (most recent call last):
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/launcher/server.py", line 1913, in _processProviderNode
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: self._checkNodeCreate(node, log)
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/launcher/server.py", line 2143, in _checkNodeCreate
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: instance = node.create_state_machine.advance()
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/driver/kubernetes/kubernetesendpoint.py", line 147, in advance
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: self.endpoint._createRole(self.namespace, self.user,
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/driver/kubernetes/kubernetesendpoint.py", line 382, in _createRole
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: self.rbac_client.create_namespaced_role(namespace, role_body)
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/kubernetes/client/api/rbac_authorization_v1_api.py", line 335, in create_namespaced_role
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: return self.create_namespaced_role_with_http_info(namespace, body, **kwargs) # noqa: E501
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/kubernetes/client/api/rbac_authorization_v1_api.py", line 434, in create_namespaced_role_with_http_info
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: return self.api_client.call_api(
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/kubernetes/client/api_client.py", line 348, in call_api
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: return self.__call_api(resource_path, method,
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/kubernetes/client/api_client.py", line 180, in __call_api
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: response_data = self.request(
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: ^^^^^^^^^^^^^
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/kubernetes/client/api_client.py", line 391, in request
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: return self.rest_client.POST(url,
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^^
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/kubernetes/client/rest.py", line 279, in POST
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: return self.request("POST", url,
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/kubernetes/client/rest.py", line 238, in request
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: raise ApiException(http_resp=r)
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: kubernetes.client.exceptions.ApiException: (403)
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: Reason: Forbidden
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: HTTP response headers: HTTPHeaderDict({'Audit-Id': '9816da54-a3a0-4e06-9920-315627eec4cc', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '51296335-897e-4bb6-a0ac-8c5b6cae4d30', 'X-Kubernetes-Pf-Prioritylevel-Uid': '60aff477-c56d-4d54-ae51-a4eceb9a3166', 'Date': 'Wed, 06 May 2026 22:42:13 GMT', 'Content-Length': '603'})
May 06 22:42:13 zuul1001 docker[1046084]: 2026-05-06 22:42:13,331 ERROR zuul.Launcher: HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":
{},"status":"Failure","message":"roles.rbac.authorization.k8s.io \"zuul-restricted\" is forbidden: user \"system:serviceaccount:default:nodepool-svc\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:default\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n
{APIGroups:[\"\"], Resources:[\"pods/logs\"], Verbs:[\"create\" \"delete\" \"get\" \"list\" \"patch\" \"update\" \"watch\"]}","reason":"Forbidden","details":{"name":"zuul-restricted","group":"rbac.authorization.k8s.io","kind":"roles"},"code":403}
[... (2 more tries)]
May 06 22:42:18 zuul1001 docker[1046084]: 2026-05-06 22:42:18,327 ERROR zuul.Launcher:
May 06 22:42:18 zuul1001 docker[1046084]: 2026-05-06 22:42:18,327 ERROR zuul.Launcher:
May 06 22:42:19 zuul1001 docker[1046084]: 2026-05-06 22:42:19,246 ERROR zuul.Launcher: [e: 13a2c6097375453a847d0a3e6ba454e7] [req: 69a5b0c9453a46558c170f84669b587f] Marking request <NodesetRequest uuid=69a5b0c9453a46558c170f84669b587f, state=accepted, labels=['trixie'], path=/zuul/nodeset/requests/69a5b0c9453a46558c170f84669b587f> as failed: No provider found for label trixie
May 06 22:42:19 zuul1001 docker[1046299]: 2026-05-06 22:42:19,356 INFO zuul.Pipeline.wikimedia.check: [e: 13a2c6097375453a847d0a3e6ba454e7] Node(set) request <NodesetRequest uuid=69a5b0c9453a46558c170f84669b587f, state=failed, labels=['trixie'], path=/zuul/nodeset/requests/69a5b0c9453a46558c170f84669b587f>: failure for echo
May 06 22:42:19 zuul1001 docker[1046299]: 2026-05-06 22:42:19,420 INFO zuul.Pipeline.wikimedia.check: [e: 13a2c6097375453a847d0a3e6ba454e7] Completed node(set) request <NodesetRequest uuid=69a5b0c9453a46558c170f84669b587f, state=failed, labels=['trixie'], path=/zuul/nodeset/requests/69a5b0c9453a46558c170f84669b587f> for job echo of item <QueueItem f40e5182044b49e18334dfb0e5c4ce91 live for [<Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>] in check> with nodes ['865fea887f9e48aa97160763124ecd49']
May 06 22:42:19 zuul1001 docker[1046299]: 2026-05-06 22:42:19,435 INFO zuul.Pipeline.wikimedia.check: [e: 13a2c6097375453a847d0a3e6ba454e7] Reporting item <QueueItem f40e5182044b49e18334dfb0e5c4ce91 live for [<Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>] in check>, actions: [<GerritReporter: failure>]
May 06 22:42:19 zuul1001 docker[1046299]: 2026-05-06 22:42:19,636 INFO zuul.GerritConnection: [e: f7d149e4d29841b79c9509471bc3e19a] Updating <Change 0x7fd88c1f71d0 test/gerrit-ping 1283997,1>Second, which I'm guessing is in fact a secondary error due to the nodeset creation failure, following the buildset URL yields an endless spinner. From the zuul-web logs:
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: [req: 28d43b2ebf07420d8155a01efebf7cf7] [06/May/2026:22:58:44] HTTP
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: Traceback (most recent call last):
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/local/lib/python3.11/dist-packages/cherrypy/_cprequest.py", line 659, in respond
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: self._do_respond(path_info)
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/local/lib/python3.11/dist-packages/cherrypy/_cprequest.py", line 723, in _do_respond
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: response.finalize()
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/local/lib/python3.11/dist-packages/cherrypy/_cprequest.py", line 936, in finalize
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: content = self.collapse_body()
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: ^^^^^^^^^^^^^^^^^^^^
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/local/lib/python3.11/dist-packages/cherrypy/_cprequest.py", line 894, in collapse_body
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: new_body = b''.join(self.body)
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: ^^^^^^^^^^^^^^^^^^^
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/local/lib/python3.11/dist-packages/cherrypy/_json.py", line 23, in encode
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: for chunk in _encode(value):
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/lib/python3.11/json/encoder.py", line 432, in _iterencode
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: yield from _iterencode_dict(o, _current_indent_level)
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/lib/python3.11/json/encoder.py", line 406, in _iterencode_dict
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: yield from chunks
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/lib/python3.11/json/encoder.py", line 439, in _iterencode
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: o = _default(o)
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: ^^^^^^^^^^^
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: File "/usr/lib/python3.11/json/encoder.py", line 180, in default
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: raise TypeError(f'Object of type {o.__class__.__name__} '
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,871 ERROR cherrypy.error.140418188164944: TypeError: Object of type bytes is not JSON serializable
May 06 22:58:44 zuul1001 docker[1045849]: 2026-05-06 22:58:44,874 INFO cherrypy.access.140418188164944: [req: 28d43b2ebf07420d8155a01efebf7cf7] 172.17.0.1 - - [06/May/2026:22:58:44] "GET /api/tenant/wikimedia/buildset/15fb63323838447081d51892400083c3 HTTP/1.1" 500 825 "https://zuul.wikimedia.org/t/wikimedia/buildset/15fb63323838447081d51892400083c3" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36"The current RBAC policies were setup with https://gitlab.wikimedia.org/repos/releng/zuul/tofu-provisioning/-/blob/main/modules/k8s_auth/main.tf
The primary error seems due to a bug in the upstream kubernetes driver. I've submitted a patch.
@Dzahn and I debugged the TypeError: Object of type bytes is not JSON serializable error yesterday based on a hunch that the database table's default binary charset was the root cause. We looked at the zuul_buildset and it was indeed created with a DEFAULT CHARSET=binary:
mysql:zuul@m1-master.eqiad.wmnet [zuul]> show create table zuul_buildset\G
*************************** 1. row ***************************
Table: zuul_buildset
Create Table: CREATE TABLE `zuul_buildset` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`pipeline` varbinary(255) DEFAULT NULL,
`message` blob DEFAULT NULL,
`tenant` varbinary(255) DEFAULT NULL,
`result` varbinary(255) DEFAULT NULL,
`uuid` varbinary(36) DEFAULT NULL,
`event_id` varbinary(255) DEFAULT NULL,
`event_timestamp` datetime DEFAULT NULL,
`first_build_start_time` datetime DEFAULT NULL,
`last_build_end_time` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
`queue_item_uuid` varbinary(36) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `zuul_buildset_tenant_idx` (`tenant`),
KEY `zuul_buildset_queue_item_uuid_idx` (`queue_item_uuid`),
KEY `zuul_buildset_uuid_idx` (`uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=binary
1 row in set (0.207 sec)Note the varbinary column types. Looking at the same database on zuul-1001.zuul3.eqiad1.wikimedia.cloud showed varchar types and a default charset of utf8mb4 so the hunch of this being the root cause seemed somewhat validated.
We thought that perhaps the ?charset=utf8mb4 in Zuul's dburi might have been the fix for that but was added only after the initial schema was created. Since we're not operational yet, I went ahead and dropped all the Zuul tables and let the auto database migration re-create them after restarting zuul-scheduler. However, the charset and types remained the same.
Instead of trying to solve the issue on the Zuul configuration side, I did:
ALTER DATABASE zuul CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
After dropping the tables again and letting Zuul re-create them following a zuul-sceduler restart, the error no longer occurs and the build and buildset pages now render their results. \o/ I confirmed that the zuul_buildset and other table charset's were correct and used varchar types where appropriate.
mysql:zuul@m1-master.eqiad.wmnet [zuul]> show create table zuul_buildset\G
*************************** 1. row ***************************
Table: zuul_buildset
Create Table: CREATE TABLE `zuul_buildset` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`pipeline` varchar(255) DEFAULT NULL,
`message` text DEFAULT NULL,
`tenant` varchar(255) DEFAULT NULL,
`result` varchar(255) DEFAULT NULL,
`uuid` varchar(36) DEFAULT NULL,
`event_id` varchar(255) DEFAULT NULL,
`event_timestamp` datetime DEFAULT NULL,
`first_build_start_time` datetime DEFAULT NULL,
`last_build_end_time` datetime DEFAULT NULL,
`updated` datetime DEFAULT NULL,
`queue_item_uuid` varchar(36) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `zuul_buildset_tenant_idx` (`tenant`),
KEY `zuul_buildset_uuid_idx` (`uuid`),
KEY `zuul_buildset_queue_item_uuid_idx` (`queue_item_uuid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
1 row in set (0.001 sec)Hopefully this is a suitable long-term fix.
Change #1285879 had a related patch set uploaded (by Dduvall; author: Dduvall):
[integration/config@master] zuul: Provide a `namespace` type label
Change #1285879 merged by jenkins-bot:
[integration/config@master] zuul: Provide a `namespace` type label
Change #1285887 had a related patch set uploaded (by Dduvall; author: Dduvall):
[integration/config@master] zuul: Remove `echo` job from gerrit-ping
Change #1285887 merged by jenkins-bot:
[integration/config@master] zuul: Remove `echo` job from gerrit-ping
@Dzahn thanks for fixing the SSH private key issues. On to the next error:
May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: [e: 7c100b55477943c98ba06ae3add95f3b] [req: d74ab49c80db4aaea374505a793b85ea] Marking node <KubernetesProviderNode uuid=80464b2263ec40bea8ba60821d007f4e, label=namespace, state=requested, provider=gerrit.wikimedia.org%2Fintegration%2Fconfig/untrusted> as failed May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: Traceback (most recent call last): May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/launcher/server.py", line 1913, in _processProviderNode May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: self._checkNodeCreate(node, log) May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/launcher/server.py", line 2128, in _checkNodeCreate May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: provider.getCreateStateMachine( May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/driver/kubernetes/kubernetesprovider.py", line 317, in getCreateStateMachine May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: return KubernetesCreateStateMachine( May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: File "/usr/local/lib/python3.11/dist-packages/zuul/driver/kubernetes/kubernetesendpoint.py", line 119, in __init__ May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: self.endpoint.core_client.api_client.configuration.host May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: ^^^^^^^^^^^^^^^^^^^^^^^^^ May 15 17:09:11 zuul1001 docker[1939097]: 2026-05-15 17:09:11,562 ERROR zuul.Launcher: AttributeError: 'KubernetesProviderEndpoint' object has no attribute 'core_client' May 15 17:09:12 zuul1001 docker[1939097]: 2026-05-15 17:09:12,507 ERROR zuul.Launcher: [e: 7c100b55477943c98ba06ae3add95f3b] [req: d74ab49c80db4aaea374505a793b85ea] Marking request <NodesetRequest uuid=d74ab49c80db4aaea374505a793b85ea, state=accepted, labels=['namespace'], path=/zuul/nodeset/requests/d74ab49c80db4aaea374505a793b85ea> as failed: No provider found for label namespace
Change #1287933 had a related patch set uploaded (by Dzahn; author: Dzahn):
[operations/puppet@production] zuul: let the launcher use the zuul user, not a separate one
Change #1287933 merged by Dzahn:
[operations/puppet@production] zuul: let the launcher use the zuul user, not a separate one
After we made the zuul-launcher also just use the "zuul" user instead of a separate launcher system user.. and then restarting the launcher. I now see:
May 15 20:03:13 zuul1001 docker[2203241]: 2026-05-15 20:03:13,184 INFO zuul.ComponentRegistry: Component scheduler dba26d3fea970000000965 updated: {'hostname': 'dba26>
May 15 20:03:13 zuul1001 docker[2203241]: 2026-05-15 20:03:13,186 INFO zuul.ComponentRegistry: Noticed new executor component 1a84797d06210000000017
May 15 20:03:13 zuul1001 docker[2203241]: 2026-05-15 20:03:13,187 INFO zuul.ComponentRegistry: Component executor 1a84797d06210000000017 updated: {'accepting_work': T>
May 15 20:03:13 zuul1001 docker[2203241]: 2026-05-15 20:03:13,192 INFO zuul.ComponentRegistry: Noticed new web component aa2d35c937140000000034
...New error, this time on the zuul-executor (zuul-launcher is fulfilling its part now and creating a namespace and running pod in WMCS).
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,993 ERROR zuul.AnsibleJob: [e: 79f2a5b95ac64647884964598ea65527] [build: 29ab26e692ea492e9f2d8d6ddec26161] Unable to start port forward:
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,993 ERROR zuul.AnsibleJob: Traceback (most recent call last):
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,993 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/dist-packages/zuul/executor/server.py", line 3024, in prepareNodes
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,993 ERROR zuul.AnsibleJob: fwd.start()
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,993 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/dist-packages/zuul/executor/server.py", line 498, in start
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,993 ERROR zuul.AnsibleJob: raise Exception("Unable to start kubectl port forward")
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,993 ERROR zuul.AnsibleJob: Exception: Unable to start kubectl port forward
May 15 20:33:04 zuul1002 docker[120902]: 2026-05-15 20:33:04,994 ERROR zuul.AnsibleJob: [e: 79f2a5b95ac64647884964598ea65527] [build: 29ab26e692ea492e9f2d8d6ddec26161] Kubectl and socat are required for streaming logs
May 15 20:33:23 zuul1002 docker[120902]: 2026-05-15 20:33:23,551 INFO zuul.AnsibleJob: [e: 79f2a5b95ac64647884964598ea65527] [build: 29ab26e692ea492e9f2d8d6ddec26161] Early unreachable hostChugga chugga chugga...
on Monday let's verify if socat is really available in the image?
We have already checked the RBAC and "PortForward" is a separate privilege that needs to be granted but it was already in there.
@dduvall Hi! I am going to be out of office until early September but if you have an update here meanwhile that would be great.
Like how were the tests going with the VMs? What did not go so well that lead us to getting the physical hardware instead over at T427353 ?
Also, wanted to let you know you can now test zuul1004.eqiad.wmnet (physical) and compare it to zuul1001.eqiad.wmnet (VM) both with the "zuul::main" role. I set it up with the puppet role (but nothing manual yet) and you have access, so feel free to use it as you like. The other 3 physical machines are there but have no zuul roles yet.
Also, was replacing VM with physical always just about the "main" machines or also about the executors and trusted builders?