Page MenuHomePhabricator
Create Task
Maniphest T424879

Upgrade to Zuul 14.x
Closed, ResolvedPublic
Actions

Description

The newest version of Zuul is now 14.x. We should upgrade. The most notable change is the introduction of zuul-launcher which replaces nodepool.

  • Build new images via https://gitlab.wikimedia.org/repos/releng/zuul/zuul
  • Bump Zuul service image refs in puppet
  • Define zuul.conf connection for WMCS k8s (using existing nodepool kube config as reference)
  • Add provider configuration to integration/config (using existing nodepool.yaml as reference)
  • Deploy new zuul-launcher via Puppet
  • Ensure zuul-nodepool absent
  • Remove zuul-nodepool from Puppet

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
bake: Default to newest Zuul 14.x and drop Nodepoolrepos/releng/zuul/zuul!14dduvallreview/zuul-14main
Customize query in GitLab

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDzahnT395938 puppetize setup of new zuul VMs
 ResolveddduvallT424879 Upgrade to Zuul 14.x

Event Timeline

dduvall created this task.Apr 29 2026, 5:18 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptApr 29 2026, 5:18 PM
dduvall changed the task status from Open to In Progress.Apr 29 2026, 5:18 PM
dduvall triaged this task as Medium priority.
dduvall moved this task from Backlog to In Progress on the Continuous-Integration-Infrastructure (Zuul upgrade) board.
Dzahn added a comment.Apr 29 2026, 7:21 PM

Oh, nodepool is already deprecated before we get to use it? :p Ok!

Let me make some changes to take care of:

Ensure zuul-nodepool absent
Remove zuul-nodepool from Puppet
gerritbot added a comment.Apr 29 2026, 7:30 PM

Change #1279461 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: remove zuul-nodepool config, user, stop service

https://gerrit.wikimedia.org/r/1279461

gerritbot added a project: Patch-For-Review.Apr 29 2026, 7:30 PM
gerritbot added a comment.Apr 29 2026, 7:45 PM

Change #1279470 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: create profile for new zuul-builder replacing nodepool

https://gerrit.wikimedia.org/r/1279470

CodeReviewBot added a comment.Apr 29 2026, 8:39 PM

dduvall opened https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/14

bake: Default to newest Zuul 14.x and drop Nodepool

CodeReviewBot added a comment.Apr 29 2026, 8:49 PM

dduvall merged https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/14

bake: Default to newest Zuul 14.x and drop Nodepool

dduvall updated the task description. (Show Details)Apr 29 2026, 11:06 PM
gerritbot added a comment.Apr 29 2026, 11:11 PM

Change #1279500 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] zuul: Upgrade to Zuul 14.2.0

https://gerrit.wikimedia.org/r/1279500

Dzahn added a comment.Apr 29 2026, 11:49 PM

Should I just remove nodepool any time or should it wait?

gerritbot added a comment.Apr 30 2026, 3:04 PM

Change #1279500 merged by Dzahn:

[operations/puppet@production] zuul: Upgrade to Zuul 14.2.0

https://gerrit.wikimedia.org/r/1279500

Dzahn added a comment.Apr 30 2026, 3:10 PM

restarted zuul-executor on zuul1002 after deploying version upgrade change:

Apr 30 15:09:19 zuul1002 docker[1886062]: 5e0b457beafe: Verifying Checksum
...
Apr 30 15:09:20 zuul1002 docker[1886062]: bc0c9b6444f9: Verifying Checksum
Apr 30 15:09:20 zuul1002 docker[1886062]: bc0c9b6444f9: Download complete
Apr 30 15:09:21 zuul1002 docker[1886062]: acf4d8e1c14d: Pull complete
Apr 30 15:09:21 zuul1002 docker[1886062]: 68a4a6c14700: Pull complete
...
Apr 30 15:10:33 zuul1002 docker[1886062]: Status: Downloaded newer image for docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-executor@sha256:850a8d82ed9d0e20c43836b47d119281b8d>
Dzahn added a comment.Apr 30 2026, 3:14 PM
[zuul1002:~] $ sudo docker ps
CONTAINER ID   IMAGE                                                                         COMMAND                  CREATED          STATUS          PORTS     NAMES
0c7a2e98b70d   docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-executor:14.2.0-1   "/usr/bin/dumb-init …"   30 seconds ago   Up 26 seconds             zuul-executor

same for zuul-web and zuul-scheduler on zuul1001. restarted. pulled new images. docker ps showing new version:

[zuul1001:~] $ sudo docker ps
CONTAINER ID   IMAGE                                                                                 COMMAND                  CREATED         STATUS         PORTS                                       NAMES
dcaaf98c2ac5   docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-scheduler:14.2.0-1          "/usr/bin/dumb-init …"   9 seconds ago   Up 8 seconds                                               zuul-scheduler
d257668fcf8d   docker-registry.wikimedia.org/repos/releng/zuul/zuul/zuul-web:14.2.0-1                "/usr/bin/dumb-init …"   9 seconds ago   Up 8 seconds   0.0.0.0:9000->9000/tcp, :::9000->9000/tcp   zuul-web
Dzahn updated the task description. (Show Details)Apr 30 2026, 3:14 PM
Stashbot added a comment.Apr 30 2026, 3:19 PM

Mentioned in SAL (#wikimedia-releng) [2026-04-30T15:19:10Z] <mutante> upgrading zuul to 14.2.0-1 on "new zuul" machines (T424879)

dduvall updated the task description. (Show Details)Apr 30 2026, 5:09 PM
gerritbot added a comment.Apr 30 2026, 8:53 PM

Change #1279461 merged by Dzahn:

[operations/puppet@production] zuul: remove zuul-nodepool config, user, stop service

https://gerrit.wikimedia.org/r/1279461

Dzahn updated the task description. (Show Details)Apr 30 2026, 9:04 PM

zuul-nodepool has been removed from zuul::main machines. (config, systemd unit, docker container not running, etc)

gerritbot added a comment.Apr 30 2026, 9:42 PM

Change #1280729 had a related patch set uploaded (by Dzahn; author: Dzahn):

[labs/private@master] zuul: rename nodepool::user_token to launcher::user_token

https://gerrit.wikimedia.org/r/1280729

gerritbot added a comment.Apr 30 2026, 9:43 PM

Change #1280729 merged by Dzahn:

[labs/private@master] zuul: rename nodepool::user_token to launcher::user_token

https://gerrit.wikimedia.org/r/1280729

gerritbot added a comment.Apr 30 2026, 11:30 PM

Change #1279470 merged by Dzahn:

[operations/puppet@production] zuul: create profile for new zuul-launcher replacing nodepool

https://gerrit.wikimedia.org/r/1279470

gerritbot added a comment.Apr 30 2026, 11:40 PM

Change #1280815 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: remove nodepool profile from zuul::main role

https://gerrit.wikimedia.org/r/1280815

gerritbot added a comment.Apr 30 2026, 11:43 PM

Change #1280815 merged by Dzahn:

[operations/puppet@production] zuul: remove nodepool profile from zuul::main role

https://gerrit.wikimedia.org/r/1280815

gerritbot added a comment.Apr 30 2026, 11:50 PM

Change #1280820 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: add placeholder template for launcher config

https://gerrit.wikimedia.org/r/1280820

gerritbot added a comment.Apr 30 2026, 11:55 PM

Change #1280820 merged by Dzahn:

[operations/puppet@production] zuul: add placeholder template for launcher config

https://gerrit.wikimedia.org/r/1280820

gerritbot added a comment.May 1 2026, 12:05 AM

Change #1280829 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: rename zuul-nodepool systemd template to zuul-launcher, adjust it

https://gerrit.wikimedia.org/r/1280829

gerritbot added a comment.May 1 2026, 12:07 AM

Change #1280829 merged by Dzahn:

[operations/puppet@production] zuul: rename zuul-nodepool systemd template to zuul-launcher, adjust it

https://gerrit.wikimedia.org/r/1280829

gerritbot added a comment.May 1 2026, 12:12 AM

Change #1280832 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: remove nodepool-related code

https://gerrit.wikimedia.org/r/1280832

gerritbot added a comment.May 1 2026, 12:19 AM

Change #1280832 merged by Dzahn:

[operations/puppet@production] zuul: remove nodepool-related code

https://gerrit.wikimedia.org/r/1280832

Dzahn updated the task description. (Show Details)May 1 2026, 12:21 AM
gerritbot added a comment.May 1 2026, 12:26 AM

Change #1280840 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: update zuul-launcher version to 14.2.0-1

https://gerrit.wikimedia.org/r/1280840

gerritbot added a comment.May 1 2026, 12:29 AM

Change #1280840 merged by Dzahn:

[operations/puppet@production] zuul: update zuul-launcher version to 14.2.0-1

https://gerrit.wikimedia.org/r/1280840

gerritbot added a comment.May 1 2026, 4:09 PM

Change #1281536 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] zuul: Mount /etc/zuul and /etc/zuul-launcher for zuul-launcher

https://gerrit.wikimedia.org/r/1281536

gerritbot added a comment.May 1 2026, 4:15 PM

Change #1281536 merged by Dzahn:

[operations/puppet@production] zuul: Mount /etc/zuul and /etc/zuul-launcher for zuul-launcher

https://gerrit.wikimedia.org/r/1281536

Dzahn updated the task description. (Show Details)May 1 2026, 4:34 PM
gerritbot added a comment.May 1 2026, 6:32 PM

Change #1281595 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: add launcher_connection Hiera key to executor role

https://gerrit.wikimedia.org/r/1281595

gerritbot added a comment.May 1 2026, 6:33 PM

Change #1281595 merged by Dzahn:

[operations/puppet@production] zuul: add launcher_connection Hiera key to executor role

https://gerrit.wikimedia.org/r/1281595

dduvall updated the task description. (Show Details)May 5 2026, 9:58 PM
gerritbot added a comment.May 5 2026, 9:58 PM

Change #1283098 had a related patch set uploaded (by Dduvall; author: Dduvall):

[integration/config@master] zuul: Define a WMCS/untrusted section and provider

https://gerrit.wikimedia.org/r/1283098

gerritbot added a comment.May 5 2026, 10:06 PM

Change #1283098 merged by jenkins-bot:

[integration/config@master] zuul: Define a WMCS/untrusted section and provider

https://gerrit.wikimedia.org/r/1283098

dduvall closed this task as Resolved.May 5 2026, 10:24 PM
dduvall updated the task description. (Show Details)
gerritbot added a comment.Jun 28 2026, 4:03 PM

Change #1306122 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] zuul: disable nodepool fallback

https://gerrit.wikimedia.org/r/1306122

gerritbot added a comment.Jun 29 2026, 10:31 PM

Change #1306122 merged by Dzahn:

[operations/puppet@production] zuul: disable nodepool fallback

https://gerrit.wikimedia.org/r/1306122

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits