Page MenuHomePhabricator
Create Task
Maniphest T393873

eqiad/codfw: 6 VM request for Zuul upgrade project
Closed, ResolvedPublic
Actions

Description

Cloud VPS Project Tested: zuul3 WMCS project
Site/Location: eqiad & codfw
Number of systems: 2
Service: zuul3 (main)
Networking Requirements: internal
Processor Requirements: 4
Memory: 8
Disks: 100GB
Other Requirements: n/a

Cloud VPS Project Tested: zuul3 WMCS project
Site/Location: eqiad & codfw
Number of systems: 2
Service: zuul3 (executor)
Networking Requirements: internal
Processor Requirements: 4
Memory: 2
Disks: 100GB
Other Requirements: n/a

Cloud VPS Project Tested: zuul3 WMCS project
Site/Location: eqiad & codfw
Number of systems: 2
Service: zuul3 (trusted runner)
Networking Requirements: internal
Processor Requirements: 4
Memory: 4
Disks: 100GB
Other Requirements: n/a

2 VMs - main zuul (8GB)

  • zuul1001
  • zuul2001

2 VMs - executor (2GB)

  • zuul1002
  • zuul2002

2 VMs - trusted runner (4GB)

  • zuul1003
  • zuul2003

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
installserver: add partman stanza for zuul* hostsoperations/puppetproduction+3 -0
site: add zuul VMs with collab-insetup-roleoperations/puppetproduction+5 -0
site: switch zuul3 VMs to use ferm insetup role, not nftablesoperations/puppetproduction+1 -1
site: broaden regex for zuul hosts to [12]00[1-3]operations/puppetproduction+1 -1
site: separate zuul regex, make it clear what is doing whatoperations/puppetproduction+14 -2
zuul: create role/profile for new zuul main servers, install docker.iooperations/puppetproduction+15 -1
zuul: let puppet manage docker service, install docker-composeoperations/puppetproduction+7 -1
zuul: enforce puppet7 on new zuul::main roleoperations/puppetproduction+1 -0
zuul::main: stop installing python docker-compose packageoperations/puppetproduction+1 -1
site/zuul: create skeleton role/profile for new zuul executors/runnersoperations/puppetproduction+42 -2
Customize query in gerrit

Related Objects

Event Timeline

Jelto created this task.May 12 2025, 9:13 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2025, 9:13 AM
Jelto added projects: collaboration-services, Release-Engineering-Team.May 12 2025, 9:14 AM
Jelto updated the task description. (Show Details)
Jelto added a subscriber: MoritzMuehlenhoff.
Paladox subscribed.May 12 2025, 11:42 AM
• joanna_borun triaged this task as Medium priority.May 12 2025, 2:34 PM
MoritzMuehlenhoff added a comment.May 13 2025, 10:17 AM

If we can pick freely, then let's use codfw/C.

thcipriani edited projects, added Release-Engineering-Team (Radar); removed Release-Engineering-Team.May 14 2025, 3:38 PM
Dzahn claimed this task.May 19 2025, 3:30 PM
Dzahn added a comment.EditedMay 19 2025, 6:29 PM

I will do this. But for both data centers. Not just one. We have said we do not want to create
single VMs again in the future. And doing so will just lead to follow-up work in the future.

(Almost certainly it will turn into actually used machines, not a temporary proof of concept.)

gerritbot added a comment.May 19 2025, 7:08 PM

Change #1147855 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: add zuul VMs with collab-insetup-role

https://gerrit.wikimedia.org/r/1147855

gerritbot added a project: Patch-For-Review.May 19 2025, 7:08 PM
gerritbot added a comment.May 19 2025, 7:28 PM

Change #1147855 merged by Dzahn:

[operations/puppet@production] site: add zuul VMs with collab-insetup-role

https://gerrit.wikimedia.org/r/1147855

Maintenance_bot removed a project: Patch-For-Review.May 19 2025, 7:30 PM
ops-monitoring-bot added a comment.May 19 2025, 7:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 19 2025, 8:24 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul2001.codfw.wmnet with OS bookworm executed with errors:

  • zuul2001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console zuul2001.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
gerritbot added a comment.May 19 2025, 8:47 PM

Change #1147878 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] installserver: add partman stanza for zuul* hosts

https://gerrit.wikimedia.org/r/1147878

gerritbot added a project: Patch-For-Review.May 19 2025, 8:47 PM
gerritbot added a comment.May 19 2025, 8:52 PM

Change #1147878 merged by Dzahn:

[operations/puppet@production] installserver: add partman stanza for zuul* hosts

https://gerrit.wikimedia.org/r/1147878

Maintenance_bot removed a project: Patch-For-Review.May 19 2025, 9:30 PM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.May 20 2025, 7:30 AM
Jelto added a comment.May 20 2025, 12:14 PM

Thanks @Dzahn for picking this up. I left a comment in operations/puppet/+/1147855. I don't think we can use nftables because we most likely use docker/containers to run zuul. So we have to use the insetup role without nftables unfortunately.

MoritzMuehlenhoff added a comment.May 20 2025, 1:57 PM
In T393873#10838820, @Jelto wrote:

Thanks @Dzahn for picking this up. I left a comment in operations/puppet/+/1147855. I don't think we can use nftables because we most likely use docker/containers to run zuul. So we have to use the insetup role without nftables unfortunately.

Indeed

gerritbot added a comment.May 20 2025, 2:34 PM

Change #1148359 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: switch zuul3 VMs to use ferm insetup role, not nftables

https://gerrit.wikimedia.org/r/1148359

gerritbot added a project: Patch-For-Review.May 20 2025, 2:34 PM
gerritbot added a comment.May 20 2025, 2:52 PM

Change #1148359 merged by Dzahn:

[operations/puppet@production] site: switch zuul3 VMs to use ferm insetup role, not nftables

https://gerrit.wikimedia.org/r/1148359

Dzahn added a comment.May 20 2025, 3:00 PM
In T393873#10838820, @Jelto wrote:

I don't think we can use nftables because we most likely use docker/containers to run zuul. So we have to use the insetup role without nftables unfortunately.

Yep, good point of course! Adjusted to ferm. Thanks

Maintenance_bot removed a project: Patch-For-Review.May 20 2025, 3:30 PM
Dzahn renamed this task from codfw: 1VM request for zuul3+ to eqiad/codfw: 6 VM request for zuul3+.May 20 2025, 4:22 PM
Dzahn updated the task description. (Show Details)
Dzahn added a comment.EditedMay 20 2025, 4:43 PM

after today's meetings with SRE-collab/releng/zuul author we update this request to 3 VMs per DC, for a total of 6

  • 1 machine for "main zuul components"
  • 1 machine as the executor
  • 1 machine to run trusted jobs

all with the same specs for now.

ops-monitoring-bot added a comment.May 20 2025, 4:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul2001.codfw.wmnet with OS bullseye

Dzahn mentioned this in T394819: give contint-roots access to new zuul VMs (was: create new admin group for "zuul devs").May 20 2025, 5:15 PM
Dzahn added a project: Continuous-Integration-Infrastructure (Zuul upgrade).May 20 2025, 5:23 PM
Dzahn moved this task from Backlog to In Progress on the Continuous-Integration-Infrastructure (Zuul upgrade) board.
ops-monitoring-bot added a comment.May 20 2025, 5:27 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul2001.codfw.wmnet with OS bullseye completed:

  • zuul2001 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505201711_dzahn_3962313_zuul2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.May 20 2025, 5:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul1001.eqiad.wmnet with OS bookworm

Dzahn updated the task description. (Show Details)May 20 2025, 5:55 PM
Dzahn updated the task description. (Show Details)May 20 2025, 5:58 PM
Dzahn updated the task description. (Show Details)
gerritbot added a comment.May 20 2025, 6:07 PM

Change #1148411 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: broaden regex for zuul hosts to [12]00[1-3]

https://gerrit.wikimedia.org/r/1148411

gerritbot added a project: Patch-For-Review.May 20 2025, 6:07 PM
gerritbot added a comment.May 20 2025, 6:10 PM

Change #1148411 merged by Dzahn:

[operations/puppet@production] site: broaden regex for zuul hosts to [12]00[1-3]

https://gerrit.wikimedia.org/r/1148411

ops-monitoring-bot added a comment.May 20 2025, 6:14 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul1001.eqiad.wmnet with OS bookworm completed:

  • zuul1001 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505201756_dzahn_3969838_zuul1001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Maintenance_bot removed a project: Patch-For-Review.May 20 2025, 6:31 PM
ops-monitoring-bot added a comment.May 20 2025, 7:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul1002.eqiad.wmnet with OS bookworm

Dzahn mentioned this in T394838: allow new zuul executor VMs in prod to talk to cloud VPS.May 20 2025, 7:23 PM
ops-monitoring-bot added a comment.May 20 2025, 7:52 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul1002.eqiad.wmnet with OS bookworm completed:

  • zuul1002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505201936_dzahn_3982637_zuul1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Dzahn mentioned this in T394844: Request mariadb database for Zuul.May 20 2025, 8:22 PM
Dzahn updated the task description. (Show Details)May 20 2025, 9:10 PM
Dzahn updated the task description. (Show Details)
Dzahn updated the task description. (Show Details)May 20 2025, 10:05 PM
ops-monitoring-bot added a comment.May 20 2025, 10:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 20 2025, 10:47 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul1003.eqiad.wmnet with OS bookworm completed:

  • zuul1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505202232_dzahn_4011770_zuul1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
hashar renamed this task from eqiad/codfw: 6 VM request for zuul3+ to eqiad/codfw: 6 VM request for Zuul upgrade project.May 21 2025, 6:52 AM
Dzahn closed this task as Resolved.May 21 2025, 3:55 PM
Dzahn updated the task description. (Show Details)

6 VMs have been created:

2 VMs - main zuul (8GB)

zuul1001
zuul2001

2 VMs - executor (2GB)

zuul1002
zuul2002

2 VMs - trusted runner (4GB)

zuul1003
zuul2003
gerritbot added a comment.May 21 2025, 4:01 PM

Change #1148902 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: separate zuul regex, make it clear what is doing what

https://gerrit.wikimedia.org/r/1148902

gerritbot added a project: Patch-For-Review.May 21 2025, 4:01 PM
gerritbot added a comment.May 21 2025, 6:12 PM

Change #1148902 merged by Dzahn:

[operations/puppet@production] site: separate zuul regex, make it clear what is doing what

https://gerrit.wikimedia.org/r/1148902

gerritbot added a comment.May 21 2025, 6:14 PM

Change #1148930 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: create basic role/profile for zuul::man and install docker.io

https://gerrit.wikimedia.org/r/1148930

gerritbot added a comment.May 22 2025, 6:59 PM

Change #1148930 merged by Dzahn:

[operations/puppet@production] zuul: create role/profile for new zuul main servers, install docker.io

https://gerrit.wikimedia.org/r/1148930

Maintenance_bot removed a project: Patch-For-Review.May 22 2025, 7:30 PM
gerritbot added a comment.May 22 2025, 8:07 PM

Change #1149474 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: let puppet manage docker service, install docker-compose

https://gerrit.wikimedia.org/r/1149474

gerritbot added a project: Patch-For-Review.May 22 2025, 8:07 PM
gerritbot added a comment.May 22 2025, 8:30 PM

Change #1149474 merged by Dzahn:

[operations/puppet@production] zuul: let puppet manage docker service, install docker-compose

https://gerrit.wikimedia.org/r/1149474

Maintenance_bot removed a project: Patch-For-Review.May 22 2025, 8:31 PM
gerritbot added a comment.May 22 2025, 8:32 PM

Change #1149476 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: enforce puppet7 on new zuul::main role

https://gerrit.wikimedia.org/r/1149476

gerritbot added a project: Patch-For-Review.May 22 2025, 8:32 PM
gerritbot added a comment.May 22 2025, 8:52 PM

Change #1149476 merged by Dzahn:

[operations/puppet@production] zuul: enforce puppet7 on new zuul::main role

https://gerrit.wikimedia.org/r/1149476

Maintenance_bot removed a project: Patch-For-Review.May 22 2025, 9:30 PM
MoritzMuehlenhoff reopened this task as Open.May 27 2025, 7:25 AM

zuul2001 is running Bullseye, that seems like a mistake? The other five are on Bookworm

Dzahn added a comment.May 27 2025, 3:59 PM

uhm, yea, that is a mistake. not sure how that happened since I think i just went "cursor up" to edit my cookbook commands :)

thanks for pointing it out.

ops-monitoring-bot added a comment.May 27 2025, 4:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host zuul2001.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.May 27 2025, 5:28 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host zuul2001.codfw.wmnet with OS bookworm completed:

  • zuul2001 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Set boot media to disk
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202505271711_dzahn_3409845_zuul2001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.May 27 2025, 5:41 PM

Change #1151279 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site/zuul: create skeleton role/profile for new zuul executors/runners

https://gerrit.wikimedia.org/r/1151279

gerritbot added a project: Patch-For-Review.May 27 2025, 5:41 PM
Dzahn added a comment.May 27 2025, 5:47 PM

zuul2001 has been reimaged with bookworm now.

gerritbot added a comment.May 27 2025, 5:57 PM

Change #1151281 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul::main: stop installing python docker-compose package

https://gerrit.wikimedia.org/r/1151281

gerritbot added a comment.May 27 2025, 5:58 PM

Change #1151279 merged by Dzahn:

[operations/puppet@production] site/zuul: create skeleton role/profile for new zuul executors/runners

https://gerrit.wikimedia.org/r/1151279

Dzahn added a comment.May 27 2025, 6:02 PM

new roles created and added to https://wikitech.wikimedia.org/wiki/SRE/Infrastructure_naming_conventions#Hostname_prefixes

Dzahn closed this task as Resolved.May 27 2025, 6:02 PM
gerritbot added a comment.May 27 2025, 6:03 PM

Change #1151281 merged by Dzahn:

[operations/puppet@production] zuul::main: stop installing python docker-compose package

https://gerrit.wikimedia.org/r/1151281

Maintenance_bot removed a project: Patch-For-Review.May 27 2025, 6:31 PM
Dzahn mentioned this in T395938: puppetize setup of new zuul VMs.Jun 3 2025, 5:20 PM

Further setup from here on will be part of T395938 to avoid amending to the VM request again and again.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits