Page MenuHomePhabricator

Reimage cp-upload nodes as cp-text nodes
Closed, ResolvedPublic

Description

As a result of T427465 (Move thumbnail caching from upload cluster to text) there will be a need to move some number of cp-nodes from the upload cluster, to the text cluster.

The example below moved cp3074.esams.wmnet from upload to text.

  • Depool node

Use confctl to depool and verify.

$ ssh puppetserver1001.eqiad.wmnet
$ sudo -i confctl select 'name=cp3074.esams.wmnet' set/pooled=no
$ sudo confctl select name=cp3074.esams.wmnet get
  • Reconfigure

Move the node in puppet (manifests/site.pp), see example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1331590

$ ssh puppetserver1001.eqiad.wmnet
$ tmux
$ sudo -i puppet-merge 

Update conftool:

$ ssh cumin1003.eqiad.wmnet
$ sudo -i cumin P:puppetserver "run-puppet-agent"
  • Reimage

cp hosts have been switched to EFI installs - any hosts that do not boot with EFI need to be reprovisioned. Check on puppetboard.

$ ssh cumin1003.eqiad.wmnet
$ tmux
$ sudo -i cookbook sre.hosts.provision --no-switch --no-users --no-dhcp cpXXXX  # If not UEFI booted
$ sudo -i cookbook sre.hosts.reimage --os trixie -t T436363 cpXXXX
  • Repool

Use confctl to verify that the node is now a text node and repool, if required.

$ ssh puppetserver1001.eqiad.wmnet
$ sudo -i confctl select name=cp3074.esams.wmnet get
$ sudo -i confctl select 'name=cp3074.esams.wmnet' set/pooled=yes

If the select command returns weight=0, we need to set that back to 1, for Liberica to pick up the node. This can be done using the following confctl command:

sudo -i confctl select 'name=cp7010.magru.wmnet' set/weight=1

For drmrs,

sudo -i confctl select 'name=cp6001.drmrs.wmnet,service=cdn' set/weight=1
sudo -i confctl select 'name=cp6001.drmrs.wmnet,service=ats-be' set/weight=1

You may get an alert about a diff in Liberica, see: https://wikitech.wikimedia.org/wiki/Liberica#LibericaDiffFPCheck and https://wikitech.wikimedia.org/wiki/Liberica#LibericaStaleConfig for resolving the issue.

To move:

magru

  • cp7009.magru.wmnet
  • cp7010.magru.wmnet

ulsfo

  • cp4045.ulsfo.wmnet
  • cp4046.ulsfo.wmnet

eqiad

  • cp1101.eqiad.wmnet
  • cp1103.eqiad.wmnet

codfw

  • cp2044.codfw.wmnet
  • cp2046.codfw.wmnet

esams

  • cp3074.esams.wmnet
  • cp3075.esams.wmnet

drmrs

  • cp6001.drmrs.wmnet
  • cp6002.drmrs.wmnet

eqsin

  • cp5025.eqsin.wmnet
  • cp5026.eqsin.wmnet

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1341951 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] site: Move cp7009 from upload to text

https://gerrit.wikimedia.org/r/1341951

Change #1341957 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] site: Move cp404[56] from upload to text

https://gerrit.wikimedia.org/r/1341957

Change #1341951 merged by BCornwall:

[operations/puppet@production] site: Move cp7009 from upload to text

https://gerrit.wikimedia.org/r/1341951

Change #1341958 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site.pp/conftool: move cp1101 to text from upload

https://gerrit.wikimedia.org/r/1341958

Change #1341959 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site.pp/conftool: move cp1103 to text from upload

https://gerrit.wikimedia.org/r/1341959

Change #1341958 merged by Ssingh:

[operations/puppet@production] site.pp/conftool: move cp1101 to text from upload

https://gerrit.wikimedia.org/r/1341958

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie

Change #1341957 merged by BCornwall:

[operations/puppet@production] site: Move cp404[56] from upload to text

https://gerrit.wikimedia.org/r/1341957

Change #1341970 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: Move cp3074 from upload to text

https://gerrit.wikimedia.org/r/1341970

Change #1341971 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: Move cp3075 from upload to text

https://gerrit.wikimedia.org/r/1341971

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie

Change #1341973 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] site: move cp2044 and cp2046 from upload to text

https://gerrit.wikimedia.org/r/1341973

Change #1341970 merged by Ssingh:

[operations/puppet@production] site: Move cp3074 from upload to text

https://gerrit.wikimedia.org/r/1341970

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp3074.esams.wmnet with OS trixie

Change #1341976 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: Move cp6001 from upload to text

https://gerrit.wikimedia.org/r/1341976

Change #1341977 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: Move cp6002 from upload to text

https://gerrit.wikimedia.org/r/1341977

Change #1341973 merged by CDobbins:

[operations/puppet@production] site: move cp2044 from upload to text

https://gerrit.wikimedia.org/r/1341973

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host cp2044.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie executed with errors:

  • cp4046 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp4046.ulsfo.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie executed with errors:

  • cp4045 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp4045.ulsfo.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie executed with errors:

  • cp1101 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp1101.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie completed:

  • cp7009 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151758_brett_2270898_cp7009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie executed with errors:

  • cp4045 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp4045.ulsfo.wmnet" to get a root shell, but depending on the failure this may not work.

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp3074.esams.wmnet with OS trixie completed:

  • cp3074 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151807_sukhe_2836325_cp3074.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host cp2044.codfw.wmnet with OS trixie completed:

  • cp2044 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151809_cdobbins_209639_cp2044.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change #1342004 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] site: move cp2046 from upload to text

https://gerrit.wikimedia.org/r/1342004

Change #1341976 abandoned by Ssingh:

[operations/puppet@production] site: Move cp6001 from upload to text

https://gerrit.wikimedia.org/r/1341976

Change #1341977 abandoned by Ssingh:

[operations/puppet@production] site: Move cp6002 from upload to text

https://gerrit.wikimedia.org/r/1341977

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp1101.eqiad.wmnet with OS trixie completed:

  • cp1101 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151830_sukhe_2840122_cp1101.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp3075.esams.wmnet with OS trixie

Change #1341971 merged by Ssingh:

[operations/puppet@production] site: Move cp3075 from upload to text

https://gerrit.wikimedia.org/r/1341971

Change #1341959 merged by Ssingh:

[operations/puppet@production] site.pp/conftool: move cp1103 to text from upload

https://gerrit.wikimedia.org/r/1341959

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp1103.eqiad.wmnet with OS trixie

Change #1342027 had a related patch set uploaded (by CDobbins; author: CDobbins):

[operations/puppet@production] site: move cp2046 from upload to text

https://gerrit.wikimedia.org/r/1342027

Change #1342004 abandoned by CDobbins:

[operations/puppet@production] site: move cp2046 from upload to text

Reason:

burning it down and starting over

https://gerrit.wikimedia.org/r/1342004

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4045.ulsfo.wmnet with OS trixie completed:

  • cp4045 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151918_brett_2294317_cp4045.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp4046.ulsfo.wmnet with OS trixie completed:

  • cp4046 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151927_brett_2295473_cp4046.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change #1342027 merged by CDobbins:

[operations/puppet@production] site: move cp2046 from upload to text

https://gerrit.wikimedia.org/r/1342027

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp7009.magru.wmnet with OS trixie completed:

  • cp7009 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151937_brett_2296252_cp7009.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1004 for host cp2046.codfw.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp3075.esams.wmnet with OS trixie completed:

  • cp3075 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151941_sukhe_2857062_cp3075.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp1103.eqiad.wmnet with OS trixie completed:

  • cp1103 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609151945_sukhe_2857411_cp1103.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2026-09-15T20:08:32Z] <brett@cumin2003> START - Cookbook sre.loadbalancer.admin config_reloading A:liberica-ulsfo (T436363)

Mentioned in SAL (#wikimedia-operations) [2026-09-15T20:10:33Z] <brett@cumin2003> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading A:liberica-ulsfo (T436363)

Mentioned in SAL (#wikimedia-operations) [2026-09-15T20:16:32Z] <brett@cumin2003> START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo (T436363)

Mentioned in SAL (#wikimedia-operations) [2026-09-15T20:19:53Z] <brett@cumin2003> END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo (T436363)

Change #1342039 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] site: Move cp6001 from upload to text

https://gerrit.wikimedia.org/r/1342039

Change #1342040 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] site: Move cp6002 from upload to text

https://gerrit.wikimedia.org/r/1342040

Change #1342039 merged by BCornwall:

[operations/puppet@production] site: Move cp6001 from upload to text

https://gerrit.wikimedia.org/r/1342039

@SLyngshede-WMF and @Fabfur: Be advised that we ran into those same annoying PXE boot problems we've had in the past on multiple hosts. We decided to make the partman EFI recipe the default for all cp hosts going forward. This means that we'll need to reprovision the server so it's set to UEFI booting instead of BIOS. I've updated the task description with instructions.

Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1004 for host cp2046.codfw.wmnet with OS trixie completed:

  • cp2046 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609152024_cdobbins_2866539_cp2046.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp6001.drmrs.wmnet with OS trixie

Also note that I had to upgrade the BIOS on cp6001 to get it to even properly set the bios values, so if the reprovisioning fails, perhaps an upgrade is in order...

Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp6001.drmrs.wmnet with OS trixie completed:

  • cp6001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609152133_brett_2329067_cp6001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2026-09-15T22:43:20Z] <brett@cumin2003> START - Cookbook sre.loadbalancer.upgrade restart A:liberica-drmrs (T436363)

Mentioned in SAL (#wikimedia-operations) [2026-09-15T22:46:57Z] <brett@cumin2003> END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-drmrs (T436363)

Change #1342114 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] site.pp move cp5025 from upload to text

https://gerrit.wikimedia.org/r/1342114

Change #1342115 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] site.pp move cp5026 from upload to text

https://gerrit.wikimedia.org/r/1342115

Change #1342127 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] site.pp: move cp6002 from upload to text

https://gerrit.wikimedia.org/r/1342127

Change #1342114 merged by Slyngshede:

[operations/puppet@production] site.pp move cp5025 from upload to text

https://gerrit.wikimedia.org/r/1342114

Icinga downtime and Alertmanager silence (ID=1f3524ce-e01c-441f-8517-6b989c2f4db7) set by slyngshede@cumin1003 for 4:00:00 on 1 host(s) and their services with reason: reimaging

cp5025.eqsin.wmnet

Icinga downtime and Alertmanager silence (ID=f93ce7f9-6cc1-4e08-afd7-61f4166200d0) set by slyngshede@cumin1003 for 2 days, 0:00:00 on 1 host(s) and their services with reason: reimaging

cp5025.eqsin.wmnet

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp5025.eqsin.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp5025.eqsin.wmnet with OS trixie completed:

  • cp5025 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609161139_slyngshede_327181_cp5025.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Change #1342115 merged by Slyngshede:

[operations/puppet@production] site.pp move cp5026 from upload to text

https://gerrit.wikimedia.org/r/1342115

Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1003 for host cp5026.eqsin.wmnet with OS trixie

Change #1342040 merged by Ssingh:

[operations/puppet@production] site: Move cp6002 from upload to text

https://gerrit.wikimedia.org/r/1342040

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp6002.drmrs.wmnet with OS trixie

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp6002.drmrs.wmnet with OS trixie completed:

  • cp6002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609161359_sukhe_3007666_cp6002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1003 for host cp5026.eqsin.wmnet with OS trixie completed:

  • cp5026 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202609161403_slyngshede_345766_cp5026.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
SLyngshede-WMF updated the task description. (Show Details)

Change #1331590 abandoned by Slyngshede:

[operations/puppet@production] site.pp move cp3074 from upload to text

Reason:

Done in separate patch

https://gerrit.wikimedia.org/r/1331590