Page MenuHomePhabricator
Create Task
Maniphest T400195

Q1:rack/setup/install es2049-es2057
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of es2049-es2055

Hostname / Racking / Installation Details

Hostnames: es2049.codfw.wmnet es2050.codfw.wmnet es2051.codfw.wmnet es2052.codfw.wmnet es2053.codfw.wmnet es2054.codfw.wmnet es2055.codfw.wmnet es2056.codfw.wmnet es2057.codfw.wmnet
Racking Proposal: Ideally scattered across rows and racks, and if possible do not share it with other hosts.
Networking Setup: # of Connections:1/2 - Speed:10G. - VLAN:Private/Public/Other(Specify) :
OS Distro: Bookworm
Boot Method: Legacy BIOS or UEFI. Please note UEFI must have partman updates applied in advance of setup and is currently in pilot program: https://wikitech.wikimedia.org/wiki/UEFI_Boot
Sub-team Technical Contact: @Marostegui

Per host setup checklist

es2049
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2050
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2051
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2052
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2053
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2054
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2055
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2056
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es2057
  • Receive in system on procurement task T398511 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mariadb: Productionze es2049-es2057operations/puppetproduction+23 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedFCeratto-WMFT402859 Productionize es2049-es2057
 ResolvedJhancock.wmT400195 Q1:rack/setup/install es2049-es2057

Event Timeline

RobH created this task.Jul 22 2025, 5:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 22 2025, 5:59 PM
RobH assigned this task to • Marostegui.Jul 22 2025, 5:59 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

@Marostegui,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH renamed this task from Q1:rack/setup/install es2049-es2053 to Q1:rack/setup/install es2049-es2055.Jul 22 2025, 6:02 PM
RobH updated the task description. (Show Details)
RobH renamed this task from Q1:rack/setup/install es2049-es2055 to Q1:rack/setup/install es2049-es2057.Jul 22 2025, 6:04 PM
RobH updated the task description. (Show Details)

@Marostegui please note the racking details didn't list 9 hostnames, but the order is for 9 hosts. I've appended 2 additional hostnames to the list.

Maintenance_bot added a project: SRE.Jul 22 2025, 6:29 PM
gerritbot added a comment.Jul 24 2025, 6:57 AM

Change #1172192 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Productionze es2049-es2057

https://gerrit.wikimedia.org/r/1172192

gerritbot added a project: Patch-For-Review.Jul 24 2025, 6:57 AM
gerritbot added a comment.Jul 24 2025, 7:00 AM

Change #1172192 merged by Marostegui:

[operations/puppet@production] mariadb: Productionze es2049-es2057

https://gerrit.wikimedia.org/r/1172192

• Marostegui added a comment.Jul 24 2025, 7:01 AM

Patches done

• Marostegui updated the task description. (Show Details)Jul 24 2025, 7:01 AM
Maintenance_bot removed a project: Patch-For-Review.Jul 24 2025, 7:30 AM
Jhancock.wm updated the task description. (Show Details)Aug 11 2025, 4:38 PM
ops-monitoring-bot added a comment.Aug 11 2025, 6:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host es2049.codfw.wmnet with OS bookworm

Jhancock.wm added subscribers: Papaul, Jhancock.wm.Aug 11 2025, 7:39 PM

@Papaul install of the os went through without issue on es2049. But looks like it's going to fail here. I did find entries for the servers in site and preseed files.

[39/50, retrying in 117.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb.<locals>.poll_puppetdb' raised: Nagios_host resource with title es2049 not found yet
[40/50, retrying in 120.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb.<locals>.poll_puppetdb' raised: Nagios_host resource with title es2049 not found yet
ops-monitoring-bot added a comment.Aug 11 2025, 7:57 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host es2049.codfw.wmnet with OS bookworm executed with errors:

  • es2049 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2049.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Papaul added a comment.Aug 11 2025, 8:13 PM

@Jhancock.wm indeed the server is already in site.pp and also i dno't see it's puppet cert on the wrong puppet server. so i need to look more and see why it did failed.

Thanks

ops-monitoring-bot added a comment.Aug 14 2025, 4:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host es2050.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 14 2025, 5:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host es2050.codfw.wmnet with OS bookworm executed with errors:

  • es2050 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2050.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 14 2025, 7:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host es2049.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 14 2025, 8:05 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host es2049.codfw.wmnet with OS bookworm executed with errors:

  • es2049 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2049.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 14 2025, 9:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host es2049.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 14 2025, 10:09 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host es2049.codfw.wmnet with OS bookworm completed:

  • es2049 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508142148_pt1979_4154498_es2049.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul added a subscriber: Dzahn.Aug 14 2025, 10:17 PM

We were getting the error below while re-image es2049 my thinking was that the entry in site.pp for the new es host was not right to be sure i ping @Dzahn to double check also if that was the issue.

`
2025-08-14 19:01:44,957 pt1979 4054560 [DEBUG conftool:72 in query]
               label db2187 did not match regex ^es2049$
2025-08-14 19:01:44,957 pt1979 4054560 [DEBUG conftool:72 in query]
               label db2178 did not match regex ^es2049$
 2025-08-14 19:01:44,958 pt1979 4054560 [DEBUG conftool:72 in query]
               label db2180 did not match regex ^es2049$
 2025-08-14 19:01:44,958 pt1979 4054560 [DEBUG conftool:72 in query]
               label db2143 did not match regex ^es2049$
 2025-08-14 19:01:44,958 pt1979 4054560 [DEBUG conftool:72 in query]
               label es2040 did not match regex ^es2049$

We changed

node /^es20[49|50|51|52|53|54|55|56|57]\.codfw\./ {
node /^es20(49|50|51|52|53|54|55|56|57)\.codfw\./ {
role(insetup::data_persistence_ferm)

After the change es2049 install completed. @Jhancock.wm you can resume with the install thanks. Thanks @Dzahn

Papaul updated the task description. (Show Details)Aug 15 2025, 2:55 PM
Jhancock.wm claimed this task.Aug 15 2025, 3:02 PM
Jhancock.wm updated the task description. (Show Details)
ops-monitoring-bot added a comment.Aug 15 2025, 10:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host es2050.codfw.wmnet with OS bookworm

Jhancock.wm updated the task description. (Show Details)Aug 15 2025, 10:57 PM
ops-monitoring-bot added a comment.Aug 15 2025, 10:58 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host es2050.codfw.wmnet with OS bookworm completed:

  • es2050 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508152242_jhancock_3951611_es2050.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Jhancock.wm updated the task description. (Show Details)Aug 15 2025, 11:29 PM
ops-monitoring-bot added a comment.Aug 16 2025, 12:06 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2051.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 12:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2052.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 12:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2053.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 12:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2054.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 12:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2055.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 12:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2056.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 12:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2057.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 12:48 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2051.codfw.wmnet with OS bookworm completed:

  • es2051 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160028_jhancock_2208791_es2051.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 16 2025, 12:51 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2057.codfw.wmnet with OS bookworm completed:

  • es2057 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160032_jhancock_2208977_es2057.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 16 2025, 12:53 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2054.codfw.wmnet with OS bookworm completed:

  • es2054 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160034_jhancock_2208873_es2054.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 16 2025, 12:54 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2056.codfw.wmnet with OS bookworm completed:

  • es2056 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160038_jhancock_2208948_es2056.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 16 2025, 1:00 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2053.codfw.wmnet with OS bookworm completed:

  • es2053 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160041_jhancock_2208824_es2053.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 16 2025, 1:03 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2055.codfw.wmnet with OS bookworm executed with errors:

  • es2055 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160045_jhancock_2208898_es2055.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es2055.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Aug 16 2025, 1:08 AM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host es2055.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Aug 16 2025, 1:11 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2052.codfw.wmnet with OS bookworm completed:

  • es2052 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160052_jhancock_2208801_es2052.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Aug 16 2025, 1:24 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host es2055.codfw.wmnet with OS bookworm completed:

  • es2055 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202508160114_jhancock_2227129_es2055.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Jhancock.wm closed this task as Resolved.Aug 16 2025, 1:42 AM
Jhancock.wm updated the task description. (Show Details)

@Marostegui these are ready!

Ladsgroup subscribed.Aug 17 2025, 12:59 PM
In T400195#11091192, @Jhancock.wm wrote:

@Marostegui these are ready!

Thanks! Manuel is out. I will try to start getting them into production.

FCeratto-WMF mentioned this in T402859: Productionize es2049-es2057.Aug 25 2025, 8:39 PM
FCeratto-WMF added a parent task: T402859: Productionize es2049-es2057.
Jhancock.wm mentioned this in T400485: Q1:rack/setup/install deploy2003.Aug 29 2025, 2:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits