Page MenuHomePhabricator
Create Task
Maniphest T400198

Q1:rack/setup/install es1049-es1057
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of es1049-es1057

Hostname / Racking / Installation Details

Hostnames: es1049 es1050 es1051 es1052 es1053 es1054 es1055 es1056 es1057
Racking Proposal: Ideally scattered across rows and racks, and if possible do not share it with other hosts.
Networking Setup: # of Connections:1/2 - Speed:1G. - VLAN:Private/Public/Other(Specify) :
OS Distro: Bookworm
Boot Method: Legacy BIOS or UEFI. Please note UEFI must have partman updates applied in advance of setup and is currently in pilot program: https://wikitech.wikimedia.org/wiki/UEFI_Boot
Sub-team Technical Contact: @Marostegui

Per host setup checklist

es1049
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1050
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1051
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1052
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1053
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1054
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1055
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1056
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
es1057
  • Receive in system on procurement task T398512 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add es1056 to preseed.yamloperations/puppetproduction+1 -1
mariadb: Productionze es1049-es1057operations/puppetproduction+25 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
VRiley-WMF updated the task description. (Show Details)Aug 18 2025, 11:04 AM
Jclark-ctr reassigned this task from Marostegui to VRiley-WMF.Aug 20 2025, 12:55 PM
VRiley-WMF added a comment.Aug 26 2025, 12:01 AM

These have been added into Netbox

VRiley-WMF added a comment.Aug 27 2025, 2:55 PM

Verified to try to avoid any racks that currently have es hosts.

VRiley-WMF added a comment.Aug 27 2025, 8:30 PM

es1049 - rack A5, U06
es1050 - rack B6, U30
es1051 - rack D1, U11
es1052 - rack D3, U08
es1053 - rack D6, U09

these have been racked. @Ladsgroup at this point, we will need to put the rest of these servers in racks that also have es hosts. Unless there are any units from other racks that could be decommed? Let me know, thank you!

Ladsgroup added a comment.Aug 29 2025, 11:09 AM

These are refreshes so they should replace es1026-es1034. Feel free to rack these where the old ones are.

VRiley-WMF added a comment.EditedAug 29 2025, 10:19 PM

es1049 - rack A5, U06
es1050 - rack B6, U30
es1051 - rack D1, U11
es1052 - rack D3, U08
es1053 - rack D6, U09
es1054 - rack A1, U09
es1055 - rack A3, U16
es1056 - rack B1, U15

Racked a few more

VRiley-WMF updated the task description. (Show Details)Aug 29 2025, 10:20 PM
VRiley-WMF added a comment.Sep 4 2025, 6:35 PM

es1057 - rack C3, U14

VRiley-WMF updated the task description. (Show Details)Sep 5 2025, 10:51 PM
ops-monitoring-bot added a comment.Sep 5 2025, 11:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1049.eqiad.wmnet with OS bookworm

VRiley-WMF updated the task description. (Show Details)Sep 5 2025, 11:45 PM
ops-monitoring-bot added a comment.Sep 6 2025, 12:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1050.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 6 2025, 12:17 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1049.eqiad.wmnet with OS bookworm completed:

  • es1049 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509052358_vriley_1209524_es1049.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF updated the task description. (Show Details)Sep 6 2025, 12:19 AM
ops-monitoring-bot added a comment.Sep 6 2025, 1:07 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1050.eqiad.wmnet with OS bookworm completed:

  • es1050 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509060043_vriley_1213303_es1050.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 7 2025, 11:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1051.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 12:02 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1050.eqiad.wmnet with OS bookworm

VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 12:09 AM
ops-monitoring-bot added a comment.Sep 8 2025, 12:21 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1051.eqiad.wmnet with OS bookworm completed:

  • es1051 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509080003_vriley_1490313_es1051.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 8 2025, 12:43 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1050.eqiad.wmnet with OS bookworm executed with errors:

  • es1050 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1050.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 1:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1052.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 2:12 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1052.eqiad.wmnet with OS bookworm completed:

  • es1052 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509080153_vriley_1502205_es1052.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 2:54 AM
VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 3:35 AM
VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 4:13 AM
ops-monitoring-bot added a comment.Sep 8 2025, 4:27 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1053.eqiad.wmnet with OS bookworm

VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 4:27 AM
ops-monitoring-bot added a comment.Sep 8 2025, 4:32 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 5:17 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1053.eqiad.wmnet with OS bookworm completed:

  • es1053 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509080458_vriley_1524177_es1053.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 8 2025, 5:22 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1055.eqiad.wmnet with OS bookworm

VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 5:29 AM
ops-monitoring-bot added a comment.Sep 8 2025, 5:32 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm executed with errors:

  • es1054 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1054.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 5:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 5:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 5:47 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1057.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 6:01 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm executed with errors:

  • es1054 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1054.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 6:16 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1055.eqiad.wmnet with OS bookworm completed:

  • es1055 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509080554_vriley_1533085_es1055.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 8 2025, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1057.eqiad.wmnet with OS bookworm completed:

  • es1057 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509080618_vriley_1534252_es1057.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 4:18 PM
ops-monitoring-bot added a comment.Sep 8 2025, 4:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 4:24 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 4:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 5:09 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm executed with errors:

  • es1054 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1054.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 5:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 5:26 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

VRiley-WMF updated the task description. (Show Details)Sep 8 2025, 5:43 PM
ops-monitoring-bot added a comment.Sep 8 2025, 6:06 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1054.eqiad.wmnet with OS bookworm completed:

  • es1054 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509081743_vriley_1656553_es1054.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Sep 8 2025, 6:22 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 8 2025, 6:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 8 2025, 7:03 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
VRiley-WMF updated the task description. (Show Details)Sep 9 2025, 10:52 AM
VRiley-WMF added a comment.Sep 9 2025, 10:56 AM

es1056 seems to be having issues imagining. Looking into this.

ops-monitoring-bot added a comment.Sep 9 2025, 11:33 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 9 2025, 12:59 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 9 2025, 8:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 9 2025, 8:36 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 11 2025, 5:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 11 2025, 6:13 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 11 2025, 6:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

VRiley-WMF added a subscriber: Papaul.Sep 11 2025, 7:55 PM

Only server left is es1056, which is giving me a strange error. Looking into this with @Papaul Was informed to check to see if in BIOS all disks are being seen, which they are.

es1056.png (568×987 px, 19 KB)

Papaul added a comment.Sep 12 2025, 11:35 PM

@VRiley-WMF the issue is that es1056 is missing in the this patch
https://gerrit.wikimedia.org/r/c/operations/puppet/+/1172182/1/modules/profile/data/profile/installserver/preseed.yaml
you can add es1056 and add me to +2 your code.

gerritbot added a comment.Sep 15 2025, 9:51 PM

Change #1188460 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add es1056 to preseed.yaml

https://gerrit.wikimedia.org/r/1188460

gerritbot added a project: Patch-For-Review.Sep 15 2025, 9:51 PM
gerritbot added a comment.Sep 15 2025, 9:53 PM

Change #1188460 merged by Papaul:

[operations/puppet@production] Add es1056 to preseed.yaml

https://gerrit.wikimedia.org/r/1188460

Papaul added a comment.Sep 15 2025, 10:01 PM

@VRiley-WMF es1056 added, you can resume with your install.

Maintenance_bot removed a project: Patch-For-Review.Sep 15 2025, 10:31 PM
ops-monitoring-bot added a comment.Sep 16 2025, 4:47 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm executed with errors:

  • es1056 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1056.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 16 2025, 4:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 16 2025, 5:39 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1003 for host es1056.eqiad.wmnet with OS bookworm completed:

  • es1056 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202509161719_vriley_2844030_es1056.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF closed this task as Resolved.Sep 16 2025, 5:41 PM
VRiley-WMF updated the task description. (Show Details)
Marostegui reopened subtask T406488: Productionize es1049 - es1057 as Open.Oct 6 2025, 1:28 PM
Marostegui reopened this task as Open.Oct 7 2025, 5:48 AM

@VRiley-WMF es1050 doesn't seem to be installed correctly, I am investigating

ops-monitoring-bot added a comment.Oct 7 2025, 5:52 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 7 2025, 6:35 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm executed with errors:

  • es1050 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1050.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 7 2025, 6:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 7 2025, 7:10 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm executed with errors:

  • es1050 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1050.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Oct 7 2025, 7:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 7 2025, 7:33 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm executed with errors:

  • es1050 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1050.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Marostegui added a comment.Oct 7 2025, 7:34 AM

The first issue was that the host was running puppet 5.
Now after running the whole process again, it got stuck on the installer, which is taking ages and timedout

ops-monitoring-bot added a comment.Oct 7 2025, 7:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm

Marostegui added a comment.Oct 7 2025, 8:08 AM

The issue is that the host keeps booting into the installer on a loop even if I disabled the PXE boot via IPMI

ops-monitoring-bot added a comment.Oct 7 2025, 8:20 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm executed with errors:

  • es1050 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console es1050.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
elukey subscribed.Oct 7 2025, 8:39 AM

re-run provisioning:

Updated value for attribute BIOS.Setup.1-1 -> SetBootOrderEn: NIC.Embedded.1-1-1,HardDisk.List.1-1 => HardDisk.List.1-1,NIC.Embedded.1-1-1
Updated value for attribute BIOS.Setup.1-1 -> BiosBootSeq (marked Set On Import to True): NIC.Embedded.1-1-1, HardDisk.List.1-1 => HardDisk.List.1-1, NIC.Embedded.1-1-1
ops-monitoring-bot added a comment.Oct 7 2025, 8:44 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Oct 7 2025, 9:26 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1003 for host es1050.eqiad.wmnet with OS bookworm completed:

  • es1050 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510070907_marostegui_1645063_es1050.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Marostegui closed this task as Resolved.Oct 7 2025, 9:26 AM

es1050 has been successfully reimaged and it is reachable now - thanks @elukey for all the help

Marostegui closed subtask T406488: Productionize es1049 - es1057 as Resolved.Oct 17 2025, 7:45 AM
Jclark-ctr mentioned this in T407851: Audit Eqiad racks for variance from Netbox.Oct 21 2025, 12:51 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits