Page MenuHomePhabricator
Create Task
Maniphest T405958

Q2:rack/setup/install ms-be209[0-4]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ms-be209[0-4]

Hostname / Racking / Installation Details

Hostnames: ms-be209[0-4]
Racking Proposal: Where should these systems be racked? Ideally avoiding racks with existing ms-be2* nodes
Networking Setup: # of Connections:1 - Speed: 10G. - VLAN:Private
OS Distro: Bullseye
Boot Method: UEFI
Sub-team Technical Contact: @MatthewVernon
Disk Setup: JBOD

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ms-be2090
  • Receive in system on procurement task T404771 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be2091
  • Receive in system on procurement task T404771 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be2092
  • Receive in system on procurement task T404771 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be2093
  • Receive in system on procurement task T404771 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be2094
  • Receive in system on procurement task T404771 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
swift: load/drain 5 codfw backends (h/w refresh)operations/puppetproduction+15 -5
install_server: set ms-be209* to use ms-be_simple-efi.cfg preseedoperations/puppetproduction+1 -1
Customize query in gerrit

Event Timeline

RobH created this task.Sep 29 2025, 6:59 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 29 2025, 6:59 PM
RobH assigned this task to MatthewVernon.Sep 29 2025, 6:59 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

@MatthewVernon,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH mentioned this in Unknown Object (Task).Sep 29 2025, 7:00 PM
Maintenance_bot added a project: SRE.Sep 29 2025, 7:29 PM
gerritbot added a comment.Sep 30 2025, 3:41 PM

Change #1192575 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] install_server: set ms-be209* to use ms-be_simple-efi.cfg preseed

https://gerrit.wikimedia.org/r/1192575

gerritbot added a project: Patch-For-Review.Sep 30 2025, 3:41 PM
gerritbot added a comment.Sep 30 2025, 4:09 PM

Change #1192575 merged by MVernon:

[operations/puppet@production] install_server: set ms-be209* to use ms-be_simple-efi.cfg preseed

https://gerrit.wikimedia.org/r/1192575

MatthewVernon removed MatthewVernon as the assignee of this task.Sep 30 2025, 4:09 PM

Done, thanks.

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2025, 4:32 PM
Jhancock.wm claimed this task.Nov 3 2025, 9:45 PM
Jhancock.wm updated the task description. (Show Details)
ops-monitoring-bot added a comment.Thu, Nov 6, 6:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2090.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Nov 6, 6:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2091.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Nov 6, 6:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2092.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Thu, Nov 6, 6:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2093.codfw.wmnet with OS bullseye

Jhancock.wm added a comment.Thu, Nov 6, 6:53 PM

note to self: 90 needs the mgmt connection checked for connectivity.

Jhancock.wm updated the task description. (Show Details)Thu, Nov 6, 6:54 PM
ops-monitoring-bot added a comment.Thu, Nov 6, 7:55 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2092.codfw.wmnet with OS bullseye completed:

  • ms-be2092 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511061937_jhancock_687882_ms-be2092.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Fri, Nov 7, 10:29 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2090.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2090 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511061927_jhancock_687821_ms-be2090.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2090.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Nov 7, 10:35 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2091.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2091 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511061933_jhancock_687834_ms-be2091.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2091.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Fri, Nov 7, 10:43 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2093.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2093 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511061939_jhancock_687913_ms-be2093.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2093.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Jhancock.wm updated the task description. (Show Details)Fri, Nov 7, 2:25 PM
ops-monitoring-bot added a comment.Mon, Nov 10, 6:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2090.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 10, 6:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2091.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 10, 6:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2093.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 10, 6:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2094.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 10, 7:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2094.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2094 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2094.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 11, 9:18 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2091.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2091 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511101824_jhancock_1780423_ms-be2091.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2091.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 11, 9:24 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2093.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2093 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511101828_jhancock_1780979_ms-be2093.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2093.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Tue, Nov 11, 10:01 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2090.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2090 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511101858_jhancock_1779986_ms-be2090.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2090.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Mon, Nov 17, 12:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be2090.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 17, 12:48 PM

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be2090.codfw.wmnet with OS bullseye completed:

  • ms-be2090 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511171229_mvernon_3046299_ms-be2090.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 17, 1:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be2091.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 17, 1:58 PM

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be2091.codfw.wmnet with OS bullseye completed:

  • ms-be2091 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511171339_mvernon_3081387_ms-be2091.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 17, 2:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be2092.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 17, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be2092.codfw.wmnet with OS bullseye completed:

  • ms-be2092 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511171429_mvernon_3109463_ms-be2092.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mon, Nov 17, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be2093.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 17, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be2093.codfw.wmnet with OS bullseye completed:

  • ms-be2093 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511171529_mvernon_3140922_ms-be2093.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MatthewVernon added a comment.Mon, Nov 17, 3:58 PM

@Jhancock.wm for each of ms-be209[0-3] the install was failing because puppet couldn't run, because one of the spinning disks had a vfat partition on containing an EFI setup. In each case, wiping that filesystem (and the partition table of the drive) unwedged puppet and I could then reimage them OK.

ms-be2094 seems to have broken much earlier in the image process, could you look at it again, please?

ops-monitoring-bot added a comment.Mon, Nov 17, 4:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2094.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mon, Nov 17, 6:03 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host ms-be2094.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2094 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2094.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Jhancock.wm added a comment.Mon, Nov 17, 6:05 PM

@MatthewVernon got the main issue with this one fixed. it failed cause it's trying to reach the wrong puppet server. Gonna take another crack this afternoon but feel free to take it away from me if you sign on before i finish it.

ops-monitoring-bot added a comment.Mon, Nov 17, 8:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host ms-be2094.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Tue, Nov 18, 8:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-be2094.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Tue, Nov 18, 9:23 AM

Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-be2094.codfw.wmnet with OS bullseye completed:

  • ms-be2094 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202511180904_mvernon_3682372_ms-be2094.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
MatthewVernon added a comment.Tue, Nov 18, 9:28 AM

@Jhancock.wm reimage had stalled again because puppet wasn't happy, again because of an EFI/vfat partition on one of the spinning disks

Notice: /Stage[main]/Profile::Swift::Storage::Configure_disks/Exec[mkfs-pci-0000:50:00.0-scsi-0:2:12:0]/returns: mkfs.xfs: /dev/disk/by-path/pci-0000:50:00.0-scsi-0:2:12:0-part1 appears to contain an existing filesystem (vfat).

I wiped that partition (and then partition table of the drive), then the reimage went OK.

Jhancock.wm closed this task as Resolved.Tue, Nov 18, 2:21 PM
Jhancock.wm updated the task description. (Show Details)

@MatthewVernon thanks for your help!

gerritbot added a comment.Tue, Dec 2, 3:22 PM

Change #1214077 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] swift: load/drain 5 codfw backends (h/w refresh)

https://gerrit.wikimedia.org/r/1214077

gerritbot added a project: Patch-For-Review.Tue, Dec 2, 3:22 PM
gerritbot added a comment.Tue, Dec 2, 4:33 PM

Change #1214077 merged by MVernon:

[operations/puppet@production] swift: load/drain 5 codfw backends (h/w refresh)

https://gerrit.wikimedia.org/r/1214077

Maintenance_bot removed a project: Patch-For-Review.Tue, Dec 2, 5:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits