Page MenuHomePhabricator
Create Task
Maniphest T371389

Q1:rack/setup/install ms-be10{83-91}
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ms-be10{83-91}

Hostname / Racking / Installation Details

Hostnames: ms-be10{83-91}
Racking Proposal: Where should these systems be racked? Can they share with any existing systems or should they avoid any other systems sharing their rack or row? Spread across rows as evenly as possible.
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private/: AAAA records:Y, Additional IP records (Cassandra)? No
Partitioning/Raid: All JBOD
OS Distro: bullseye
Sub-team Technical Contact: @MatthewVernon

Per host setup checklist

ms-be1083
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1084
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1085
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1086
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1087
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1088
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1089
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1090
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
ms-be1091
  • Receive in system on procurement task T368930 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Check for sub-team update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
preseed - use ms-be_simple-efi.cfg for new SM Config-J nodesoperations/puppetproduction+3 -3
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT371389 Q1:rack/setup/install ms-be10{83-91}

Event Timeline

RobH created this task.Jul 30 2024, 2:20 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH assigned this task to MatthewVernon.Jul 30 2024, 2:22 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

@MatthewVernon,

Please note there has been a slight change in the workflow for racking and installing hosts. The DC ops team, as a whole, does not have merge rights/root on the puppet repo. So we're now assigning racking tasks to the SRE sub-team at time of hardware order, which gives each sub-team a week or two to push the puppet repo updates for site.pp and preseed.yaml. Please update site.pp with these new hosts with the insetup role (not its final role) and update preseed.yaml with the paritition info.

Once you have merged these changes live (please feel free to reference this task in the patchset), please unassign yourself as task assignee and leave unasisgned in the 'racking tasks' column of ops-eqiad.

Thanks!

RobH unsubscribed.Jul 30 2024, 2:23 PM
Maintenance_bot added a project: SRE.Jul 30 2024, 2:29 PM
MatthewVernon removed MatthewVernon as the assignee of this task.Jul 30 2024, 2:47 PM

Already done via this change, thanks.

MatthewVernon mentioned this in T372207: Disk (sdc) failed on ms-be1058.Aug 13 2024, 9:15 AM
VRiley-WMF subscribed.Oct 31 2024, 11:35 AM

ms-be1083
Rack A4
U 37
Port 10
CableID: 202431

ms-be1085
Rack: B2
U 11
Port 7
CableID 5073

ms-be1084
Rack: B4
U 12
Port 18
CableID 4777

ms-be1086
Rack: C2
U 17
Port 15
CableID 5206

ms-be1091
Rack: D2
U
Port 17
CableID 5098

ms-be1090
Rack: E2
U 5
Port 2
CableID 230304500085

ms-be1089
Rack E7
U 3
Port 21
CableID 230304500084

ms-be1088
Rack F2
U 5
Port 2
CableID 230304500078

ms-be1087
Rack F6
U 3
Port 21
CableID 230304500088

gerritbot added a comment.Nov 6 2024, 5:06 PM

Change #1087949 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] preseed - use ms-be_simple-efi.cfg for new SM Config-J nodes

https://gerrit.wikimedia.org/r/1087949

gerritbot added a project: Patch-For-Review.Nov 6 2024, 5:06 PM
gerritbot added a comment.Nov 6 2024, 5:32 PM

Change #1087949 merged by MVernon:

[operations/puppet@production] preseed - use ms-be_simple-efi.cfg for new SM Config-J nodes

https://gerrit.wikimedia.org/r/1087949

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2024, 6:30 PM
elukey added subscribers: Jclark-ctr, elukey.Nov 27 2024, 9:28 AM

@VRiley-WMF @Jclark-ctr Hi! We are ready to start provisioning these nodes, but the procedure is a little bit more convoluted than the usual since we need to force UEFI and there are still some Supermicro bugs that upstream is working on.

High level steps:

  • The provision cookbook needs to be run with --uefi, otherwise we'll not be able to configure the 24 disks managed by the SAS raid controller in JBOD.
  • Before the reimage we need to configure the 24 disks in JBOD. This is the procedure indicated by Papaul:

Enter the BIOS again after reboot go to ADVANCED--- BROADCOM CONFIGURATION utility---Configure---Make JBOD---Check ALL--OK--Confirm-EXIT and save and reboot

  • Reimage will run without any changes, but you may get to experience a bug that we are trying to solve with Supermicro. For mysterious reasons (probably related to the BMC firmware), it may happen that right after the Debian install the host reboots and tries to PXE again, ending up a second time in the Debian install. This messes up the reimage workflow, that ends up failing the puppet run step. The workaround is to kill reimage and kick off another one, that should hopefully end up finishing correctly. If this doesn't happen please ping me and I'll check/help!
VRiley-WMF updated the task description. (Show Details)Dec 2 2024, 7:01 PM
ops-monitoring-bot added a comment.Dec 3 2024, 12:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1091.eqiad.wmnet with OS bullseye

Jclark-ctr updated the task description. (Show Details)Dec 3 2024, 12:34 AM
VRiley-WMF updated the task description. (Show Details)Dec 3 2024, 12:39 AM
ops-monitoring-bot added a comment.Dec 3 2024, 1:26 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1091.eqiad.wmnet with OS bullseye executed with errors:

  • ms-be1091 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1091.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 3 2024, 1:36 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1091.eqiad.wmnet with OS bullseye

VRiley-WMF updated the task description. (Show Details)Dec 3 2024, 1:58 AM
ops-monitoring-bot added a comment.Dec 3 2024, 2:56 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1091.eqiad.wmnet with OS bullseye executed with errors:

  • ms-be1091 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1091.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 3 2024, 5:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1091.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 3 2024, 6:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1091.eqiad.wmnet with OS bullseye completed:

  • ms-be1091 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412031720_jclark_1988335_ms-be1091.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Dec 3 2024, 6:01 PM
Jclark-ctr updated the task description. (Show Details)Dec 3 2024, 8:49 PM
Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Dec 3 2024, 9:05 PM
VRiley-WMF updated the task description. (Show Details)Dec 3 2024, 10:16 PM
ops-monitoring-bot added a comment.Dec 3 2024, 10:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host ms-be1083.eqiad.wmnet with OS bullseye

VRiley-WMF updated the task description. (Show Details)Dec 3 2024, 10:27 PM
ops-monitoring-bot added a comment.Dec 3 2024, 10:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1087.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 3 2024, 10:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1090.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 3 2024, 10:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1088.eqiad.wmnet with OS bullseye

VRiley-WMF updated the task description. (Show Details)Dec 3 2024, 10:39 PM
ops-monitoring-bot added a comment.Dec 3 2024, 10:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1089.eqiad.wmnet with OS bullseye

Jclark-ctr updated the task description. (Show Details)Dec 3 2024, 10:54 PM
ops-monitoring-bot added a comment.Dec 3 2024, 11:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1088.eqiad.wmnet with OS bullseye completed:

  • ms-be1088 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412032250_jclark_2036964_ms-be1088.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 3 2024, 11:11 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1090.eqiad.wmnet with OS bullseye completed:

  • ms-be1090 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412032252_jclark_2037018_ms-be1090.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 3 2024, 11:12 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1087.eqiad.wmnet with OS bullseye completed:

  • ms-be1087 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412032246_jclark_2036959_ms-be1087.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Dec 3 2024, 11:12 PM
VRiley-WMF updated the task description. (Show Details)Dec 3 2024, 11:15 PM
ops-monitoring-bot added a comment.Dec 3 2024, 11:19 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host ms-be1083.eqiad.wmnet with OS bullseye completed:

  • ms-be1083 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412032237_vriley_2034194_ms-be1083.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Dec 3 2024, 11:20 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host ms-be1084.eqiad.wmnet with OS bullseye

Jclark-ctr updated the task description. (Show Details)Dec 3 2024, 11:21 PM
ops-monitoring-bot added a comment.Dec 3 2024, 11:27 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1089.eqiad.wmnet with OS bullseye completed:

  • ms-be1089 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412032308_jclark_2040213_ms-be1089.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF updated the task description. (Show Details)Dec 3 2024, 11:40 PM
ops-monitoring-bot added a comment.Dec 4 2024, 12:13 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host ms-be1084.eqiad.wmnet with OS bullseye completed:

  • ms-be1084 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412032334_vriley_2048868_ms-be1084.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF updated the task description. (Show Details)Dec 4 2024, 12:44 AM
ops-monitoring-bot added a comment.Dec 4 2024, 12:47 AM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host ms-be1085.eqiad.wmnet with OS bullseye

VRiley-WMF updated the task description. (Show Details)Dec 4 2024, 12:54 AM
ops-monitoring-bot added a comment.Dec 4 2024, 2:08 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host ms-be1085.eqiad.wmnet with OS bullseye executed with errors:

  • ms-be1085 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1085.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
elukey added a comment.Dec 4 2024, 10:25 AM

@Jclark-ctr I fixed the provisioning of ms-be1086, for some reasons if the BMC doesn't have IPv6 enabled the settings that errored out are read only (in fact provisioning was failing during the BMC network settings rollout). The workaround is to connect to the WebUI, Configuration -> Network and enable IPv6. Then you can re-run the cookbook and it works.

ops-monitoring-bot added a comment.Dec 4 2024, 9:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1002 for host ms-be1086.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 4 2024, 10:12 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1002 for host ms-be1086.eqiad.wmnet with OS bullseye completed:

  • ms-be1086 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412042149_jclark_2252444_ms-be1086.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr updated the task description. (Show Details)Dec 4 2024, 10:12 PM
ops-monitoring-bot added a comment.Dec 4 2024, 10:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host ms-be1085.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 4 2024, 11:26 PM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host ms-be1085.eqiad.wmnet with OS bullseye executed with errors:

  • ms-be1085 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be1085.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 4 2024, 11:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1002 for host ms-be1085.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 5 2024, 12:15 AM

Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1002 for host ms-be1085.eqiad.wmnet with OS bullseye completed:

  • ms-be1085 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202412042357_vriley_2279376_ms-be1085.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
VRiley-WMF updated the task description. (Show Details)Dec 5 2024, 12:16 AM
Jclark-ctr closed this task as Resolved.Dec 5 2024, 12:34 AM
Jclark-ctr claimed this task.
MatthewVernon mentioned this in T382056: ms backend hardware refresh for 24/25.Dec 12 2024, 9:59 AM
wiki_willy mentioned this in T392796: ms-be1060 crashed, then went into an exception in the uEFI pre-boot environment.Apr 29 2025, 4:56 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits