Page MenuHomePhabricator
Create Task
Maniphest T381919

Supermicro: unable to set boot order after using Redfish to boot once
Open, LowPublic
Actions

Description

We are having difficulties setting the UEFI boot order on our, "Supermicro X12DSC-A6 BIOS Date:07/11/2024 Rev:2.1", hosts. On a normal boot we are able to add boot entries and set the boot order. However, when we issue a one time boot via Redfish updates are lost:

Normal

  1. Boot once disabled
'BootSourceOverrideEnabled': 'Disabled'
  1. Boot into Linux
  1. Using efibootmgr add a new entry, which is added first in the boot order
  1. Reboot
  1. New boot order is preserved

Boot once

  1. Boot once set:
"BootSourceOverrideEnabled": "Once",
"BootSourceOverrideTarget": "Pxe", # or Hdd
"BootSourceOverrideMode": "UEFI",
  1. Boot into Linux
  1. Using efibootmgr add a new entry, which is added first in the boot order
  1. Reboot
  1. New boot order is lost, though new entry remains

Ticket on Supermicro side: #FAV-941-81182

Related Objects
Search...

Event Timeline

jhathaway created this task.Dec 10 2024, 8:04 PM
Stashbot added a comment.Dec 10 2024, 8:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-10T20:04:38Z] <jhathaway@cumin1002> START - Cookbook sre.hosts.downtime for 3:00:00 on ms-be1088.eqiad.wmnet with reason: T381919

Stashbot added a comment.Dec 10 2024, 8:04 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-10T20:04:52Z] <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be1088.eqiad.wmnet with reason: T381919

Stashbot added a comment.Dec 10 2024, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-10T20:28:08Z] <jhathaway@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on ms-be1088.eqiad.wmnet with reason: T381919

Stashbot added a comment.Dec 10 2024, 8:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-10T20:28:11Z] <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be1088.eqiad.wmnet with reason: T381919

Stashbot added a comment.Dec 10 2024, 10:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-10T22:54:18Z] <jhathaway@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on ms-be1088.eqiad.wmnet with reason: T381919

Stashbot added a comment.Dec 10 2024, 10:54 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-10T22:54:21Z] <jhathaway@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on ms-be1088.eqiad.wmnet with reason: T381919

jhathaway triaged this task as Low priority.Dec 16 2024, 3:28 PM
bking added a comment.Dec 16 2024, 9:49 PM

Per IRC conversation with @elukey , I just wanted to let y'all know that I successfully reimaged cloudelastic1012 just now. No Puppet 5, no CSR or any other errors.

Stashbot added a comment.Jan 30 2025, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-30T20:27:51Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Jan 30 2025, 9:57 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-30T21:57:36Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Jan 31 2025, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-31T17:52:10Z] <jhathaway@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sretest1001.eqiad.wmnet with reason: T381919

Stashbot added a comment.Jan 31 2025, 5:59 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-31T17:59:46Z] <jhathaway@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on sretest1002.eqiad.wmnet with reason: T381919

Stashbot added a comment.Jan 31 2025, 7:59 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-31T19:59:34Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

jhathaway added a comment.Feb 3 2025, 3:05 PM

Supermicro X12DSC-A6 system:

  1. Boot into Linux
  2. Copy an EFI bootable image, e.g. memtest.efi into /EFI/memtest/memtest.efi
  3. Use efibootmgr to add memtest.efi as a boot entry and first in the boot order
  4. reboot
  5. *system does not boot into memtest*

What appears to happen after rebooting, before step (5):

  1. The boot order is not preserved.

    In fact it appears it is not possible to alter the boot order through the operating system UEFI interface, or through the Redfish API.
  1. The boot order does not include Hard drive entries directly as under a typical UEFI system. Instead, adding "Hard Drive" to the fixed boot order delegates to the "UEFI Hard Disk Drive BBS Priorities" list to determine which UEFI boot entry to try.
  1. Entries must be manually added to the "UEFI Hard Disk Drive BBS Priorities" list?
  1. It is not possible to alter the "UEFI Hard Disk Drive BBS Priorities" from within Linux?
Stashbot added a comment.Feb 20 2025, 8:10 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-20T20:10:54Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 20 2025, 10:06 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-20T22:06:21Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 21 2025, 5:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-21T17:25:44Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 21 2025, 8:31 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-21T20:31:06Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 21 2025, 10:31 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-21T22:31:46Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 24 2025, 9:08 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-24T21:07:59Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 25 2025, 5:37 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-25T17:37:29Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 25 2025, 8:25 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-25T20:25:20Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 25 2025, 10:04 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-25T22:04:44Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 25 2025, 11:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-25T23:27:18Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.Feb 26 2025, 5:26 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-26T17:26:44Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on ms-be2088.codfw.wmnet with reason: T381919

ops-monitoring-bot added a comment.Feb 26 2025, 8:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 26 2025, 8:32 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 26 2025, 8:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 26 2025, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 26 2025, 9:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 26 2025, 9:39 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 26 2025, 9:40 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 26 2025, 10:24 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 26 2025, 10:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 26 2025, 11:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 27 2025, 3:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 27 2025, 3:53 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202502271549_jhathaway_3741735_ms-be2088.out, asking the operator what to do
    • First Puppet run failed and the operator aborted
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 27 2025, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 27 2025, 4:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Feb 27 2025, 5:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Feb 27 2025, 5:37 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bookworm executed with errors:

  • ms-be2088 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console ms-be2088.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
jhathaway added a comment.EditedFeb 27 2025, 10:14 PM

Current reproduction

Prerequisites

  1. Using the virtual media support connect a SystemRescue ISO hosted on an HTTP server, https://www.system-rescue.org/
  1. Configure the ATEN Virtual CDROM to be first in the boot order
  1. Deploy files to boot SystemRescue via HTTP boot on http server
/srv/systemrescue/
└── x86_64
    ├── airootfs.sfs
    ├── amd_ucode.img
    ├── intel_ucode.img
    ├── sysresccd.img
    └── vmlinuz
  1. Configure dhcp
host ms-be2088 {
    host-identifier option agent.circuit-id "lsw1-d7-codfw:xe-0/0/14.0:private1-d7-codfw";
    fixed-address 10.192.42.7;
    filename "http://208.80.154.10/efiboot-systemrescue/snponly.efi";
    option vendor-class-identifier "HTTPClient";
}
  1. Configure iPXE on http server
/srv/efiboot-systemrescue/
├── autoexec.ipxe
└── snponly.efi

$ cat /srv/efiboot-systemrescue/autoexec.ipxe
#!ipxe

set extrabootoptions initrd=amd_ucode.img initrd=intel_ucode.img initrd=sysresccd.img ip=dhcp net.ifnames=0 BOOTIF=01-${netX/mac} console=tty0 console=ttyS1,115200n8
kernel http://apt.wikimedia.org/systemrescue/x86_64/vmlinuz archiso_http_srv=http://apt.wikimedia.org/ archisobasedir=systemrescue ${extrabootoptions} loglevel=3
initrd http://apt.wikimedia.org/systemrescue/x86_64/amd_ucode.img
initrd http://apt.wikimedia.org/systemrescue/x86_64/intel_ucode.img
initrd http://apt.wikimedia.org/systemrescue/x86_64/sysresccd.img

boot

UEFI Boot reproduction

  1. Boot off of the ATEN CDROM into SystemRescue
  1. List current UEFI boot entries, NOTE: ATEN Virtual CDROM loaded with SystemRescue is first in boot order
$ efibootmgr -v | grep -i order
$ efibootmgr -v | grep -i aten
  1. Using Redfish force UEFI HTTP boot once to SystemRescue
http_boot = {
    "Boot": {
        "BootSourceOverrideMode": "UEFI",
        "BootSourceOverrideEnabled": "Once",
        "BootSourceOverrideTarget": "Pxe",
    }
}
r.request("patch", "/redfish/v1/Systems/1", json=http_boot)
  1. Reboot via UEFI HTTP boot in SystemRescue
  1. Manually add UEFI boot entry for memtest and place first in order
$ mount /dev/sda1 /mnt
$ cp /boot/memtest86+/memtest.efi /mnt/EFI/
$ umount /mnt
$ efibootmgr -c -d /dev/sda1 -L memtest -l '\EFI\memtest.efi' -v
  1. Reboot, expecting to boot into memtest, instead boot back into SystemRescue
ops-monitoring-bot added a comment.Feb 28 2025, 3:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 28 2025, 4:49 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin2002 for host ms-be2088.codfw.wmnet with OS bullseye completed:

  • ms-be2088 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202502281603_jhathaway_213661_ms-be2088.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202502281632_jhathaway_213661_ms-be2088.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.May 13 2025, 5:41 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T17:41:49Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.May 13 2025, 7:40 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-13T19:40:29Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.May 14 2025, 4:11 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T16:11:36Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be2088.codfw.wmnet with reason: T381919

jhathaway added a comment.May 14 2025, 6:55 PM

Met with Brian Nguyen from Supermicro on a video call. On the video call
I demoed the reproducer:

  1. Booted off a virtual CDROM into SystemRescue and removed all hard drive boot entries. Then added an entry for Memtest and placed it first in the boot order. Rebooted and Memtest booted as expected.
  2. Booted off of UEFI HTTP Boot and removed all hard drive boot entries. Then added an entry for Memtest and placed it first in the boot order. Rebooted and instead of booting into Memtest the machine booted off of the virtual CDROM into SystemRescue, thus displaying the bug.

Brian said he would speak with the Redfish developers and then get back
to me.

Stashbot added a comment.May 14 2025, 9:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T21:47:44Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.May 14 2025, 10:10 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-14T22:10:55Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.May 22 2025, 7:23 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-22T19:23:40Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be2088.codfw.wmnet with reason: T381919

Stashbot added a comment.May 22 2025, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-22T21:13:04Z] <jhathaway@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-be2088.codfw.wmnet with reason: T381919

elukey mentioned this in T404356: UEFI installer not installing grub correctly (at least on systems where / is RAID).Oct 30 2025, 2:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits