Page MenuHomePhabricator
Create Task
Maniphest T299123

es1022 troubles with PXE
Closed, ResolvedPublic
Actions

Description

When trying to reimage es1022 it I have found several things:

  • PXE boot wasn't working until I did it manually - it attempted it and then I guess (the screen was going blank) it timed out and booted from disk
  • After the reimage, then trying to remove the override for PXE, it failed with:
Running IPMI command: ipmitool -I lanplus -H es1022.mgmt.eqiad.wmnet -U root -E chassis bootparam get 5
Exception raised while executing cookbook sre.hosts.reimage:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
    raw_ret = runner.run()
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 487, in run
    self.ipmi.check_bootparams()
  File "/usr/lib/python3/dist-packages/spicerack/ipmi.py", line 125, in check_bootparams
    raise IpmiCheckError(f"Expected BIOS boot params in {IPMI_SAFE_BOOT_PARAMS} got: {param}")
spicerack.ipmi.IpmiCheckError: Expected BIOS boot params in ('0000000000', '8000020000') got: 0000020000

If possible I would like to:

  • Get bios and firmware upgraded
  • Verify that PXE is configured to boot from the ethernet which has this MAC: b0:26:28:f5:35:dc

Please coordinate with us on a day/time so we can shutdown mysql.

Thanks

Details

ProjectBranchLines +/-Subject
operations/software/spicerackmaster+17 -1redfish: better support of parsing JSON responses
operations/cookbooksmaster+12 -10sre.hosts.provision: disable PXE on all other NICs
operations/puppetproduction+1 -0es1022: Disable notifications
operations/puppetproduction+1 -0es1022: Disable notifications
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui triaged this task as Medium priority.Jan 13 2022, 10:22 AM
• Marostegui created this task.
• Marostegui moved this task from Triage to Blocked on the DBA board.
Maintenance_bot added a project: SRE.Jan 13 2022, 10:45 AM
• Marostegui updated the task description. (Show Details)Jan 13 2022, 11:10 AM
wiki_willy assigned this task to Cmjohnson.Jan 13 2022, 4:58 PM
Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Jan 13 2022, 6:07 PM
Cmjohnson added a comment.Jan 19 2022, 5:10 PM

@Marostegui Can we schedule this for me to power down tomorrow (20 Jan) 1530UTC?

Cmjohnson moved this task from High Priority Task to Hardware Failure / Troubleshoot on the ops-eqiad board.Jan 19 2022, 5:10 PM
• Marostegui added a comment.Jan 19 2022, 5:12 PM
In T299123#7633254, @Cmjohnson wrote:

@Marostegui Can we schedule this for me to power down tomorrow (20 Jan) 1530UTC?

Sounds good. I will leave the host powered down for you, thanks!

Stashbot added a comment.Jan 20 2022, 8:18 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-20T08:18:09Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1022 for on-site maintenance T299123', diff saved to https://phabricator.wikimedia.org/P18917 and previous config saved to /var/cache/conftool/dbconfig/20220120-081809-marostegui.json

gerritbot added a comment.Jan 20 2022, 8:19 AM

Change 755630 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1022: Disable notifications

https://gerrit.wikimedia.org/r/755630

gerritbot added a project: Patch-For-Review.Jan 20 2022, 8:19 AM

Change 755630 merged by Marostegui:

[operations/puppet@production] es1022: Disable notifications

https://gerrit.wikimedia.org/r/755630

Maintenance_bot removed a project: Patch-For-Review.Jan 20 2022, 9:10 AM
Stashbot added a comment.Jan 20 2022, 1:55 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-20T13:55:47Z] <marostegui> Power off es1022 for onsite maintenance T299123

• Marostegui added a comment.Jan 20 2022, 1:58 PM

@Cmjohnson is now off. You can proceed as needed.

Cmjohnson added a comment.Jan 20 2022, 4:22 PM

@Marostegui BIOS and network Firmware updated, this should fix your issue. I will leave task open until you confirm all is well.

Cmjohnson reassigned this task from Cmjohnson to • Marostegui.Jan 20 2022, 4:52 PM
Cmjohnson added a subscriber: Cmjohnson.
• Marostegui added a comment.Jan 20 2022, 5:39 PM

Thanks Chris - I will try a reimage on Monday to see if it PXE boots fine.
I have started mysql now so it can start catching up

Stashbot added a comment.Jan 24 2022, 6:02 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T06:02:48Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1022 T299123', diff saved to https://phabricator.wikimedia.org/P18980 and previous config saved to /var/cache/conftool/dbconfig/20220124-060248-marostegui.json

ops-monitoring-bot added a comment.Jan 24 2022, 6:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by marostegui@cumin1001 for host es1022.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 24 2022, 6:44 AM

Cookbook cookbooks.sre.hosts.reimage started by marostegui@cumin1001 for host es1022.eqiad.wmnet with OS bullseye completed:

  • es1022 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201240605_marostegui_6269_es1022.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
• Marostegui reassigned this task from • Marostegui to Cmjohnson.Jan 24 2022, 7:16 AM

@Cmjohnson the host keeps ignoring PXE boot even if it attempts to do so from the boot menu. Not sure what could be root cause for this. It only works if selected manually via IDRAC by pressing F12. Maybe something within the PXE Boot card or options on the BIOS?
Let me know which date/time you'd like to try to troubleshoot this so I can make sure the host is off for you.

Cmjohnson added a subscriber: Volans.Jan 24 2022, 5:49 PM

Marostegui this actually seems like a script issue, you may want to ping @Volans

• Marostegui added a comment.Jan 25 2022, 5:58 AM

@Volans any thoughts? I can try this reimage with you if that'd help with the troubleshooting.

Stashbot added a comment.Jan 25 2022, 10:29 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-25T10:29:13Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Depool es1022 T299123', diff saved to https://phabricator.wikimedia.org/P19119 and previous config saved to /var/cache/conftool/dbconfig/20220125-102912-marostegui.json

gerritbot added a comment.Jan 25 2022, 10:30 AM

Change 756956 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] es1022: Disable notifications

https://gerrit.wikimedia.org/r/756956

gerritbot added a project: Patch-For-Review.Jan 25 2022, 10:30 AM

Change 756956 merged by Marostegui:

[operations/puppet@production] es1022: Disable notifications

https://gerrit.wikimedia.org/r/756956

Maintenance_bot removed a project: Patch-For-Review.Jan 25 2022, 11:11 AM
ops-monitoring-bot added a comment.Jan 25 2022, 1:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by volans@cumin1001 for host es1022.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2022, 1:48 PM

Cookbook cookbooks.sre.hosts.reimage started by volans@cumin1001 for host es1022.eqiad.wmnet with OS bullseye executed with errors:

  • es1022 (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 25 2022, 1:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by volans@cumin1001 for host es1022.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 25 2022, 1:55 PM

Cookbook cookbooks.sre.hosts.reimage started by volans@cumin1001 for host es1022.eqiad.wmnet with OS bullseye executed with errors:

  • es1022 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 25 2022, 3:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by volans@cumin1001 for host es1022.eqiad.wmnet with OS bullseye

Volans added a comment.Jan 25 2022, 3:46 PM

The issue here was that both NIC.Integrated.1-1-1 and NIC.Integrated.1-3-1 had the LegacyBootProto set to PXE while the host has a cable only on the 3rd NIC (see Netbox ).
I've used the Redfish API support in Spicerack to change that:

INFO:spicerack.redfish:Updated value for attribute NIC.Integrated.1-1-1 -> LegacyBootProto: PXE => NONE

And trigger a new reimage that is currently running. The automation for new hosts (new sre.hosts.provision cookbook) should ensure there is only one NIC with PXE set on it for newly provisioned hosts.

ops-monitoring-bot added a comment.Jan 25 2022, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage started by volans@cumin1001 for host es1022.eqiad.wmnet with OS bullseye completed:

  • es1022 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201251534_volans_7987_es1022.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Volans closed this task as Resolved.Jan 25 2022, 4:12 PM

Root cause found, problem solved, host reimaged. Resolving.

• Marostegui added a comment.Jan 25 2022, 4:16 PM

Thank you @Volans for troubleshooting this issue!

• Marostegui added a comment.Jan 26 2022, 7:52 AM

I think es1020 is having the same issue - @Volans do you have the fixing command somewhere?
Given that es1020 and es1022 are from the same batch it could make sense. Also, expecting es1021 (current master) to also have that misconfiguration.

• Marostegui mentioned this in T300005: Upgrade es4 to Bullseye.Jan 26 2022, 8:22 AM
Volans added a comment.Jan 26 2022, 10:20 AM

I've fixed es1020 manually and checked the other hosts in the same batch ( https://netbox.wikimedia.org/dcim/devices/?cf_ticket=T235659 ), apart es1024 all of them have the same misconfiguration.
I tested if I could just run the provisioning cookbook on them to fix them and found a couple of issues (one bug in Dell's reply to Redfish API that I need to workaround and a use case not fully supported in the cookbook).
I'm fixing them so we can test the cookbook on the next hosts to reimage of the batch.

jcrespo added a subscriber: jcrespo.Jan 26 2022, 10:31 AM
gerritbot added a comment.Jan 26 2022, 11:22 AM

Change 757410 had a related patch set uploaded (by Volans; author: Volans):

[operations/cookbooks@master] sre.hosts.provision: disable PXE on all other NICs

https://gerrit.wikimedia.org/r/757410

gerritbot added a project: Patch-For-Review.Jan 26 2022, 11:22 AM
gerritbot added a comment.Jan 26 2022, 12:35 PM

Change 757410 merged by jenkins-bot:

[operations/cookbooks@master] sre.hosts.provision: disable PXE on all other NICs

https://gerrit.wikimedia.org/r/757410

gerritbot added a comment.Jan 26 2022, 1:31 PM

Change 757435 had a related patch set uploaded (by Volans; author: Volans):

[operations/software/spicerack@master] redfish: better support of parsing JSON responses

https://gerrit.wikimedia.org/r/757435

gerritbot added a comment.Jan 26 2022, 2:03 PM

Change 757435 merged by jenkins-bot:

[operations/software/spicerack@master] redfish: better support of parsing JSON responses

https://gerrit.wikimedia.org/r/757435

Volans mentioned this in rCCKB357c1ceea9ee: sre.hosts.provision: disable PXE on all other NICs.Dec 14 2022, 3:30 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL