Page MenuHomePhabricator
Create Task
Maniphest T306835

Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of an-presto10[06-15].eqiad.wmnet

Hostname / Racking / Installation Details

Hostnames: an-presto10[06-15].eqiad.wmnet
Racking Proposal: Anywhere across the existing rows A-F
Networking/Subnet/VLAN/IP: Analytics VLAN - One 10 Gbps SFP+ connection per server
Partitioning/Raid: As per the reference system (an-presto1001)
OS Distro: Buster preferred, unless raid controller won't work in it. (sub-team originally requested Buster, but H750 testing has only happened for Bullseye at this time.)

Per host setup checklist

an-presto1006:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1007:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1008:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1009:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1010:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1011:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details

[x]x - firmware update (idrac, bios, network, raid controller)

  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1012:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1013:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1014:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-presto1015:
  • - receive in system on procurement task T297729 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Update the partman configuration for the new presto hostsoperations/puppetproduction+1 -1
Add netboot.cfg and site.pp for an-presto hostsoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jclark-ctr subscribed.May 27 2022, 6:53 PM
• Cmjohnson updated the task description. (Show Details)Jun 15 2022, 2:47 PM
gerritbot added a comment.Jun 21 2022, 5:35 PM

Change 807187 had a related patch set uploaded (by Cmjohnson; author: Cmjohnson):

[operations/puppet@production] Add netboot.cfg and site.pp for an-presto hosts

https://gerrit.wikimedia.org/r/807187

gerritbot added a project: Patch-For-Review.Jun 21 2022, 5:35 PM
gerritbot added a comment.Jun 21 2022, 5:53 PM

Change 807187 merged by Cmjohnson:

[operations/puppet@production] Add netboot.cfg and site.pp for an-presto hosts

https://gerrit.wikimedia.org/r/807187

• Cmjohnson updated the task description. (Show Details)Jun 21 2022, 5:54 PM
• Cmjohnson added a comment.Jun 21 2022, 6:07 PM

@BTullis can you confirm what the raid configuration is supposed to be please. 2 SSD Raid 1? and Raid 10 the spinning disks?

Maintenance_bot removed a project: Patch-For-Review.Jun 21 2022, 6:30 PM
BTullis added a comment.Jun 22 2022, 4:53 PM

@Cmjohnson yes please, that's perfect.

ops-monitoring-bot added a comment.Jun 22 2022, 7:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS buster

• Cmjohnson updated the task description. (Show Details)Jun 22 2022, 7:16 PM
ops-monitoring-bot added a comment.Jun 22 2022, 7:42 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS buster executed with errors:

  • an-presto1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 22 2022, 7:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 22 2022, 8:13 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 22 2022, 8:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jun 22 2022, 8:27 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS buster executed with errors:

  • an-presto1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 22 2022, 8:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 22 2022, 8:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 22 2022, 9:28 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 22 2022, 9:45 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1007 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
• Cmjohnson added a subscriber: RobH.Jun 23 2022, 2:32 PM

@RobH this may be a controller issue, the servers were able to go through the installation without any issue, after the install, the server reboots. And the server will typically boot to the OS. However, after going through post, the console I see "Booting... Booting from Hard drive C:" but the OS never actually comes up and the install fails. I tried this on 1006 and 1007 and it's the same issue. Fixing this is outside of my expertise so I am hoping you might be able to help. I am thinking it is the controller

For reference
I had to downgrade the Broadom NIC firmware, the image would not load otherwise
Buster does not install on the server, fails during the installation, and does not see the disks.
Bullseye worked without any issue.

• Cmjohnson moved this task from Racking Tasks to Blocked on the ops-eqiad board.Jun 27 2022, 12:56 AM
BTullis added a comment.Jun 27 2022, 12:13 PM

@Cmjohnson - I think we should just use bullseye for these hosts, if there's a controller issue with buster. We should be able to get presto working with bullseye, all being well.

ops-monitoring-bot added a comment.Jun 29 2022, 8:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jun 29 2022, 8:37 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS buster executed with errors:

  • an-presto1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 29 2022, 8:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jun 29 2022, 8:48 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS buster executed with errors:

  • an-presto1007 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jun 29 2022, 8:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jun 29 2022, 9:35 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
• Cmjohnson added a comment.EditedJun 29 2022, 10:09 PM

@BTullis Robh said you figured out the workaround to get the right raid volume to boot first. I tried on an-presto1006 and everything seemed to be going smoothly, the image was installed and the server rebooted but did not go into the OS. I get this instead. Did you change anything on the partman recipe?

Booting from Hard drive C:
GRUB loading.
Welcome to GRUB!

error: unknown filesystem.
grub rescue>

BTullis added a subscriber: fgiunchedi.Jun 30 2022, 1:13 PM

Hi @Cmjohnson - that's really interesting. I think that you're one step closer to a working system than I am, but ultimately I think we might be affected by the same issue.

I've been working with stat1010 on T307399 and I've been trying the partman/custom/kafka-jumbo.cfg recipe.
You've applied the partman/custom/cloudvirtan.cfg to this presto server, right, as per the reference an-presto1001?

Ultimately, both the stat and presto servers should have pretty similar requirements in therms of their disk configuration, which is:

  • OS installed to SSDs in /dev/sda with one big LVM volume assigned to the root logical volume and mounted at /
  • The HDDs configured as /dev/sdb with one big LVM volume assigned to the srv logical volume and mounted at /srv

There's a small difference between the two receipes that we're trying, in that:

  • in the cloudvirtan recipe /boot is a separate non LVM volume
  • in the kafka-jumbo recipe the newer future-proofed method of leaving a 300 MB biosgrub partition is used instead. /boot is beneath / in this case so grub needs to be able to activate LVM.

However, my attempts aren't working either. I can't even get grub to be found at all, it just keps on failing to find a bootloader on /dev/sda at all and falling through to PXE booting and therefore a boot-install-boot... loop.
It's a useful data point that you're getting as far as a grub rescue shell, but an unknown filesystem.

I'm currently looking into this with @fgiunchedi as well, so hopefully we will be able to work out how to proceed. It might be that we come up with a new partman recipe that is suitable for both these presto and the stat servers.
In the meantime, I might try to replicate your results by applying the cloudvirtan recipe to stat1010.

I'd also like to look into how the originally successful builds of dumpsdata1007 happened and whether there is anything we can gain by looking at those more closely.

Apologies for the inconvenience this delay is causing for you.

gerritbot added a comment.Jul 1 2022, 10:46 AM

Change 810305 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the partman configuration for the new presto hosts

https://gerrit.wikimedia.org/r/810305

gerritbot added a project: Patch-For-Review.Jul 1 2022, 10:46 AM
gerritbot added a comment.Jul 1 2022, 10:50 AM

Change 810305 merged by Btullis:

[operations/puppet@production] Update the partman configuration for the new presto hosts

https://gerrit.wikimedia.org/r/810305

BTullis added a comment.Jul 1 2022, 11:01 AM

@Cmjohnson I think this should be good to go now. We've identified an additional step that we need to carry out in terms of the RAID controller configuration, so that it boots from the RAID1 mirror.
That step is outlined here: T297913#8041258

I don't know if there's a way of automating it, but there might be. I've also updated the partman recipe for these new hosts to use custom/kafka-jumbo.cfg in https://gerrit.wikimedia.org/r/810305

We might well rename that in due course to make it less custom and more standard, for hosts using hardware RAID, two drives, and a big /srv volume.
For now though, we can live with the dodgy naming.

Please feel free to continue installing these hosts and let me know if anything doesn't look right.

Maintenance_bot removed a project: Patch-For-Review.Jul 1 2022, 11:30 AM
ops-monitoring-bot added a comment.Jul 1 2022, 9:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:34 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1006.eqiad.wmnet with OS bullseye completed:

  • an-presto1006 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012104_cmjohnson_1376385_an-presto1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1012.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1014.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1013.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1008.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1011.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1010.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1015.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 9:50 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1012.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1012 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 1 2022, 9:52 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1015.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1015 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 1 2022, 9:57 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1007 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012136_cmjohnson_1383722_an-presto1007.out
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 1 2022, 9:57 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1011.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1011 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012136_cmjohnson_1383729_an-presto1011.out
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 1 2022, 9:57 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1009 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012136_cmjohnson_1383727_an-presto1009.out
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 1 2022, 10:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1015.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 10:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1012.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 10:04 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1010.eqiad.wmnet with OS bullseye completed:

  • an-presto1010 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012136_cmjohnson_1383724_an-presto1010.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Jul 1 2022, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1008.eqiad.wmnet with OS bullseye completed:

  • an-presto1008 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012136_cmjohnson_1383728_an-presto1008.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Jul 1 2022, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1013.eqiad.wmnet with OS bullseye completed:

  • an-presto1013 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012136_cmjohnson_1383723_an-presto1013.out
    • Checked BIOS boot parameters are back to normal
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Jul 1 2022, 10:08 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1014.eqiad.wmnet with OS bullseye completed:

  • an-presto1014 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012136_cmjohnson_1383725_an-presto1014.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Jul 1 2022, 10:22 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1012.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1012 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jul 1 2022, 10:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1012.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jul 1 2022, 10:32 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1015.eqiad.wmnet with OS bullseye completed:

  • an-presto1015 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012201_cmjohnson_1396423_an-presto1015.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
RobH unsubscribed.Jul 1 2022, 10:59 PM
ops-monitoring-bot added a comment.Jul 1 2022, 11:02 PM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1012.eqiad.wmnet with OS bullseye completed:

  • an-presto1012 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207012231_cmjohnson_1403122_an-presto1012.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
• Cmjohnson closed this task as Resolved.Jul 2 2022, 1:05 AM
• Cmjohnson updated the task description. (Show Details)
BTullis updated the task description. (Show Details)Aug 23 2022, 8:53 AM
ops-monitoring-bot added a comment.Aug 24 2022, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 24 2022, 4:15 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1007 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Aug 24 2022, 4:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Aug 24 2022, 5:06 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1009 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202208241616_btullis_2784016_an-presto1009.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • The reimage failed, see the cookbook logs for the details
ayounsi reopened this task as Open.Sep 2 2022, 8:54 AM
ayounsi subscribed.

FYI, Netbox is alerting with:
an-presto1009 (WMF11494) Device is in PuppetDB but is Planned in Netbox (should be Staged, Active or Failed)
in https://netbox.wikimedia.org/extras/reports/puppetdb.PhysicalHosts/

ops-monitoring-bot added a comment.Sep 5 2022, 9:23 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2022, 12:09 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1007 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 5 2022, 12:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Sep 5 2022, 12:47 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1007 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Sep 5 2022, 12:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host an-presto1011.eqiad.wmnet with OS bullseye

BTullis added a comment.Sep 5 2022, 1:09 PM

@Cmjohnson - sorry to trouble you with this old ticket, but I'm having an issue with three of these new an-presto hosts.

  • an-presto1007
  • an-presto1009
  • an-presto1011

I'm not sure that the reimage cookbook was ever completed successfully on these hosts.
I've tried running it myself to reinstall them and they're failing. Sometimes I can't get the console (e.g. sudo install_console an-presto1007.eqiad.wmnet) to connect, when I would expect it to be able to do so. Sometimes they fail at the first puppet run, but I can't get into the hosts to have a good look.

I also changed the hosts' status from 'planned' to 'staged' in netbox, but it didn't make any difference.

If there's any chance that you could take a look again at these three hosts, I'd be grateful.

ops-monitoring-bot added a comment.Sep 5 2022, 1:13 PM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host an-presto1011.eqiad.wmnet with OS bullseye executed with errors:

  • an-presto1011 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202209051248_btullis_1582525_an-presto1011.out
    • Checked BIOS boot parameters are back to normal
    • Unable to run puppet on puppetmaster2001.codfw.wmnet,puppetmaster1001.eqiad.wmnet to update configmaster.wikimedia.org with the new host SSH public key for wmf-update-known-hosts-production
    • The reimage failed, see the cookbook logs for the details
BTullis moved this task from Blocked to Hardware Failure / Troubleshoot on the ops-eqiad board.Oct 4 2022, 10:06 AM

Moving to 'Troubleshoot' column on the ops-eqiad board. @Cmjohnson have you been able to look into this at all? Thanks.

ayounsi mentioned this in T320696: Reduce the count of Netbox devices with incorrect status.Oct 13 2022, 8:19 AM
ops-monitoring-bot added a comment.Oct 18 2022, 11:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 19 2022, 12:34 AM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye completed:

  • an-presto1007 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210182358_cmjohnson_2093654_an-presto1007.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Oct 19 2022, 12:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 19 2022, 1:04 AM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye completed:

  • an-presto1009 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210190034_cmjohnson_2101249_an-presto1009.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Oct 19 2022, 1:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-presto1011.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 19 2022, 1:35 AM

Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-presto1011.eqiad.wmnet with OS bullseye completed:

  • an-presto1011 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210190104_cmjohnson_2107654_an-presto1011.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
• Cmjohnson closed this task as Resolved.Oct 19 2022, 1:35 AM

@BTullis these have been fixed, I updated the nic firmware and re-ran the image script.

BTullis added a comment.Oct 19 2022, 2:03 PM

@Cmjohnson many thanks indeed, that's great. 👍

odimitrijevic added a comment.Oct 21 2022, 4:57 PM

@Cmjohnson Thank you!

BTullis mentioned this in T323783: Add an-presto10[06-15] to the presto cluster.Nov 24 2022, 5:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL