Page MenuHomePhabricator
Create Task
Maniphest T349936

Q2:rack/setup/install an-worker11[57-75]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of an-worker11[57-75]

Hostname / Racking / Installation Details

Hostnames: an-worker11[57-75]
Racking Proposal: DIstributed roughly evenly across eqiad rows A-F - (The hosts being replaced are 6 in row A, 7 in row B, 5 in row C, 5 in row D, in case that helps)
Networking Setup: Speed:10G. - VLAN: Analytics : AAAA records:Y, Additional IP records (Cassandra)? No
Partitioning/Raid: HW Raid: Y (only the O/S disks), Partman recipe and/or desired Raid Level: partman/custom/analytics-flex.cfg
OS Distro: Bullseye
Sub-team Technical Contact: @BTullis on behalf of the Data-Platform-SRE team.

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

an-worker1157:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1158:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1159:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1160:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1161:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1162:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1163:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1164:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1165:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1166:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1167:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1168:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1169:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1170:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1171:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1172:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1173:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1174:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
an-worker1175:
  • - receive in system on procurement task T348054 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
Remove obsolete Hiera entriesoperations/puppetproduction+0 -38
Set tty serial to com2 for Dell Poweredge R750 model variantsoperations/cookbooksmaster+2 -0
Add an-worker1157-75.yaml file T349936operations/puppetproduction+38 -0
update site.pp and partition info for new an-workersoperations/puppetproduction+6 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJclark-ctrT349936 Q2:rack/setup/install an-worker11[57-75]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Maintenance_bot removed a project: Patch-For-Review.Nov 21 2023, 6:30 PM
ops-monitoring-bot added a comment.Nov 21 2023, 6:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1158.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 21 2023, 7:38 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1157.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1157 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 21 2023, 7:58 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1158.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1158 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 22 2023, 12:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1158.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 22 2023, 2:01 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1158.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1158 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 22 2023, 6:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host an-worker1160.eqiad.wmnet with OS bullseye

RobH added a comment.EditedNov 22 2023, 6:26 PM

attempting to reimage an-worker1160 it sticks at requesting a lease for boot, host shows the MAC of the eth0 attempting to request a dhcp lease for boot.

on install1004:

Nov 22 18:22:06 install1004 dhcpd[3030461]: DHCPDISCOVER from 28:80:88:6e:bf:80 via 10.65.0.1: network 10.65.0.0/16: no free leases

The host is in the analytics vlan, so this seems odd and should work, but doesn't. Possibly related to T350179, but that seems to show it not even attempting to get dhcp lease where this definitely is attempting a lease request and being denied by install1004.

ops-monitoring-bot added a comment.Nov 22 2023, 7:36 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host an-worker1160.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1160 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 22 2023, 8:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1175.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 22 2023, 10:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1175.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1175 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 23 2023, 4:13 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host an-worker1160.eqiad.wmnet with OS bullseye

cmooney added subscribers: Jclark-ctr, RobH, cmooney.Nov 23 2023, 5:05 PM

@RobH @Jclark-ctr I kicked off the reimage of an-worker1160 again.

I think the problem here wasn't actually an error on the DHCP config, but a problem we have with the sre.network.configure-switch-interfaces cookbook. The cookbook is fairly basic, and only adds the interface configuration snippet to the switch. It doesn't have the logic, as Homer does, to add the actual definition for a given vlan to the box if it's missing and needs to be there. So for instance in this case, this was the first connection to this switch for the analytics1-e6-eqiad vlan, so the definition for that vlan needs to get added under "vlans" before it can be used under "interfaces".

Did the sre.network.configure-switch-interfaces cookbook report any error when run? Seems it didn't configure anything, which makes sense, but I wonder was there any indication of a problem?

Anyway I ran homer against the switch which added all the required config. On a new attempt the DHCP worked fine, during reimage I can ping the server:

cmooney@cumin1001:~$ ping -4 an-worker1160.eqiad.wmnet
PING  (10.64.155.2) 56(84) bytes of data.
64 bytes from an-worker1160.eqiad.wmnet (10.64.155.2): icmp_seq=1 ttl=61 time=0.347 ms
64 bytes from an-worker1160.eqiad.wmnet (10.64.155.2): icmp_seq=2 ttl=61 time=0.338 ms

The reimage itself is failing for some reason though. Linux starts to load, looking at the virtual vga console:

image.png (123×627 px, 9 KB)

But nothing ever shows on the virtual tty console via ssh. I'm not 100% what the reason for that is, but I think network-wise things are ok for these hosts.

ops-monitoring-bot added a comment.Nov 27 2023, 2:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1160.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 27 2023, 3:01 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1157.eqiad.wmnet with OS bullseye

Jclark-ctr added a comment.Nov 27 2023, 3:09 PM

@cmooney thanks for looking at it previously we where not even getting to the Debian installer the sre.network.configure-switch-interface was ran without errors during the provisioning script

Jclark-ctr claimed this task.Nov 27 2023, 3:57 PM
Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Nov 27 2023, 4:36 PM
ops-monitoring-bot added a comment.Nov 27 2023, 4:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host an-worker1157.eqiad.wmnet with OS bullseye

Papaul subscribed.Nov 27 2023, 6:27 PM
  • first issue on 1157 the serial port address was set to COM1 and not com2
  • second issue on 1157 boot order was set to network then disk making the server to keep rebooting into pxe after each OS install
  • third The way the raid was setup was making to not boot to the correct disk (raid 1 on the ssd and raid 0 on each othe disk) I had to delete raid configuration and create only raid 1 on the ssd and install the os
  • Fourth issue ii57 is sending the CSR on the puppetmaster and not on the puppetserver when you enter puppet 7 ( i am having John and Richardo check this)

@Jclark-ctr can you double check the other hosts for issues 1 to 3 and fix but do not start the OS install until I hear something from the automation team.
Thanks

ops-monitoring-bot added a comment.Nov 27 2023, 8:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host an-worker1157.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 27 2023, 9:07 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host an-worker1157.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1157 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 27 2023, 9:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhathaway@cumin1001 for host an-worker1157.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 27 2023, 10:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jhathaway@cumin1001 for host an-worker1157.eqiad.wmnet with OS bullseye completed:

  • an-worker1157 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311272159_jhathaway_565933_an-worker1157.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
gerritbot added a comment.Nov 28 2023, 3:02 PM

Change 978061 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/cookbooks@master] Set tty serial to com2 for Dell Poweredge R750 model variants

https://gerrit.wikimedia.org/r/978061

gerritbot added a project: Patch-For-Review.Nov 28 2023, 3:02 PM
gerritbot added a comment.Nov 28 2023, 3:56 PM

Change 978061 merged by jenkins-bot:

[operations/cookbooks@master] Set tty serial to com2 for Dell Poweredge R750 model variants

https://gerrit.wikimedia.org/r/978061

Maintenance_bot removed a project: Patch-For-Review.Nov 28 2023, 4:10 PM
ops-monitoring-bot added a comment.Nov 28 2023, 8:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1158.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 28 2023, 8:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 28 2023, 8:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1158.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 28 2023, 9:42 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1161 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 28 2023, 11:15 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1158.eqiad.wmnet with OS bullseye completed:

  • an-worker1158 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311282104_jclark_1166872_an-worker1158.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 28 2023, 11:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1159.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 28 2023, 11:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1160.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 28 2023, 11:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 12:04 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1159.eqiad.wmnet with OS bullseye completed:

  • an-worker1159 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311282342_jclark_1236709_an-worker1159.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
RobH unsubscribed.Nov 29 2023, 12:27 AM
ops-monitoring-bot added a comment.Nov 29 2023, 12:27 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1160.eqiad.wmnet with OS bullseye completed:

  • an-worker1160 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290003_jclark_1242782_an-worker1160.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 1:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1161 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 1:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1162.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1163.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:38 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1165.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:40 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1164.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1166.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:42 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1167.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1168.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1169.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:43 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1170.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1171.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1173.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1174.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 1:46 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1175.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 2:12 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1164.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1164 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290203_jclark_1296552_an-worker1164.out
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 2:17 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1163.eqiad.wmnet with OS bullseye completed:

  • an-worker1163 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290155_jclark_1295603_an-worker1163.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:18 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1168.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1168 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290210_jclark_1297646_an-worker1168.out
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 2:21 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1166.eqiad.wmnet with OS bullseye completed:

  • an-worker1166 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290158_jclark_1297090_an-worker1166.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:24 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1169.eqiad.wmnet with OS bullseye completed:

  • an-worker1169 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290201_jclark_1297808_an-worker1169.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:26 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1165.eqiad.wmnet with OS bullseye completed:

  • an-worker1165 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290203_jclark_1296547_an-worker1165.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:27 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1175.eqiad.wmnet with OS bullseye completed:

  • an-worker1175 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290209_jclark_1299346_an-worker1175.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Nov 29 2023, 2:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1164.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 2:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1168.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 2:28 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1173.eqiad.wmnet with OS bullseye completed:

  • an-worker1173 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290204_jclark_1298728_an-worker1173.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:30 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1167.eqiad.wmnet with OS bullseye completed:

  • an-worker1167 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290205_jclark_1297359_an-worker1167.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:31 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1174.eqiad.wmnet with OS bullseye completed:

  • an-worker1174 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290209_jclark_1299289_an-worker1174.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1170.eqiad.wmnet with OS bullseye completed:

  • an-worker1170 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290210_jclark_1298115_an-worker1170.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 2:58 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1162.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1162 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 2:58 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1161 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 3:05 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1171.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1171 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 3:05 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 3:08 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1168.eqiad.wmnet with OS bullseye completed:

  • an-worker1168 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290245_jclark_1335823_an-worker1168.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 3:11 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1164.eqiad.wmnet with OS bullseye completed:

  • an-worker1164 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311290248_jclark_1335850_an-worker1164.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 3:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 3:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1162.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 3:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1171.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 3:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 4:32 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1161 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 4:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1162.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1162 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 4:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1171.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1171 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 4:33 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1172.eqiad.wmnet with OS bullseye executed with errors:

  • an-worker1172 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Nov 29 2023, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1171.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1162.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host an-worker1172.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 29 2023, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1161.eqiad.wmnet with OS bullseye completed:

  • an-worker1161 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291539_jclark_1670612_an-worker1161.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1162.eqiad.wmnet with OS bullseye completed:

  • an-worker1162 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291542_jclark_1670599_an-worker1162.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 4:07 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1172.eqiad.wmnet with OS bullseye completed:

  • an-worker1172 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291547_jclark_1670568_an-worker1172.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
ops-monitoring-bot added a comment.Nov 29 2023, 4:08 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host an-worker1171.eqiad.wmnet with OS bullseye completed:

  • an-worker1171 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202311291544_jclark_1670621_an-worker1171.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Jclark-ctr closed this task as Resolved.Nov 30 2023, 3:20 PM
Jclark-ctr updated the task description. (Show Details)
gerritbot added a comment.Jan 26 2024, 9:51 AM

Change 993065 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove obsolete Hiera entries

https://gerrit.wikimedia.org/r/993065

gerritbot added a project: Patch-For-Review.Jan 26 2024, 9:51 AM
Stevemunene mentioned this in T353776: Bring an-worker11[57-75] into service.Jan 29 2024, 12:23 PM
gerritbot added a comment.Jan 31 2024, 11:27 AM

Change 993065 merged by Muehlenhoff:

[operations/puppet@production] Remove obsolete Hiera entries

https://gerrit.wikimedia.org/r/993065

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL