Page MenuHomePhabricator
Create Task
Maniphest T329863

Q3:rack/setup/install cloudvirtlocal10[01-03]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudlocalstorage10[01-03]

Hostname / Racking / Installation Details

Hostnames: cloudvirtlocal10[01-03]
Racking Proposal: WMCS Racks. They should be racked in separate rows to ensure HA. The goal is to not share a ToR for each one (So be careful of rows E and F).
Networking Setup: 10G, cloud-hosts1-eqiad VLAN
Partitioning/Raid: HW Raid: N
OS Distro: Bullseye
Sub-team Technical Contact: @Andrew

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

cloudvirtlocal1001:
  • - receive in system on procurement task T329126 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::wmcs
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cloudvirtlocal1002:
  • - receive in system on procurement task T329126 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::wmcs
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cloudvirtlocal1003:
  • - receive in system on procurement task T329126 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::wmcs
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Revert "Move cloudvirtlocal1001 back to 'insetup'"operations/puppetproduction+1 -5
cloudvirtlocal: move instance dir to /srv/instancesoperations/puppetproduction+8 -1
cloudvirtlocal: link nova instances dir to /srvoperations/puppetproduction+28 -17
Update profile::openstack::base::nova::instance_dev for new cloudvirtlocalsoperations/puppetproduction+3 -3
Make cloudvirtlocal100[1-3] into actual cloudvirtsoperations/puppetproduction+4 -1
Fix hostname for new cloudvirtlocal hostsoperations/puppetproduction+1 -1
cloudvirt100[1-3] fix yet another partman typooperations/puppetproduction+1 -1
Fix partman for cloudvirtlocal100[1-3]operations/puppetproduction+1 -1
Update partman for cloudvirtlocal100[1-3]operations/puppetproduction+1 -1
Initial puppet setup for cloudvirtlocal100[1-3]operations/puppetproduction+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ops-monitoring-bot added a comment.Apr 12 2023, 7:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 12 2023, 7:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 12 2023, 8:38 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 13 2023, 1:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

gerritbot added a comment.Apr 13 2023, 1:14 PM

Change 908541 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudvirt100[1-3] fix yet another partman typo

https://gerrit.wikimedia.org/r/908541

gerritbot added a project: Patch-For-Review.Apr 13 2023, 1:14 PM

Change 908541 merged by Andrew Bogott:

[operations/puppet@production] cloudvirt100[1-3] fix yet another partman typo

https://gerrit.wikimedia.org/r/908541

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 1:30 PM
ops-monitoring-bot added a comment.Apr 13 2023, 1:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 1:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 2:02 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Apr 13 2023, 3:08 PM

Change 908576 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Fix hostname for new cloudvirtlocal hosts

https://gerrit.wikimedia.org/r/908576

gerritbot added a project: Patch-For-Review.Apr 13 2023, 3:08 PM

Change 908576 merged by Andrew Bogott:

[operations/puppet@production] Fix hostname for new cloudvirtlocal hosts

https://gerrit.wikimedia.org/r/908576

ops-monitoring-bot added a comment.Apr 13 2023, 3:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 3:09 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 3:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 3:11 PM
ops-monitoring-bot added a comment.Apr 13 2023, 3:47 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1003.eqiad.wmnet with OS bullseye completed:

  • cloudvirtlocal1003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304131509_andrew_646206_cloudvirtlocal1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Apr 13 2023, 3:48 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye completed:

  • cloudvirtlocal1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304131511_andrew_646465_cloudvirtlocal1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Apr 13 2023, 3:51 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1002.eqiad.wmnet with OS bullseye completed:

  • cloudvirtlocal1002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304131509_andrew_646129_cloudvirtlocal1002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
gerritbot added a comment.Apr 13 2023, 3:55 PM

Change 908587 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Make cloudvirtlocal100[1-3] into actual cloudvirts

https://gerrit.wikimedia.org/r/908587

gerritbot added a project: Patch-For-Review.Apr 13 2023, 3:55 PM
gerritbot added a comment.Apr 13 2023, 3:58 PM

Change 908587 merged by Andrew Bogott:

[operations/puppet@production] Make cloudvirtlocal100[1-3] into actual cloudvirts

https://gerrit.wikimedia.org/r/908587

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 4:11 PM
gerritbot added a comment.Apr 13 2023, 4:46 PM

Change 908595 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Update profile::openstack::base::nova::instance_dev for new cloudvirtlocals

https://gerrit.wikimedia.org/r/908595

gerritbot added a project: Patch-For-Review.Apr 13 2023, 4:46 PM

Change 908595 merged by Andrew Bogott:

[operations/puppet@production] Update profile::openstack::base::nova::instance_dev for new cloudvirtlocals

https://gerrit.wikimedia.org/r/908595

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 5:11 PM
gerritbot added a comment.Apr 13 2023, 5:57 PM

Change 908608 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudvirtlocal: link nova instances dir to /srv

https://gerrit.wikimedia.org/r/908608

gerritbot added a project: Patch-For-Review.Apr 13 2023, 5:57 PM
gerritbot added a comment.Apr 13 2023, 6:00 PM

Change 908608 merged by Andrew Bogott:

[operations/puppet@production] cloudvirtlocal: link nova instances dir to /srv

https://gerrit.wikimedia.org/r/908608

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 6:10 PM
gerritbot added a comment.Apr 13 2023, 6:23 PM

Change 908612 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] cloudvirtlocal: move instance dir to /srv/instances

https://gerrit.wikimedia.org/r/908612

gerritbot added a project: Patch-For-Review.Apr 13 2023, 6:23 PM
gerritbot added a comment.Apr 13 2023, 6:25 PM

Change 908612 merged by Andrew Bogott:

[operations/puppet@production] cloudvirtlocal: move instance dir to /srv/instances

https://gerrit.wikimedia.org/r/908612

ops-monitoring-bot added a comment.Apr 13 2023, 6:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 6:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Apr 13 2023, 6:29 PM
ops-monitoring-bot added a comment.Apr 13 2023, 7:16 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1002.eqiad.wmnet with OS bullseye completed:

  • cloudvirtlocal1002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304131825_andrew_688779_cloudvirtlocal1002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Apr 13 2023, 7:18 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 13 2023, 7:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 8:14 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 13 2023, 8:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304132015_pt1979_1479363_cloudvirtlocal1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 13 2023, 9:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 9:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 13 2023, 9:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 13 2023, 9:38 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 13 2023, 11:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 12:04 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304132325_pt1979_1611807_cloudvirtlocal1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 12:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

Papaul subscribed.Apr 14 2023, 12:11 AM

@Jclark-ctr when you are next on site can you please replace the DAC cable connecting cloudvirtlocal1001 to the switch?

Thanks

ops-monitoring-bot added a comment.Apr 14 2023, 1:01 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Jclark-ctr added a comment.Apr 14 2023, 1:26 PM

@Papaul replaced dac cable

ops-monitoring-bot added a comment.Apr 14 2023, 1:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Apr 14 2023, 1:42 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 1:42 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 1:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 1:45 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 1:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 2:11 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 2:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 2:29 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 97
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 2:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 3:26 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 4:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 4:38 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 4:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 5:02 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 5:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 5:10 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 5:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 5:15 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 5:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 5:25 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 5:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 6:03 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 6:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 6:17 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 14 2023, 6:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 14 2023, 6:52 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye completed:

  • cloudvirtlocal1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304141818_pt1979_2398486_cloudvirtlocal1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Papaul updated the task description. (Show Details)Apr 14 2023, 8:16 PM
Andrew closed subtask T334694: Tenant networking not working on cloudvirtlocal hosts as Resolved.Apr 14 2023, 8:17 PM
Andrew updated the task description. (Show Details)
Papaul closed this task as Resolved.Apr 14 2023, 8:21 PM

Complete

ops-monitoring-bot added a comment.Apr 18 2023, 6:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 18 2023, 6:28 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirtlocal1001 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Apr 18 2023, 6:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 18 2023, 7:03 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cloudvirtlocal1001.eqiad.wmnet with OS bullseye completed:

  • cloudvirtlocal1001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304181828_pt1979_2145567_cloudvirtlocal1001.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Papaul closed subtask T334696: cloudvirtlocal1001.eqiad.wmnet tends to get stuck on boot as Resolved.Apr 18 2023, 7:07 PM
gerritbot added a comment.Apr 18 2023, 7:15 PM

Change 909739 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Revert "Move cloudvirtlocal1001 back to 'insetup'"

https://gerrit.wikimedia.org/r/909739

gerritbot added a project: Patch-For-Review.Apr 18 2023, 7:15 PM

Change 909739 merged by Andrew Bogott:

[operations/puppet@production] Revert "Move cloudvirtlocal1001 back to 'insetup'"

https://gerrit.wikimedia.org/r/909739

ops-monitoring-bot added a comment.Apr 18 2023, 7:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirtlocal1003.eqiad.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Apr 18 2023, 7:30 PM
ops-monitoring-bot added a comment.Apr 18 2023, 8:13 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirtlocal1003.eqiad.wmnet with OS bullseye completed:

  • cloudvirtlocal1003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202304181924_andrew_2061094_cloudvirtlocal1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits