Page MenuHomePhabricator
Create Task
Maniphest T281276

Upgrade cloud-vps openstack hosts to Debian 'Bullseye'
Closed, ResolvedPublic
Actions

Description

We are currently running OpenStack version 'Victoria'. Victoria is the release that is packaged for both Buster and Bullseye; for future OpenStack upgrades we will need our control plane on Bullseye.

Details

SubjectRepoBranchLines +/-
Rename nics for cloudvirt1023operations/puppetproduction+3 -2
Update nic labels for cloudvirt1023/bullseyeoperations/puppetproduction+3 -3
netboot: switch cloudvirt102[1-9] partman recipeoperations/puppetproduction+1 -1
P:wmcs::prometheus: update pdns portsoperations/puppetproduction+2 -2
Set profile::openstack::XXX::keystone::wsgi_server to 'keystone' everywhereoperations/puppetproduction+2 -23
Move service name openstack.eqiad1.wikimediacloud.org to cloudcontrol1005operations/dnsmaster+1 -1
cloudcontrol1004: override profile::openstack::eqiad1::keystone::wsgi_server:operations/puppetproduction+4 -0
cloudcontrols: override profile::openstack::eqiad1::keystone::wsgi_server:operations/puppetproduction+5 -0
Openstack Cinder and Nova: tweak cgroup kernel settings on Bullseyeoperations/puppetproduction+22 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedAndrewT305828 upgrade cloud-vps openstack to Openstack version 'Yoga'
 ResolvedAndrewT296561 upgrade cloud-vps openstack to Openstack version 'Xena'
 ResolvedrookT281275 upgrade cloud-vps openstack to Openstack version 'Wallaby'
 ResolvedAndrewT281276 Upgrade cloud-vps openstack hosts to Debian 'Bullseye'
 ResolvedAndrewT300254 Upgrade codfw1dev to bullseye
 ResolvedtaaviT300308 prometheus-rabbitmq-exporter for Debian Bullseye
 ResolvedaborreroT304598 cloudgw: upgrade servers to Debian 11 Bullseye
 ResolvedaborreroT305494 interface renaming via systemd .link file can race with sysctl parameters
 ResolveddcaroT302146 Galera on cloudcontrol1004 going out of sync
 ResolveddcaroT302369 cloudcontrol1003 - Check for VMs leaked by the nova-fullstack test
 ResolveddcaroT302377 [neutron] Add alert for neutron-rpc-server service being down on cloudcontrols (and neutron agents)
 ResolveddcaroT302378 [neutron] Document in fullstack runbook the neutron agents being down
 DuplicatedcaroT302380 cloudcontrol1005 - prometheus-openstack-exporter service is down
 ResolveddcaroT302381 cloudcontrol1005 - create alert for prometheus-openstack-exporter service
 ResolvedtaaviT302493 hook up prometheus @ cloudmetrics* to an alertmanager
 ResolveddcaroT302382 icinga alert: Check for snapshots leaked by cinder backup agent
 ResolvedaborreroT302482 openstack db: figure out new versions for galera & mariadb
 ResolveddcaroT302178 prometheus-openstack-exporter No module named 'urlparse'
 In ProgressNoneT333975 Cloud VPS: refresh openstack resources grafana dashboard
ResolvedAndrewT335943 prometheus-openstack-exporter: collected data shows regular null intervals
 OpenAndrewT335978 openstack: consider removing references to old hardware from the database
 OpenAndrewT362443 Learn how to do what Taavi does
 OpenNoneT362444 Taavi knowledge transfer: maintain-kubeusers
OpenNoneT362445 Taavi knowledge transfer: Toolforge k8s upgrades
 OpenNoneT362446 Taavi knowledge transfer: toolforge job investigation
 OpenNoneT362447 Taavi knowledge transfer: Toolforge misc services (e.g. mail server)
 OpenNoneT362448 Taavi knowledge transfer: rebuild toolforge docker images
 OpenAndrewT362449 Taavi knowledge transfer: python-flask-keystone, novaproxy, enc api
 OpenNoneT362450 Taavi knowledge transfer: Cloud VPS OpenTofu provider
 OpenAndrewT362452 Taavi knowledge transfer: cloud-vps monitoring
 ResolvedaborreroT339152 Cloud VPS: figure out metrics for openstack API response times and create a grafana dashboard for it
 ResolvedaborreroT340819 Cloud VPS: create alert for when the openstack API response times are too high
 DuplicateaborreroT340611 prometheus: figure out how to clean metrics for decommissioned hardware
 ResolvedAndrewT303296 cloudvirt1016.eqiad.wmnet and cloudvirt1017.eqiad.wmnet fail to PXE boot
 ResolvedAndrewT303773 cloudvirt1024.eqiad.wmnet DHCP problems
 ResolvedAndrewT303776 cloudvirt1025 and cloudvirt1026 fail to pxe boot
 ResolvedAndrewT304420 upgrade cloudnet servers to Debian 11 Bullseye
 ResolvedAndrewT304581 Upgrade cloudvirt-wdqs servers to Debian Bullseye
 ResolvedayounsiT305368 PXE boot failures on cloudvirt-wdqs100[1-3]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Mar 23 2022, 2:18 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:18:40Z] <wm-bot> Set cloudvirt 'cloudvirt1030.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 2:19 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:19:23Z] <wm-bot> Draining 'cloudvirt1031.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 2:20 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:20:08Z] <wm-bot> Set cloudvirt 'cloudvirt1031.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 2:32 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:32:55Z] <wm-bot> Drained 'cloudvirt1030.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 2:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1030.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 2:34 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:34:24Z] <wm-bot> Draining 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 2:35 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:35:09Z] <wm-bot> Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 2:44 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:44:48Z] <wm-bot> Drained 'cloudvirt1031.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 2:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1031.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 2:57 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T14:57:46Z] <wm-bot> Draining 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 3:00 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1030.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1030 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231433_andrew_2889456_cloudvirt1030.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 23 2022, 3:00 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T15:00:40Z] <wm-bot> Set cloudvirt 'cloudvirt1032.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 3:01 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T15:01:48Z] <wm-bot> Drained 'cloudvirt1032.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 3:25 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1031.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1031 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231445_andrew_2892140_cloudvirt1031.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 3:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1032.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 4:31 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1032.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1032 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231550_andrew_2903853_cloudvirt1032.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 23 2022, 4:36 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T16:36:29Z] <wm-bot> Draining 'cloudvirt1033.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 4:36 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T16:36:38Z] <wm-bot> Draining 'cloudvirt1034.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 4:37 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T16:37:14Z] <wm-bot> Set cloudvirt 'cloudvirt1033.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 4:37 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T16:37:22Z] <wm-bot> Set cloudvirt 'cloudvirt1034.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 4:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1047.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 4:51 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T16:51:05Z] <wm-bot> Drained 'cloudvirt1033.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1047.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirt1047 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Mar 23 2022, 4:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1033.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 5:03 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T17:03:12Z] <wm-bot> Drained 'cloudvirt1034.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 5:03 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T17:03:43Z] <wm-bot> Draining 'cloudvirt1035.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 5:04 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T17:04:29Z] <wm-bot> Set cloudvirt 'cloudvirt1035.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 5:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1034.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 5:32 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1028.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1028 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231658_andrew_2915140_cloudvirt1028.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 5:38 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1033.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1033 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231659_andrew_2915254_cloudvirt1033.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 5:50 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1034.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1034 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231707_andrew_2917902_cloudvirt1034.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 23 2022, 5:54 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T17:54:33Z] <wm-bot> Draining 'cloudvirt1036.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 5:55 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T17:55:18Z] <wm-bot> Set cloudvirt 'cloudvirt1036.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 6:13 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T18:13:17Z] <wm-bot> Drained 'cloudvirt1036.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 6:18 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T18:18:20Z] <wm-bot> Draining 'cloudvirt1037.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 6:19 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T18:19:04Z] <wm-bot> Set cloudvirt 'cloudvirt1037.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 6:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1035.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 6:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1036.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 6:43 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T18:43:54Z] <wm-bot> Draining 'cloudvirt1038.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 6:44 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T18:44:39Z] <wm-bot> Set cloudvirt 'cloudvirt1038.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 7:20 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1035.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1035 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231836_andrew_2933834_cloudvirt1035.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 7:20 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1036.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1036 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203231836_andrew_2933843_cloudvirt1036.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 8:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1037.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 8:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1038.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 8:14 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T20:14:50Z] <wm-bot> Draining 'cloudvirt1039.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 8:14 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T20:14:57Z] <wm-bot> Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 8:15 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T20:15:34Z] <wm-bot> Set cloudvirt 'cloudvirt1039.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 8:15 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T20:15:43Z] <wm-bot> Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 8:30 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T20:30:30Z] <wm-bot> Drained 'cloudvirt1039.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 8:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1039.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 8:52 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1038.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1038 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232014_andrew_2948775_cloudvirt1038.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 8:53 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1037.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1037 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232014_andrew_2948781_cloudvirt1037.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 23 2022, 8:54 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T20:54:42Z] <wm-bot> Draining 'cloudvirt1041.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 8:55 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T20:55:29Z] <wm-bot> Set cloudvirt 'cloudvirt1041.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 9:04 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:04:41Z] <wm-bot> Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 9:07 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:07:37Z] <wm-bot> Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1039.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1039 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232031_andrew_2950344_cloudvirt1039.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 23 2022, 9:09 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:09:11Z] <wm-bot> Draining 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 9:12 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:12:05Z] <wm-bot> Set cloudvirt 'cloudvirt1040.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 9:12 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:12:09Z] <wm-bot> Drained 'cloudvirt1040.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 9:18 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1040.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 9:19 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:19:08Z] <wm-bot> Draining 'cloudvirt1042.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 9:19 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:19:53Z] <wm-bot> Set cloudvirt 'cloudvirt1042.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 9:54 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T21:54:23Z] <wm-bot> Drained 'cloudvirt1042.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 9:55 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1040.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1040 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232118_andrew_2962276_cloudvirt1040.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1041.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1042.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 10:06 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:06:20Z] <wm-bot> Draining 'cloudvirt1043.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 10:06 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:06:25Z] <wm-bot> Draining 'cloudvirt1044.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 10:07 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:07:11Z] <wm-bot> Set cloudvirt 'cloudvirt1044.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 10:08 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:08:10Z] <wm-bot> Set cloudvirt 'cloudvirt1043.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 10:12 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:12:04Z] <wm-bot> Draining 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

Stashbot added a comment.Mar 23 2022, 10:12 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:12:50Z] <wm-bot> Set cloudvirt 'cloudvirt1045.eqiad.wmnet' maintenance. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 10:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1043.eqiad.wmnet with OS bullseye

Stashbot added a comment.Mar 23 2022, 10:38 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:38:55Z] <wm-bot> Drained 'cloudvirt1044.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 10:47 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1041.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1041 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232205_andrew_2970731_cloudvirt1041.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 10:48 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1042.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1042 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232205_andrew_2970762_cloudvirt1042.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Mar 23 2022, 10:53 PM

Mentioned in SAL (#wikimedia-cloud) [2022-03-23T22:53:55Z] <wm-bot> Drained 'cloudvirt1045.eqiad.wmnet'. (T281276) - cookbook ran by andrew@buster

ops-monitoring-bot added a comment.Mar 23 2022, 11:16 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1043.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1043 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232235_andrew_2977736_cloudvirt1043.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 23 2022, 11:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1045.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 11:48 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1044.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 23 2022, 11:51 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1046.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2022, 12:27 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1045.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1045 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232348_andrew_2986471_cloudvirt1045.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 24 2022, 12:28 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1044.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1044 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232348_andrew_2986467_cloudvirt1044.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 24 2022, 12:33 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1046.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1046 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202203232351_andrew_2986809_cloudvirt1046.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Mar 24 2022, 1:34 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt-wdqs1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 24 2022, 1:45 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt-wdqs1002.eqiad.wmnet with OS bullseye executed with errors:

  • cloudvirt-wdqs1002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
Andrew closed subtask T303773: cloudvirt1024.eqiad.wmnet DHCP problems as Resolved.Mar 24 2022, 7:59 PM
Andrew closed subtask T303296: cloudvirt1016.eqiad.wmnet and cloudvirt1017.eqiad.wmnet fail to PXE boot as Resolved.Mar 25 2022, 2:54 AM
Andrew added a comment.Mar 28 2022, 2:46 PM

Quick update: The only hosts remaining to upgrade to bullseye are the toolsdb-hosting hypervisors (cloudvirt1019 and 1020) and cloudvirt-wdqs1xxx hosts.

I'm waiting for a consult with data-persistence about the toolsdb hosts.

The cloudvirt-wdqs services are just waiting for a job to complete that's running on one of the hosted VMs.

aborrero changed the status of subtask T302178: prometheus-openstack-exporter No module named 'urlparse' from Open to In Progress.Apr 6 2022, 11:35 AM
dcaro changed the status of subtask T302178: prometheus-openstack-exporter No module named 'urlparse' from In Progress to Open.Apr 14 2022, 8:32 AM
ops-monitoring-bot added a comment.Apr 18 2022, 3:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1020.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 18 2022, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1020.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1020 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204181525_andrew_2870264_cloudvirt1020.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Apr 19 2022, 3:35 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt1019.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Apr 19 2022, 4:09 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt1019.eqiad.wmnet with OS bullseye completed:

  • cloudvirt1019 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202204191534_andrew_3615519_cloudvirt1019.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Andrew closed subtask T304581: Upgrade cloudvirt-wdqs servers to Debian Bullseye as Resolved.Apr 27 2022, 6:25 PM
Andrew closed this task as Resolved.May 27 2022, 4:11 PM
taavi closed subtask T302178: prometheus-openstack-exporter No module named 'urlparse' as Resolved.Jun 22 2022, 5:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL