Page MenuHomePhabricator
Create Task
Maniphest T300254

Upgrade codfw1dev to bullseye
Closed, ResolvedPublic
Actions

Description

In preparation for upgrading eqiad1 to the same, upgrade codfw1dev to bullseye.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
P:prometheus::ops: fix powerdns-auth portoperations/puppetproduction+1 -1
dnsrecursor: change webserver listening addressoperations/puppetproduction+1 -1
dnsrecursor: add built-in webserver supportoperations/puppetproduction+35 -5
openstack pdns_auth: fix prometheus monitoringoperations/puppetproduction+18 -6
Bare minimum port to Python 3 to support Debian Bullseyeoperations/debs/prometheus-pdns-rec-exportermaster+27 -13
pdns: support bullseyeoperations/puppetproduction+13 -7
P:openstack: fix novaenv pathoperations/puppetproduction+5 -1
P:openstack::galera: fix monitoring process name on bullseyeoperations/puppetproduction+5 -10
O:openstack::services: don't use pdns prometheus exporters on bullseyeoperations/puppetproduction+19 -4
openstack: force start barbican and trove servicesoperations/puppetproduction+4 -3
Keystone/victoria/bullseye: brute-force replace init scriptsoperations/puppetproduction+504 -9
openstack: keystone: set bind port for uwsgi processoperations/puppetproduction+38 -2
typo fix: apache2/keystoneoperations/puppetproduction+1 -1
profile::openstack::eqiad1::keystone::wsgi_server: 'keystone'operations/puppetproduction+4 -0
Another piece of removing Apache from cloudcontrol/bullseyeoperations/puppetproduction+0 -5
Exclude keystone::apache profile on Bullseye hostsoperations/puppetproduction+14 -2
Keystone: don't use apache/mod_wsgi on bullseyeoperations/puppetproduction+23 -13
admin: enforce-users-groups: Add new system user rangeoperations/puppetproduction+22 -13
upgrade codfw1dev to bullseyeoperations/puppetproduction+216 -0
upgrade codfw1dev to bullseyeoperations/puppetproduction+0 -1
openstack: remove few more python 2 packagesoperations/puppetproduction+1 -7
backy2: don't install python3-crypto in bullseyeoperations/puppetproduction+7 -1
upgrade codfw1dev to bullseyeoperations/puppetproduction+3 -2
upgrade codfw1dev to bullseyeoperations/puppetproduction+4 -2
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jan 27 2022, 6:50 PM

Change 757702 had a related patch set uploaded (by Michael DiPietro; author: Michael DiPietro):

[operations/puppet@production] upgrade codfw1dev to bullseye

https://gerrit.wikimedia.org/r/757702

gerritbot added a project: Patch-For-Review.Jan 27 2022, 6:50 PM
gerritbot added a comment.Jan 27 2022, 6:56 PM

Change 757702 merged by Michael DiPietro:

[operations/puppet@production] upgrade codfw1dev to bullseye

https://gerrit.wikimedia.org/r/757702

Maintenance_bot removed a project: Patch-For-Review.Jan 27 2022, 7:10 PM
gerritbot added a comment.Jan 27 2022, 7:16 PM

Change 757729 had a related patch set uploaded (by Michael DiPietro; author: Michael DiPietro):

[operations/puppet@production] upgrade codfw1dev to bullseye

https://gerrit.wikimedia.org/r/757729

gerritbot added a project: Patch-For-Review.Jan 27 2022, 7:16 PM

Change 757729 merged by Michael DiPietro:

[operations/puppet@production] upgrade codfw1dev to bullseye

https://gerrit.wikimedia.org/r/757729

gerritbot added a comment.Jan 27 2022, 8:07 PM

Change 757739 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: remove few more python 2 packages

https://gerrit.wikimedia.org/r/757739

gerritbot added a comment.Jan 27 2022, 8:17 PM

Change 757742 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] backy2: don't install python3-crypto in bullseye

https://gerrit.wikimedia.org/r/757742

gerritbot added a comment.Jan 27 2022, 8:27 PM

Change 757742 merged by Andrew Bogott:

[operations/puppet@production] backy2: don't install python3-crypto in bullseye

https://gerrit.wikimedia.org/r/757742

gerritbot added a comment.Jan 27 2022, 8:37 PM

Change 757745 had a related patch set uploaded (by Michael DiPietro; author: Michael DiPietro):

[operations/puppet@production] upgrade codfw1dev to bullseye

https://gerrit.wikimedia.org/r/757745

gerritbot added a comment.Jan 27 2022, 8:40 PM

Change 757739 merged by Michael DiPietro:

[operations/puppet@production] openstack: remove few more python 2 packages

https://gerrit.wikimedia.org/r/757739

gerritbot added a comment.Jan 27 2022, 8:40 PM

Change 757745 abandoned by Michael DiPietro:

[operations/puppet@production] upgrade codfw1dev to bullseye

Reason:

https://gerrit.wikimedia.org/r/757745

ops-monitoring-bot added a comment.Jan 27 2022, 10:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudcontrol2001-dev.wikimedia.org with OS buster

ops-monitoring-bot added a comment.Jan 27 2022, 11:06 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudcontrol2001-dev.wikimedia.org with OS buster executed with errors:

  • cloudcontrol2001-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 27 2022, 11:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudcontrol2001-dev.wikimedia.org with OS buster

ops-monitoring-bot added a comment.Jan 27 2022, 11:21 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudcontrol2001-dev.wikimedia.org with OS buster executed with errors:

  • cloudcontrol2001-dev (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Jan 27 2022, 11:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudcontrol2001-dev.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Jan 28 2022, 1:46 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudcontrol2001-dev.wikimedia.org with OS bullseye executed with errors:

  • cloudcontrol2001-dev (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201272322_andrew_3448_cloudcontrol2001-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
taavi added a subtask: T300308: prometheus-rabbitmq-exporter for Debian Bullseye.Jan 28 2022, 5:20 AM
• aborrero closed subtask T300308: prometheus-rabbitmq-exporter for Debian Bullseye as Resolved.Jan 28 2022, 10:28 AM
ops-monitoring-bot added a comment.Jan 28 2022, 10:29 AM

Cookbook cookbooks.sre.hosts.reimage started by mdipietro@cumin1001 for host cloudcontrol2001-dev.wikimedia.org with OS bullseye executed with errors:

  • cloudcontrol2001-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201271818_mdipietro_17188_cloudcontrol2001-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • The reimage failed, see the cookbook logs for the details
taavi subscribed.Jan 28 2022, 11:53 AM

The remaining Puppet failure seems to be caused by the Glance system user using an unusually high user ID:

taavi@cloudcontrol2001-dev ~ $ id glance
uid=64062(glance) gid=64062(glance) groups=64062(glance)

compared to for example

taavi@cloudcontrol2003-dev ~ $ id glance
uid=496(glance) gid=64062(glance) groups=64062(glance)

The current IDs seems to be hardcoded in the packaging:

# https://salsa.debian.org/openstack-team/debian/openstack-pkg-tools/-/blob/debian/victoria/pkgos_func#L855
	"glance")
		ADDGROUP_PARAM="--gid 64062"
		ADDUSER_PARAM="--uid 64062"
		;;

Not sure what changed between buster and bullseye, and why nova and cinder (others hardcoded in the same place) are using IDs under 500. Note that

taavi@cloudcontrol2001-dev ~ $ id 496
uid=496(srv-networktests) gid=1001(srv-networktests) groups=1001(srv-networktests)
taavi@cloudcontrol2003-dev ~ $ id srv-networktests 
uid=494(srv-networktests) gid=1001(srv-networktests) groups=1001(srv-networktests)
• aborrero subscribed.Jan 28 2022, 12:28 PM

We've had other clashes in the past between LDAP users and openstack package's users, see T230003: openstack: cleanup neutron user and rOPUP4b793df105ab: openstack/buster/nova: Create 'nova' system user in puppet for a couple examples.

gerritbot added a comment.Jan 28 2022, 1:27 PM

Change 757899 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] admin: enforce-users-groups: Add new system user range

https://gerrit.wikimedia.org/r/757899

gerritbot added a comment.Jan 28 2022, 3:43 PM

Change 757899 merged by Andrew Bogott:

[operations/puppet@production] admin: enforce-users-groups: Add new system user range

https://gerrit.wikimedia.org/r/757899

Andrew subscribed.Jan 28 2022, 3:51 PM

I spent most of my day yesterday fighting with Galera, trying to get it to sync over the boundary between mariadb 10.3 (buster) and mariadb 10.5 (bullseye).

It CAN work; Galera doesn't have a problem. What is a problem is the crashlog; mariadb got upset on the new host and tried to replay the crashlog, which is strictly versioned. This resulted in some angry messages about 'You can't upgrade and recover from a crash at the same time' and a failure to start.

I am pretty sure that the right solution for this is an in-place upgrade of all nodes before reimaging. Then we can follow this guide to get things syncing again:

https://fromdual.com/upgrading-from-mariadb-10.4-to-mariadb-10.5-galera-cluster

Once all the nodes are on 10.5 we should be able to reimage any one node without causing a version-mismatch freakout.

gerritbot added a comment.Jan 28 2022, 7:16 PM

Change 757956 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Keystone: don't use apache/mod_wsgi on bullseye

https://gerrit.wikimedia.org/r/757956

gerritbot added a comment.Jan 28 2022, 7:20 PM

Change 757956 merged by Andrew Bogott:

[operations/puppet@production] Keystone: don't use apache/mod_wsgi on bullseye

https://gerrit.wikimedia.org/r/757956

gerritbot added a comment.Jan 28 2022, 7:28 PM

Change 757959 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Exclude keystone::apache profile on Bullseye hosts

https://gerrit.wikimedia.org/r/757959

gerritbot added a comment.Jan 28 2022, 7:33 PM

Change 757959 merged by Andrew Bogott:

[operations/puppet@production] Exclude keystone::apache profile on Bullseye hosts

https://gerrit.wikimedia.org/r/757959

gerritbot added a comment.Jan 28 2022, 7:36 PM

Change 757962 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Another piece of removing Apache from cloudcontrol/bullseye

https://gerrit.wikimedia.org/r/757962

gerritbot added a comment.Jan 28 2022, 7:38 PM

Change 757962 merged by Andrew Bogott:

[operations/puppet@production] Another piece of removing Apache from cloudcontrol/bullseye

https://gerrit.wikimedia.org/r/757962

gerritbot added a comment.Jan 28 2022, 7:48 PM

Change 757968 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] profile::openstack::eqiad1::keystone::wsgi_server: 'keystone'

https://gerrit.wikimedia.org/r/757968

gerritbot added a comment.Jan 28 2022, 7:51 PM

Change 757968 merged by Andrew Bogott:

[operations/puppet@production] profile::openstack::eqiad1::keystone::wsgi_server: 'keystone'

https://gerrit.wikimedia.org/r/757968

gerritbot added a comment.Jan 28 2022, 7:53 PM

Change 757969 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] typo fix: apache2/keystone

https://gerrit.wikimedia.org/r/757969

gerritbot added a comment.Jan 28 2022, 7:54 PM

Change 757969 merged by Andrew Bogott:

[operations/puppet@production] typo fix: apache2/keystone

https://gerrit.wikimedia.org/r/757969

gerritbot added a comment.Jan 28 2022, 9:18 PM

Change 757982 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: keystone: set bind port for uwsgi process

https://gerrit.wikimedia.org/r/757982

gerritbot added a comment.Jan 29 2022, 1:20 AM

Change 757982 merged by Andrew Bogott:

[operations/puppet@production] openstack: keystone: set bind port for uwsgi process

https://gerrit.wikimedia.org/r/757982

gerritbot added a comment.Jan 29 2022, 2:14 AM

Change 758004 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Keystone/victoria/bullseye: brute-force replace init scripts

https://gerrit.wikimedia.org/r/758004

gerritbot added a comment.Jan 29 2022, 2:18 AM

Change 758004 merged by Andrew Bogott:

[operations/puppet@production] Keystone/victoria/bullseye: brute-force replace init scripts

https://gerrit.wikimedia.org/r/758004

ops-monitoring-bot added a comment.Jan 29 2022, 4:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudcontrol2003-dev.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Jan 29 2022, 5:25 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudcontrol2003-dev.wikimedia.org with OS bullseye executed with errors:

  • cloudcontrol2003-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201290445_andrew_24112_cloudcontrol2003-dev.out
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Jan 29 2022, 8:56 AM

Change 758036 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack: force start barbican and trove services

https://gerrit.wikimedia.org/r/758036

ops-monitoring-bot added a comment.Jan 29 2022, 4:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudcontrol2004-dev.wikimedia.org with OS bullseye

gerritbot added a comment.Jan 29 2022, 4:58 PM

Change 758036 merged by Andrew Bogott:

[operations/puppet@production] openstack: force start barbican and trove services

https://gerrit.wikimedia.org/r/758036

gerritbot added a comment.Jan 29 2022, 5:08 PM

Change 758049 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack: fix novaenv path

https://gerrit.wikimedia.org/r/758049

ops-monitoring-bot added a comment.Jan 29 2022, 5:57 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudcontrol2004-dev.wikimedia.org with OS bullseye completed:

  • cloudcontrol2004-dev (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201291656_andrew_26799_cloudcontrol2004-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Jan 29 2022, 6:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudservices2003-dev.wikimedia.org with OS bullseye

gerritbot added a comment.Jan 29 2022, 6:05 PM

Change 758052 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:openstack::galera: fix monitoring process name on bullseye

https://gerrit.wikimedia.org/r/758052

ops-monitoring-bot added a comment.Jan 29 2022, 9:08 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudservices2003-dev.wikimedia.org with OS bullseye executed with errors:

  • cloudservices2003-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201291804_andrew_5834_cloudservices2003-dev.out
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Jan 29 2022, 9:35 PM

Change 758063 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] pdns: support bullseye

https://gerrit.wikimedia.org/r/758063

gerritbot added a comment.Jan 29 2022, 9:55 PM

Change 758068 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/debs/prometheus-pdns-rec-exporter@master] Bare minimun port to Python 3 to support Debian Bullseye

https://gerrit.wikimedia.org/r/758068

gerritbot added a comment.Jan 31 2022, 3:56 PM

Change 758510 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] O:openstack::services: don't use pdns prometheus exporters on bullseye

https://gerrit.wikimedia.org/r/758510

gerritbot added a comment.Jan 31 2022, 4:57 PM

Change 758510 merged by Andrew Bogott:

[operations/puppet@production] O:openstack::services: don't use pdns prometheus exporters on bullseye

https://gerrit.wikimedia.org/r/758510

gerritbot added a comment.Jan 31 2022, 5:09 PM

Change 758052 merged by Andrew Bogott:

[operations/puppet@production] P:openstack::galera: fix monitoring process name on bullseye

https://gerrit.wikimedia.org/r/758052

gerritbot added a comment.Jan 31 2022, 5:19 PM

Change 758049 merged by Andrew Bogott:

[operations/puppet@production] P:openstack: fix novaenv path

https://gerrit.wikimedia.org/r/758049

gerritbot added a comment.Jan 31 2022, 5:42 PM

Change 758063 merged by Ssingh:

[operations/puppet@production] pdns: support bullseye

https://gerrit.wikimedia.org/r/758063

gerritbot added a comment.Jan 31 2022, 9:25 PM

Change 758068 abandoned by Majavah:

[operations/debs/prometheus-pdns-rec-exporter@master] Bare minimum port to Python 3 to support Debian Bullseye

Reason:

Using the internal prometheus metrics in powerdns seems indeed the best solution. Thanks!

https://gerrit.wikimedia.org/r/758068

ops-monitoring-bot added a comment.Jan 31 2022, 10:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Jan 31 2022, 11:04 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye executed with errors:

  • cloudnet2004-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201312212_andrew_18938_cloudnet2004-dev.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 1 2022, 2:18 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 1 2022, 3:36 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye executed with errors:

  • cloudnet2004-dev (FAIL)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010218_andrew_21035_cloudnet2004-dev.out
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 1 2022, 3:37 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 1 2022, 5:08 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye executed with errors:

  • cloudnet2004-dev (FAIL)
    • Downtimed on Icinga
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010337_andrew_2994_cloudnet2004-dev.out
    • Checked BIOS boot parameters are back to normal
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 1 2022, 5:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 1 2022, 5:56 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudnet2004-dev.codfw.wmnet with OS bullseye completed:

  • cloudnet2004-dev (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202011726_andrew_25862_cloudnet2004-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 1 2022, 9:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudnet2002-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 1 2022, 10:48 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudnet2002-dev.codfw.wmnet with OS bullseye executed with errors:

  • cloudnet2002-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202012155_andrew_8304_cloudnet2002-dev.out
    • The reimage failed, see the cookbook logs for the details
elukey subscribed.Feb 3 2022, 7:56 AM

@Andrew Hi! I have acked some alerts in icinga, they show up as "silenced" but if possible (during this task) could you ack them once in a while to reduce the noise in the icinga page?

ops-monitoring-bot added a comment.Feb 4 2022, 6:52 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudservices2002-dev.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Feb 4 2022, 7:44 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudservices2002-dev.wikimedia.org with OS bullseye executed with errors:

  • cloudservices2002-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202041852_andrew_15502_cloudservices2002-dev.out
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 5 2022, 5:41 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt2001-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 5 2022, 6:09 AM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt2001-dev.codfw.wmnet with OS bullseye executed with errors:

  • cloudvirt2001-dev (FAIL)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 5 2022, 6:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt2001-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 5 2022, 4:54 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt2001-dev.codfw.wmnet with OS bullseye completed:

  • cloudvirt2001-dev (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202050611_andrew_14326_cloudvirt2001-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 5 2022, 5:53 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt2001-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 5 2022, 6:48 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt2001-dev.codfw.wmnet with OS bullseye completed:

  • cloudvirt2001-dev (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202051753_andrew_16747_cloudvirt2001-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 5 2022, 7:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt2002-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 5 2022, 8:15 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt2002-dev.codfw.wmnet with OS bullseye completed:

  • cloudvirt2002-dev (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202051929_andrew_31252_cloudvirt2002-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 5 2022, 9:28 PM

Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host cloudvirt2003-dev.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 5 2022, 10:10 PM

Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host cloudvirt2003-dev.codfw.wmnet with OS bullseye completed:

  • cloudvirt2003-dev (PASS)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202052128_andrew_17772_cloudvirt2003-dev.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Andrew added a comment.Feb 8 2022, 10:42 PM

This is done for all cloudcontrol, cloudservices, cloudnet, and cloudvirt hosts in codfw1dev. The puppet manifests have been updated so we should be able to do the same in eqiad.

I managed to totally break/lose the existing cloud puppetmaster while doing this but I don't think there's any real lesson to learn from that other than 'make sure cloudvirts are evacuated before rebuilding'.

One important caveats for doing this in eqiad1: Galera is happy with in-place upgrades but does not like to sync from an existing Buster node to a new Bullseye node. The solution is to upgrade all cloudcontrols in place first and then reimage one by one.

gerritbot added a comment.Feb 9 2022, 12:28 PM

Change 761313 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] openstack pdns_auth: fix prometheus monitoring

https://gerrit.wikimedia.org/r/761313

gerritbot added a comment.Feb 9 2022, 12:49 PM

Change 761315 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] dnsrecursor: add built-in webserver support

https://gerrit.wikimedia.org/r/761315

gerritbot added a comment.Feb 9 2022, 5:21 PM

Change 761313 merged by Andrew Bogott:

[operations/puppet@production] openstack pdns_auth: fix prometheus monitoring

https://gerrit.wikimedia.org/r/761313

gerritbot added a comment.Feb 9 2022, 5:26 PM

Change 761315 merged by Andrew Bogott:

[operations/puppet@production] dnsrecursor: add built-in webserver support

https://gerrit.wikimedia.org/r/761315

gerritbot added a comment.Feb 17 2022, 10:36 PM

Change 763612 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] dnsrecursor: change webserver listening address

https://gerrit.wikimedia.org/r/763612

gerritbot added a comment.Feb 17 2022, 10:42 PM

Change 763612 merged by Andrew Bogott:

[operations/puppet@production] dnsrecursor: change webserver listening address

https://gerrit.wikimedia.org/r/763612

gerritbot added a comment.Mar 7 2022, 5:56 PM

Change 768770 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:prometheus::ops: fix powerdns-auth port

https://gerrit.wikimedia.org/r/768770

gerritbot added a comment.Mar 7 2022, 6:13 PM

Change 768770 merged by Andrew Bogott:

[operations/puppet@production] P:prometheus::ops: fix powerdns-auth port

https://gerrit.wikimedia.org/r/768770

Andrew closed this task as Resolved.Mar 7 2022, 10:46 PM
Andrew claimed this task.
• aborrero changed the status of subtask T304598: cloudgw: upgrade servers to Debian 11 Bullseye from Open to In Progress.Mar 24 2022, 12:54 PM
• aborrero closed subtask T304598: cloudgw: upgrade servers to Debian 11 Bullseye as Resolved.Apr 6 2022, 12:23 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits