Upgrade cloudelastic clusters to Debian Bullseye
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	RKemper
	May 26 2022, 6:46 PM

Description

In T308606 we created a new rolling reimage operation for our generic rolling-operation cookbook; let's use that to upgrade the cluster automatically.

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T248925 Make MediaWiki release tarball compatible with PHP 8.0
Resolved	Jdforrester-WMF	T300463 Make PHP 8.0 voting on MW master
Resolved	None	T283275 Make MW master tests pass on PHP 8.0
Resolved	Reedy	T268861 CirrusSearch uses Elastica's Match class
Resolved	Reedy	T268863 Translate uses Elastica's Match class
Resolved	matthiasmullie	T268866 WikibaseMediaInfo uses Elastica's Match class
Invalid	None	T268864 WikibaseCirrusSearch uses Elastica's Match class
Resolved	Reedy	T268865 WikibaseLexemeCirrusSearch uses Elastica's Match class
Resolved	EBernhardson	T271777 Bump rufin/elastica (and related libraries) to versions that support PHP 8.0
Resolved	Gehel	T263142 [EPIC] Upgrade Elasticsearch to version 7.10
Stalled	None	T302086 Set scap minimum python version to 3.7
Resolved	None	T247045 Migrate all of production metal and VMs to Buster or later
Declined	None	T244736 Migrate Elasticsearch to Debian Buster
Open	None	T291916 Tracking task for Bullseye migrations in production
Resolved	bking	T289135 Upgrade Cirrus Elasticsearch clusters to Debian Bullseye
Resolved	bking	T309343 Upgrade cloudelastic clusters to Debian Bullseye
Resolved	bking	T312298 Update NIC firmware on all Elastic PowerEdge R440 elastic hosts

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye executed with errors:

cloudelastic1004 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Mentioned in SAL (#wikimedia-operations) [2022-05-26T19:46:43Z] <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (2 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage - bking@cumin1001 - T309343

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye executed with errors:

cloudelastic1004 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye executed with errors:

cloudelastic1004 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye executed with errors:

cloudelastic1004 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Mentioned in SAL (#wikimedia-operations) [2022-05-26T20:40:54Z] <inflatador> bking@install1003 removed cloudelastic1004.conf pxe config file T309343

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye executed with errors:

cloudelastic1004 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Looking at the console, the installer keeps coming up in interactive mode. I tried clicking through, but it said it couldn't download the preseed file. Will raise this issue with the Infrastructure Foundations team tomorrow.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye executed with errors:

cloudelastic1004 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye executed with errors:

cloudelastic1006 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

bking mentioned this in T309627: Find or add cloudelastic metrics to Elasticsearch Memory dashboard.May 31 2022, 2:57 PM

^^ We've had 2 different hosts fail to reimage in the same way.

Per IRC conversation with Infrastructure Foundations, "when you hit this kind of issues your best bet is usually the dcops IRC channel. In this case it might be another occurrence of outdated HW firmwares".

Will check in #wikimedia-dcops now

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye executed with errors:

cloudelastic1006 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye executed with errors:

cloudelastic1006 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye executed with errors:

cloudelastic1006 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye

Updated firmware on cloudelastic1006 to the following
10g 21.40.25.31 to 21.85.21.92
idrac 4.00.00.00 to 5.10.10.00
bios 2.4.8 to 2.14.2

Once those were done, the installer successfully loaded up and is running through things now.

bking added a project: DC-Ops.May 31 2022, 5:05 PM

Per IRC conversation with RobH, we will work together to reimage one cloudelastic host at a time (6 in the cluster). We'll try to keep 5 hosts in the cluster at all times.

Note that the largest cluster, cloudelastic-chi-eqiad , is in red status, so we'll need to address that before any more reimages take place.

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cloudelastic1006.wikimedia.org with OS bullseye completed:

cloudelastic1006 (WARN)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205311651_robh_3130270_cloudelastic1006.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

bking mentioned this in T309648: Restore lost index in cloudelastic.May 31 2022, 6:17 PM

RhinosF1 subscribed.Jun 1 2022, 2:58 PM

MPhamWMF set the point value for this task to 3.Jun 6 2022, 3:43 PM

MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

Mentioned in SAL (#wikimedia-operations) [2022-07-06T17:06:13Z] <inflatador> bking@cloudelastic1006 "restarting elastic services in preparation for cloudelastic reimage T309343"

RobH unsubscribed.Jul 6 2022, 5:37 PM

Mentioned in SAL (#wikimedia-operations) [2022-07-06T18:02:07Z] <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage to bullseye - bking@cumin1001 - T309343

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye executed with errors:

cloudelastic1003 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Mentioned in SAL (#wikimedia-operations) [2022-07-06T18:45:51Z] <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage to bullseye - bking@cumin1001 - T309343

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye executed with errors:

cloudelastic1003 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye executed with errors:

cloudelastic1003 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye executed with errors:

cloudelastic1003 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

bking mentioned this in T312298: Update NIC firmware on all Elastic PowerEdge R440 elastic hosts.Jul 6 2022, 9:08 PM

bking added a subtask: T312298: Update NIC firmware on all Elastic PowerEdge R440 elastic hosts.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye executed with errors:

cloudelastic1003 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye executed with errors:

cloudelastic1003 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye executed with errors:

cloudelastic1003 (FAIL)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Step Zero: See https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Updating_Firmware

Once you've downloaded the o update firmwares via the iDRAC 8 web GUI:

Click "Overview", then go to "Update and Rollback" under "Quick Launch Tasks"

From there, click "update"

NIC updates failed, I tried a different set of drivers instead of Broadcom. The sparse info under Hardware > Nic Slot 2 on the iDRAC suggests the external NIC is a Marvell qLogic, so I used that package.

iDRAC firmware updates failed several times. I had to use the 64-bit Windows package.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1003.wikimedia.org with OS bullseye completed:

cloudelastic1003 (PASS)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207071602_bking_1613518_cloudelastic1003.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2022-07-07T16:48:29Z] <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage to bullseye - bking@cumin1001 - T309343

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1002.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1002.wikimedia.org with OS bullseye completed:

cloudelastic1002 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207071649_bking_1627048_cloudelastic1002.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Mentioned in SAL (#wikimedia-operations) [2022-07-07T18:36:12Z] <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage to bullseye - bking@cumin1001 - T309343

Mentioned in SAL (#wikimedia-operations) [2022-07-08T02:25:30Z] <bking@cumin1001> START - Cookbook sre.elasticsearch.rolling-operation Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage to bullseye - bking@cumin1001 - T309343

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye executed with errors:

cloudelastic1004 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

Mentioned in SAL (#wikimedia-operations) [2022-07-08T03:32:59Z] <bking@cumin1001> END (FAIL) - Cookbook sre.elasticsearch.rolling-operation (exit_code=99) Operation.REIMAGE (1 nodes at a time) for ElasticSearch cluster cloudelastic: cloudelastic cluster reimage to bullseye - bking@cumin1001 - T309343

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host cloudelastic1004.wikimedia.org with OS bullseye completed:

cloudelastic1004 (WARN)
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202207081409_bking_1843956_cloudelastic1004.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB