Maniphest T308770

Reimage relforge elastic hosts from Stretch to Bullseye
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	bking
	May 19 2022, 4:09 PM

Tags

Referenced Files

None

Subscribers

Description

See parent task T289135 for more details

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		None	T248925 Make MediaWiki release tarball compatible with PHP 8.0
		Resolved		Jdforrester-WMF	T300463 Make PHP 8.0 voting on MW master
		Resolved		None	T283275 Make MW master tests pass on PHP 8.0
		Resolved		Reedy	T268861 CirrusSearch uses Elastica's Match class
		Resolved		Reedy	T268863 Translate uses Elastica's Match class
		Resolved		matthiasmullie	T268866 WikibaseMediaInfo uses Elastica's Match class
		Invalid		None	T268864 WikibaseCirrusSearch uses Elastica's Match class
		Resolved		Reedy	T268865 WikibaseLexemeCirrusSearch uses Elastica's Match class
		Resolved		EBernhardson	T271777 Bump rufin/elastica (and related libraries) to versions that support PHP 8.0
		Resolved		Gehel	T263142 [EPIC] Upgrade Elasticsearch to version 7.10
		Stalled		None	T302086 Set scap minimum python version to 3.7
		Resolved		None	T247045 Migrate all of production metal and VMs to Buster or later
		Declined		None	T244736 Migrate Elasticsearch to Debian Buster
		Open		None	T291916 Tracking task for Bullseye migrations in production
		Resolved		bking	T289135 Upgrade Cirrus Elasticsearch clusters to Debian Bullseye
		Resolved		bking	T308770 Reimage relforge elastic hosts from Stretch to Bullseye

Event Timeline

bking created this task.May 19 2022, 4:09 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 19 2022, 4:09 PM

bking added a parent task: T289135: Upgrade Cirrus Elasticsearch clusters to Debian Bullseye.May 19 2022, 4:09 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-19T19:58:16Z] <inflatador> bking@relforge1004: banned relforge1003 from main and alpha clusters in preparation for reimage T308770

MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board.May 23 2022, 3:34 PM

MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search.

Gehel assigned this task to bking.May 23 2022, 3:34 PM

MPhamWMF set the point value for this task to 2.May 23 2022, 3:41 PM

MPhamWMF moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin1001 for host relforge1003.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin1001 for host relforge1003.eqiad.wmnet with OS bullseye completed:

relforge1003 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205232139_bking_1173818_relforge1003.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by ryankemper@cumin1001 for host relforge1004.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by ryankemper@cumin1001 for host relforge1004.eqiad.wmnet with OS bullseye completed:

relforge1004 (WARN)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202205240106_ryankemper_1208841_relforge1004.out
- Checked BIOS boot parameters are back to normal
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

RKemper moved this task from Ready for Dev -- SWE to Needs Reporting on the Discovery-Search (Current work) board.May 26 2022, 7:10 PM

Gehel closed this task as Resolved.May 30 2022, 9:39 AM