Migrate hue.wikimedia.org to bullseye
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	BTullis
	Oct 20 2023, 4:40 PM

Description

We currently run hue.wikimedia.org on a single virtual machine.

an-tool1008.eqiad.wmnet

This machine needs to be upgraded to Debian bullseye

We have previously upgraded an-test-ui1001.eqiad.wmnet to bullseye. This machine also runs Hue, but it might be useful to verify that this works before upgrading the production instance.

Details

	Subject	Repo	Branch	Lines +/-
	Build hue for Debian Bullseye by default	operations/debs/hue	master	+9 -3
	hue: rename python-snappy apt dependency	operations/puppet	production	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T291916 Tracking task for Bullseye migrations in production
Open	None	T288804 Upgrade the Data Engineering infrastructure to Debian Bullseye
Declined	None	T349400 Migrate hue.wikimedia.org to bullseye

Event Timeline

BTullis created this task.Oct 20 2023, 4:40 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2023, 4:40 PM

BTullis added a parent task: T288804: Upgrade the Data Engineering infrastructure to Debian Bullseye.Oct 20 2023, 4:58 PM

Gehel moved this task from Incoming to Ready for Work on the Data-Platform-SRE board.Nov 3 2023, 10:34 AM

BTullis triaged this task as High priority.Nov 15 2023, 9:45 AM

Gehel moved this task from Ready for Work to OS Upgrade on the Data-Platform-SRE board.Dec 6 2023, 1:10 PM

BTullis edited projects, added Data-Platform-SRE (2024.01.01 - 2024.01.21); removed Data-Platform-SRE.Jan 9 2024, 3:29 PM

BTullis edited projects, added Data-Platform-SRE (2024.01.22 - 2024.02.11); removed Data-Platform-SRE (2024.01.01 - 2024.01.21).Jan 22 2024, 2:45 PM

brouberol claimed this task.Jan 29 2024, 9:42 AM

brouberol moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.01.22 - 2024.02.11) board.Jan 29 2024, 9:46 AM

Mentioned in SAL (#wikimedia-analytics) [2024-01-29T13:06:31Z] <brouberol> I'm starting the reimaging process of an-tool1009.eqiad.wmnet, which will cause unavalability of hue.wikimedia.org while it runs - T349400

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1002 for host an-tool1009.eqiad.wmnet with OS bullseye

Change 993692 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] hue: rename python-snappy apt dependency

https://gerrit.wikimedia.org/r/993692

gerritbot added a project: Patch-For-Review.Jan 29 2024, 2:09 PM

Change 993692 merged by Brouberol:

[operations/puppet@production] hue: rename python-snappy apt dependency

https://gerrit.wikimedia.org/r/993692

Maintenance_bot removed a project: Patch-For-Review.Jan 29 2024, 2:31 PM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1002 for host an-tool1009.eqiad.wmnet with OS bullseye executed with errors:

an-tool1009 (FAIL)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401291323_brouberol_759805_an-tool1009.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- The reimage failed, see the cookbook logs for the details

We found out that the hue package hadn't been built for bullseye. We're going to revert an-tool1009 to Buster until we can build a hue package for bullseye.

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1002 for host an-tool1009.eqiad.wmnet with OS buster

Change 993708 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/debs/hue@master] Build hue for Debian Bullseye by default

https://gerrit.wikimedia.org/r/993708

gerritbot added a project: Patch-For-Review.Jan 29 2024, 3:26 PM

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1002 for host an-tool1009.eqiad.wmnet with OS buster completed:

an-tool1009 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401291534_brouberol_780742_an-tool1009.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by brouberol@cumin1002 for host an-tool1008.eqiad.wmnet with OS bullseye

Cookbook cookbooks.sre.hosts.reimage started by brouberol@cumin1002 for host an-tool1008.eqiad.wmnet with OS bullseye completed:

an-tool1008 (PASS)
- Downtimed on Icinga/Alertmanager
- Disabled Puppet
- Removed from Puppet and PuppetDB if present and deleted any certificates
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via gnt-instance
- Host up (Debian installer)
- Add puppet_version metadata to Debian installer
- Set boot media to disk
- Host up (new fresh bullseye OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga/Alertmanager
- Removed previous downtime on Alertmanager (old OS)
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202401300914_brouberol_935671_an-tool1008.out
- configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

brouberol moved this task from In Progress to Blocked / Waiting on the Data-Platform-SRE (2024.01.22 - 2024.02.11) board.Jan 30 2024, 9:30 AM

Hue has proven to be quite tricky to build for bullseye. @MoritzMuehlenhoff suggested that as https://phabricator.wikimedia.org/T340144 could deprecate the need for Hue altogether, there might be a case where Hue is simpler to deprecate than OS-upgrade. We'll revisit once we know more.