Upgrade db1108 to Bullseye
Closed, DeclinedPublic1 Estimated Story Points
Actions

Assigned To

Authored By

	Marostegui
	Mar 23 2022, 10:28 AM

Description

db1108, the analytics misc host needs to be upgraded to Bullseye.
This can be done anytime as the mariadb version isn't changing (10.4)
Reminder: shutdown mariadb before going for the reimage

Related Objects
Search...

Status	Assigned	Task
		Unknown Object (Task)
Resolved	Papaul	T326661 Q3:rack/setup/install db1207-db1225
Resolved	Marostegui	T326669 Productionize db1206-db1225
Open	None	T291916 Tracking task for Bullseye migrations in production
Resolved	Marostegui	T298585 Upgrade WMF database-and-backup-related hosts to bullseye
		Restricted Task
		Restricted Task
		Restricted Task
		Restricted Task
Declined	BTullis	T304492 Upgrade db1108 to Bullseye

Event Timeline

Marostegui created this task.Mar 23 2022, 10:28 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 23 2022, 10:28 AM

• EChetty moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Mar 24 2022, 2:53 PM

• EChetty moved this task from Event Platform Backlog to Ops Week on the Data-Engineering board.Mar 24 2022, 5:36 PM

Hi, only 4 dbs left on buster, this is one of them

JArguello-WMF removed a project: Analytics.Jun 6 2022, 3:54 PM

Marostegui mentioned this in T298585: Upgrade WMF database-and-backup-related hosts to bullseye.Jun 30 2022, 7:55 AM

RhinosF1 subscribed.Jun 30 2022, 8:47 AM

BTullis claimed this task.Jul 14 2022, 9:13 AM

BTullis edited projects, added Data-Engineering-Planning; removed Data-Engineering.

I'll carry out this upgrade, although it might be in a few weeks' time. Hope that's OK.

• EChetty moved this task from Backlog to To be Discussed on the Data-Engineering-Planning board.Jul 25 2022, 10:01 AM

• EChetty moved this task from To be Discussed to Shared Data Infra on the Data-Engineering-Planning board.Aug 15 2022, 8:39 AM

• EChetty added a project: Shared-Data-Infrastructure.Aug 15 2022, 8:50 AM

• EChetty moved this task from Backlog to To be discussed on the Shared-Data-Infrastructure board.Aug 15 2022, 9:06 AM

• EChetty moved this task from To be discussed to Estimated/Discussed on the Shared-Data-Infrastructure board.Aug 16 2022, 1:47 PM

• EChetty set the point value for this task to 1.Sep 6 2022, 1:24 PM

• EChetty moved this task from Estimated/Discussed to Sprint 01 on the Shared-Data-Infrastructure board.

• EChetty edited projects, added Shared-Data-Infrastructure (Sprint 01); removed Shared-Data-Infrastructure.

Marostegui added a parent task: Restricted Task.Sep 6 2022, 1:55 PM

• EChetty moved this task from Sprint 01 to Sprint 02 on the Shared-Data-Infrastructure board.Sep 27 2022, 12:50 PM

• EChetty edited projects, added Shared-Data-Infrastructure (Sprint 02); removed Shared-Data-Infrastructure (Sprint 01).

• EChetty moved this task from Sprint 02 to Sprint 03 on the Shared-Data-Infrastructure board.Oct 18 2022, 1:27 PM

• EChetty edited projects, added Shared-Data-Infrastructure (Sprint 03); removed Shared-Data-Infrastructure (Sprint 02).

• EChetty moved this task from Sprint 03 to Estimated/Discussed on the Shared-Data-Infrastructure board.Oct 18 2022, 1:40 PM

• EChetty edited projects, added Shared-Data-Infrastructure; removed Shared-Data-Infrastructure (Sprint 03).

Marostegui added a parent task: Restricted Task.Oct 24 2022, 6:52 AM

Any ETA on when this host will be upgraded to Bullseye?

TheresNoTime removed a subscriber: RhinosF1.Dec 15 2022, 11:35 PM

@BTullis any ETA?

Hey @Marostegui,

Is this work blocking anything at the moment? It currently on our radar, but not prioritised. Please let me know if its a blocker or urgent and we can make sure it gets prioritised accordingly?

It is not blocking anything specifically, but this host needs to _at least_ be rebooted soon as there are several kernel upgrades it has missed as it has been up for 650 days cc @MoritzMuehlenhoff

Host rebooted by btullis@cumin1001 with reason: picking up new kernel

I rebooted db1108, but the systemd service definitions for the analytics_meta and matomo database instances no longer exist for some reason. Checking the cause now.

In T304492#8589207, @BTullis wrote:

I rebooted db1108, but the systemd service definitions for the analytics_meta and matomo database instances no longer exist for some reason. Checking the cause now.

Oops. This was an error on my part. I was looking in the wrong place for the systemd units. Thanks for correcting me @Marostegui

OK, so that's a [well overdue] reboot completed. As @EChetty said we've struggled to find the time to reimage this host, but will try to do so soon.
We have a lot of other buster->bullseye reimaging to do in the next few months, so hopefully we will be able to fit this in as well.

• EChetty moved this task from Estimated/Discussed to 2022-23 Q4 Wrap up on the Shared-Data-Infrastructure board.Feb 6 2023, 12:55 PM

• EChetty edited projects, added Shared-Data-Infrastructure (2022-23 Q4 Wrap up); removed Shared-Data-Infrastructure.

I have also upgraded mariadb from 10.4.18 to 10.4.22

JArguello-WMF triaged this task as Medium priority.Mar 14 2023, 6:01 PM

I would prefer if this is given a bit of a higher priority. This task has been here for a year and we are no longer providing updates to the buster Mariadb package. Only bullseye ones.

Understood. Thanks @Marostegui - Elevated the priority as requested. This is a three week sprint for our team so I'm hoping to do it very soon.

Thanks - I'm happy to help with the process. Basically it's just to double check the host is in the right partman recipe so /srv doesn't get wiped.
Then stop mysql, reimage, and start mysql :)

db1108 will need to be replaced by the new hardware (T326669), which will be installed on bullseye, so probably we can just migrate the data to that new host.

Marostegui added a parent task: T326669: Productionize db1206-db1225.Mar 15 2023, 10:04 AM

BTullis mentioned this in T332128: Recreate replica of matomo database on db1108.Mar 15 2023, 10:44 AM

JArguello-WMF moved this task from Next Up to In Progress on the Shared-Data-Infrastructure (2022-23 Q4 Wrap up) board.Mar 20 2023, 1:39 PM