Page MenuHomePhabricator

Update all trove VMs to a modern guest image
Closed, ResolvedPublic

Description

openstack database instance rebuild

should now work for all database types (T355721), which means we can, and should, upgrade trove database VMs to the new guest image. There are several known bugs in the old guest image breaking, among other things, backups for many databases.

This process should be largely automated, but we may need to ssh into the new VMs to repair mountpoints -- for example replacing /dev/vdb with /dev/sdb the guest config in /etc/trove/conf.d.

  • db01 | HEALTHY | huma |
  • mariadb | HEALTHY | wikisp |
  • wp1-db-server | HEALTHY | mwoffliner | Monday sometime
  • thistle | HEALTHY | hoiscript |
  • copypatrol-prod-db-01 | HEALTHY | copypatrol | Noon Monday (17:00 UTC)
  • copypatrol-dev-db-01 | HEALTHY | copypatrol | Noon Monday (17:00 UTC)
  • ifis_dbs | HEALTHY | pm20database |
  • quarry-k8s | HEALTHY | quar |
  • loggerdiscordbot | HEALTHY | discordbots |
  • mwstake | HEALTHY | mwstake |
  • adiutor | HEALTHY | adiutor |
  • hash-associations | HEALTHY | spacemedia |
  • dumps-db2 | HEALTHY | dumps |
  • DBapp | HEALTHY | glamwikidashboard |
  • quarry-dev-kube | ACTIVE | quarry |
  • citefix-db | HEALTHY | citefix |
  • xtools-db01 | HEALTHY | xtools | 10AM Monday (15:00 UTC)
  • linkwatcher-db | HEALTHY | linkwatcher |
  • tools-harbordb | HEALTHY | tools | 9 AM Monday 7-29
  • terraform | HEALTHY | deployment-prep |
  • baglama2 | HEALTHY | baglama2 |
  • wmczdb01 | HEALTHY | wmcz-stats |
  • harbordb | HEALTHY | toolsbeta |
  • maps-test-2 | ERROR | maps |
  • toc-databse | HEALTHY | reading-web-staging |
  • libup-db02 | HEALTHY | library-upgrader |
  • quarry-db-02 | HEALTHY | quarry |
  • osmit-pgsql | HEALTHY | osmit |
  • metricsinfra-db-1 | HEALTHY | metricsinfra |

Event Timeline

Restricted Application removed a subscriber: taavi. · View Herald Transcript

Hi! If possible, I'm requesting a date and approximate time be chosen for when you'd like to do this for XTools. The db isn't big or anything, but it is used for shared session management, so taking it down I think will take down XTools entirely. Which date and time doesn't matter; I'd just like to put up a banner so folks know to expect it. Thank you!

It would be best for mwoffliner (which runs the WP 1.0 Bot) if the maintenance wasn't between 0:00 - 4:00 UTC, because that's when the bot is running

Hi! If possible, I'm requesting a date and approximate time be chosen for when you'd like to do this for XTools. The db isn't big or anything, but it is used for shared session management, so taking it down I think will take down XTools entirely. Which date and time doesn't matter; I'd just like to put up a banner so folks know to expect it. Thank you!

That's easy! I've scheduled your upgrade for Monday the 22nd, at 15:00 UTC. Thanks for checking in.

It would be best for mwoffliner (which runs the WP 1.0 Bot) if the maintenance wasn't between 0:00 - 4:00 UTC, because that's when the bot is running

I will plan to do your upgrade during my day on Monday -- I generally work from 14:00 to 22:00 UTC so that won't run into the danger zone.

Andrew updated the task description. (Show Details)

It would be best for mwoffliner (which runs the WP 1.0 Bot) if the maintenance wasn't between 0:00 - 4:00 UTC, because that's when the bot is running

@Audiodude, I have finished upgrading this db server. Can you please confirm that everything is still working as expected? Thank you!

Mentioned in SAL (#wikimedia-cloud) [2024-07-22T15:00:05Z] <andrewbogott> upgrading db server xtools-db01 -- T369723

Everything is now based off of trove-guest-bobcat-ubuntu-jammy. That image contains a local application of proposed patch https://review.opendev.org/c/openstack/trove/+/924285

I'm optimistic that that will be merged upstream before we need to rev base images again.