Page MenuHomePhabricator

Recover wm-bot-pg2 data if possible
Closed, ResolvedPublic

Description

I was trying to perform a buster -> bookworm upgrade on instance wm-bot-pg2 within the wm-bot cloud VPS project. An error occurred while upgrading, which resulted in me losing sudo access, and I couldn't get back in after logging out. @taavi was able to get in via console, and then rebooted, and now even they can't get in. Unfortunately to my knowledge we don't have a back of the postgresql data. If the data can be recovered, I would appreciate it.

Text logs still exist, and so not the end of the world, but not being able to recover this would mean that old logs aren't accessible via the log browser.

I think the lesson I learned is that I need to sudo su first before performing upgrades.

I was told to ping @Andrew about this, so here we go.

I am very sorry for causing this mess.

Related Objects

StatusSubtypeAssignedTask
ResolvedMacFan4000
ResolvedAndrew

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Andrew claimed this task.

@MacFan4000 You lucked out and there was a backup that predated the VM getting scrambled; I've restored it to the state it was in on June 20th. I can't make any promises about how postgres will or won't feel about the time traveling.

A few notes for future reference:

  1. Don't do in-place OS upgrades of your VMs. It often leads to disaster (as seen today) and in any case results in hosts that are in an inconsistent/mysterious state and difficult to maintain. Among other things, a VM created with a Buster base image will always appear forever after in any report we run about Buster VMs, which means it's likely to get arbitrarily deleted when we deprecate the old OS version.
  1. When you replace this VM with a new one, you'll want to move your data onto a cinder volume; that means the data will be preserved and recoverable regardless of what happens to the OS of the host.
  1. Trove (or dbaas offering) has partial Postgres support, so you might also want to experiment with that and see if it's a good alternative.

Good luck!

-A