Page MenuHomePhabricator

Instance in SHUTOFF state, not rebooting
Closed, ResolvedPublic

Description

Instance i-00000184.eqiad.wmflabs (sylvester) was up yesterday. Today it shows SHUTOFF state in Special:NovaInstance. Tried clicking "reboot" and waited a few minutes, no change. Clicking reboot again now shows a little popup message "Failed to reboot instance sylvester." Clicking "get console output" shows message "Failed to get console output for instance sylvester." Tried rebooting several times in the last hour.

No idea why this would happen. Reading something about NFS issues, I thought maybe it has something to do with disk images for the VM being inaccessible or something like that, but that's just a wild guess.

We're currently integrating catgraph with cirrus. This is a blocker for us because catgraph doesn't work without this instance. Can somebody with admin access please investigate?

Event Timeline

jkroll raised the priority of this task from to High.
jkroll updated the task description. (Show Details)
jkroll added a subscriber: jkroll.

This is almost certainly my fault, I'm trying to fix right now.

ok... is that better?

The (boring) explanation is that this instance is quite old and requires a backing image that I didn't realize was still in use. I migrated it to the new cluster w/out it's backing image so it was unable to restart. The backing image is present now, and I've updated the migration script to prevent this problem in the future.

Sorry!

-A

Great, instance is up and running now. Thanks a lot Andrew!

Andrew claimed this task.