Page MenuHomePhabricator

Recover ign.ign2commons.eqiad.wmflabs
Closed, DeclinedPublic

Description

Apparently, the instance ign.ign2commons.eqiad.wmflabs was deleted, shut down or moved somewhere in February. However, that instance has not yet done its job (T203010) and it contained valuable data and scripts. Is it possible to recover the instance or its data?

Event Timeline

abian created this task.Mar 9 2019, 2:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 9 2019, 2:31 PM
Krenair added a subscriber: Krenair.EditedMar 9 2019, 3:27 PM

The instance is listed as SHUTOFF. It was imaged Stretch and is in the new region, so I doubt it was turned off for legacy deprecation reasons. Last modified date of 2019-02-13 suggests it was part of the hardware failure.

Here it is, this instance was in the list on cloud-announce, so not sure how you weren't already aware: https://lists.wikimedia.org/pipermail/cloud-announce/2019-February/000128.html

abian added a comment.Mar 9 2019, 5:44 PM

Thanks, @Krenair. I was aware of the hardware failure but I didn't remember this instance was affected, my fault. Have these particular data been recovered?

Krenair added a subscriber: Andrew.Mar 9 2019, 6:23 PM
@Andrew on cloud-announce

Nearly every instance that was on cloudvirt1018 has suffered some degree
of disk corruption. For the most part I've repaired them enough to
allow logins, but I recommend that you check them extensively before
relying on data integrity there. In some cases you may find part or all
of your misplaced files in /lost+found.

I would suggest you try to start the instance and see what survived?

abian added a comment.Mar 9 2019, 6:38 PM

I don't know how to do it. I access to https://horizon.wikimedia.org/project/instances/ but no instances are listed.

Hm. Are you sure your project is set to ign2commons and your region is set to eqiad1-r?

abian added a comment.Mar 9 2019, 6:53 PM

Oops, no, it was just eqilad. But it seems I can't start the instance anyway.

Ah, that's very broken and will need ops intervention to fix. It reminds me of one that was seen on T216404 among other places. Hopefully someone will be able to take a look on Monday?

Andrew added a comment.Mar 9 2019, 9:28 PM

I'm sorry @abian, that VM does not appear to have survived the hardware failure. Its disk image is entirely missing so there's nothing to salvage.