[12:31:27] <hasharAway> for when a WMCS admin is around,  deployment-db03.deployment-prep  can not start for some reason  https://horizon.wikimedia.org/project/instances/27460e9d-5548-4cd6-9472-548db6402294/
[12:31:44] <hasharAway> that is the instance holding the database for the beta cluster. 
[12:31:56] <hasharAway> https://phabricator.wikimedia.org/T216067 is slighly related but that is for the database slave ( deployment-db04 )
[12:32:02] <hasharAway> (i am not around today sorry)
[12:32:43] <Krenair> that's odd it was started a few days ago?
[12:33:53] <Krenair> looks like it was up and down like mad on Wednesday
[12:34:00] <Krenair> then failed to start this morning
[12:34:10] <Krenair> didn't realise it was stopped
[12:34:58] <Krenair> Yeah something is very very wrong
[12:35:03] <arturo> deployment-db servers were running in one of the failing cloudvirts
[12:35:33] <Krenair> yeah but I thought they did come back up eventually
[12:35:40] <Krenair> it appears that didn't last long
[12:35:41] <Krenair>  Request ID
[12:35:41] <Krenair>  Action
[12:35:41] <Krenair>  Start Time
[12:35:41] <Krenair>  User ID
[12:35:42] <Krenair>  Message
[12:35:45] <Krenair> req-849202a7-8029-46de-80c0-6ecbd4fb78cd  Start  18 Feb 2019, 11:34 a.m.  krenair  Error
[12:35:47] <Krenair> req-19128734-227b-4987-a782-f57794bf6bf0  Start  18 Feb 2019, 11:27 a.m.  hashar  Error 
[12:36:01] <Krenair> additionally the console log is broken
[12:36:04] <Krenair> can someone repair that?
[12:36:19] <arturo> I would suggest you create new VMs
[12:38:17] <Krenair> please try to start it anyway
[12:44:51] <arturo> Krenair: just issued the command
[12:45:48] <Krenair> power state: no state
[12:46:04] <Krenair> yeah no luck
[12:46:10] <Krenair> any more helpful console output?
[12:47:33] <arturo> I would be happy to debug and fix this under other circumstances. I would say we have 2 options: 1) create new VMs 2) wait a bit to see if today I can have some spare cycles to look into this
[12:49:15] <Krenair> luckily deployment-db04 is still up so we can probably retrieve the data from there
[12:49:54] <Krenair> I'm going to try doing the copy from there to db05 now
[12:51:07] <Krenair> I have to say though, we do need to figure out how db03 and db04 came to be on the same host so we can prevent that happening again
[12:51:44] <arturo> Krenair: right now the nova scheduler doesn't know about the roles of a given server
[12:52:12] <arturo> we have a script https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance#wmcs-spreadcheck that can be used to report on this
[12:52:15] <Krenair> sure but I expect they were originally created on different hosts, then at some point someone moved them?
[12:53:53] <arturo> I doubt we would be checking collocation if we are rushing to manually drain a cloudvirt (to prevent further data loss)
[12:58:10] <Krenair> maybe not immediately in an emergency but some time after
[13:00:36] <arturo> fair enough, but in this case, we chained several incidents since the original issue in that cloudvirt
[13:01:12] <arturo> we had very little time for cleanups, if any
[13:01:28] <Krenair> in this case I wouldn't've expected it yet