My bet is that power supply or something else is fried- although integrated monitoring does not show anything strange.
https://racktables.wikimedia.org/index.php?page=object&tab=default&object_id=1766
This server was being reimaged- so it can be powerup/down at any time.
I am going to pool db1067 as a replacement- we cannot be without this server for long.
Mentioned in SAL (#wikimedia-operations) [2017-03-15T19:46:39Z] <jynus> shutting down db1067 for maintenance (as a db1057 replacement) T160435
Change 342905 had a related patch set uploaded (by Jcrespo):
[operations/puppet] Move db1067 from s2 to s1 as a db1057 replacement
Change 342906 had a related patch set uploaded (by Jcrespo):
[operations/mediawiki-config] Move db1067 from s2 to s1 as a db1057 replacement
Change 342905 merged by Jcrespo:
[operations/puppet] Move db1067 from s2 to s1 as a db1057 replacement
Change 342906 merged by jenkins-bot:
[operations/mediawiki-config] Move db1067 from s2 to s1 as a db1057 replacement
Change 342990 had a related patch set uploaded (by Marostegui):
[operations/software] s1,s2.hosts: Move db1067 from s2 to s1
Change 342990 merged by jenkins-bot:
[operations/software] s1,s2.hosts: Move db1067 from s2 to s1
Mentioned in SAL (#wikimedia-operations) [2017-03-16T13:15:53Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Depool db1026 - T160415, Repool db1067 - T160435 (duration: 00m 42s)
@Cmjohnson this is still blocked on you- db1057 does not start. Does it need a new PDU?
@jcrespo: I pulled the 1 pdu out that was working and the server appears to turn on but I am not getting any display or able to establish a connection. I am able to login to the idrac and see a cpu2 error and the power supply. The server is well out of warranty and should be replaced.
@Cmjohnson I might be asking something silly, but I thought I would ask just in case.
Is it doable to take some spare pieces from some other server that have been decommissioned and replace those faulty ones in db1057 to see if we can bring it back to life?
I am not sure which one you think we can pull from? All the db's that are
being decom'd are different server types and older...They're R510's from
replacement but since it's out of warranty we are not able to do this.
Ah right. I wasn't sure about it, hence my question.
It is clear now that we lost that server for good I guess :_(?
We may be able to salvage data.....I can try moving the raid card and disks to a decom R510
We thankfully saved the data before reimaging/rebooting it, it is more about the server itself (and the possibility of using it somewhere else) than the data it contained.
I would say we have to assume we have lost this chassis then :(
I will mark this as resolved, as there is not much else we can do.
Thanks for your help Chris
Change 345535 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Remove db1057 from s1
Change 345545 had a related patch set uploaded (by Marostegui):
[operations/puppet@production] site.pp: Remove db1057
Change 345535 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Remove db1057 from s1
Mentioned in SAL (#wikimedia-operations) [2017-03-30T14:50:58Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1057 entry from s1 shard - T160435 (duration: 00m 44s)
Change 345856 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Remove db1057
Change 345856 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw,db-eqiad.php: Remove db1057
Mentioned in SAL (#wikimedia-operations) [2017-04-03T06:21:32Z] <marostegui@tin> Synchronized wmf-config/db-codfw.php: Remove db1057 entry - T160435 (duration: 00m 54s)
Mentioned in SAL (#wikimedia-operations) [2017-04-03T06:25:01Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Remove db1057 entry - T160435 (duration: 00m 44s)
Change 345545 merged by Marostegui:
[operations/puppet@production] site.pp,linux-host-entries.ttyS1: Remove db1057