Page MenuHomePhabricator
Create Task
Maniphest T323925

codfw: ManagementSSHDown for ores2009 and thumbor2004
Closed, ResolvedPublic
Actions

Description

We are getting emails from sre-obervability that the mgmt interface of those server is down. Will it be possible to depool those servers if in production so we can take a look?
Thanks.

Event Timeline

Papaul created this task.Nov 28 2022, 3:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 28 2022, 3:50 PM
Papaul triaged this task as High priority.Nov 28 2022, 3:51 PM
Maintenance_bot added a project: SRE.Nov 28 2022, 4:29 PM
ops-monitoring-bot added a comment.Nov 30 2022, 3:54 PM

Icinga downtime and Alertmanager silence (ID=8b8e8a4d-71f2-462d-8e1f-ff904f7e3ed4) set by akosiaris@cumin1001 for 1:00:00 on 1 host(s) and their services with reason: work on iDrac

thumbor2004.codfw.wmnet
klausman subscribed.Nov 30 2022, 3:58 PM

ores2009 is shutting down & powering off now

Papaul added a comment.Nov 30 2022, 5:28 PM

thunbor2004 had a broken IDRAC card. I replaced it.

Papaul closed this task as Resolved.Nov 30 2022, 6:40 PM

ores2009 mgmt is back up

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL