Page MenuHomePhabricator

es1019 mgmt interface DOWN
Closed, ResolvedPublic

Description

es1019 has reported its mgmt iface down.

˜/icinga-wm 7:05> PROBLEM - Host es1019.mgmt is DOWN: PING CRITICAL - Packet loss = 100%

And it looks down indeed:

root@neodymium:~# ping es1019.mgmt.eqiad.wmnet
PING es1019.mgmt.eqiad.wmnet (10.65.4.44) 56(84) bytes of data.

If you need to reboot the host, let us know so we can depool it first.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptAug 3 2018, 5:15 AM
Marostegui triaged this task as Medium priority.Aug 3 2018, 5:15 AM
Marostegui moved this task from Triage to Next on the DBA board.Aug 3 2018, 7:37 AM
Cmjohnson moved this task from Backlog to Up next on the ops-eqiad board.Aug 3 2018, 2:44 PM
jcrespo moved this task from Next to In progress on the DBA board.Aug 7 2018, 7:32 AM

Change 450912 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool es1019 for hardware maintenance

https://gerrit.wikimedia.org/r/450912

Change 450912 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool es1019 for hardware maintenance

https://gerrit.wikimedia.org/r/450912

Mentioned in SAL (#wikimedia-operations) [2018-08-07T10:19:14Z] <jynus> shutting down es1019 for hw maintenance T201132

@Cmjohnson es1019 is fully depooled, alerts disabled and shutdown, please proceed directly with any task you need to do it, ping us here when finished.

Change 450990 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool es1019 and db1102 with low load after maintenance

https://gerrit.wikimedia.org/r/450990

Change 450990 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool es1019 and db1102 with low load after maintenance

https://gerrit.wikimedia.org/r/450990

Change 450994 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1081 with low load after maintenance

https://gerrit.wikimedia.org/r/450994

Change 450994 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db1081 with low load after maintenance

https://gerrit.wikimedia.org/r/450994

jcrespo closed this task as Resolved.Aug 7 2018, 2:46 PM

Everything looking ok now.