Page MenuHomePhabricator
Create Task
Maniphest T124642

mw1172, mw1178,mw1217, mw1257 are unresponsive, mgmt interface unreachable
Closed, ResolvedPublic
Actions

Description

After attempting a reboot, the mgmt interface is unreachable and these machines are down. Please investigate. List of hosts:

mw1172
mw1178
mw1217
mw1257

Details

SubjectRepoBranchLines +/-
conftool: re-pool recovered serversoperations/puppetproduction+4 -0
scap: re-add servers to mediawiki-installationoperations/puppetproduction+4 -4
Fixing bug in wrong server being excluded from dsh (mw1257, not mw1157)operations/puppetproduction+2 -2
Exclude mw1217, mw1178 and mw1257 from SCAP until they are brought upoperations/puppetproduction+3 -3
scap: exclude mw1172 from scap until it's properly set upoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

Joe created this task.Jan 25 2016, 10:32 AM
Joe raised the priority of this task from to Needs Triage.
Joe updated the task description. (Show Details)
Joe added projects: ops-eqiad, SRE.
Joe subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptJan 25 2016, 10:32 AM
gerritbot subscribed.Jan 25 2016, 10:36 AM

Change 266210 had a related patch set uploaded (by Giuseppe Lavagetto):
scap: exclude mw1172 from scap until it's properly set up

https://gerrit.wikimedia.org/r/266210

gerritbot added a project: Patch-For-Review.Jan 25 2016, 10:36 AM

Change 266210 merged by Giuseppe Lavagetto:
scap: exclude mw1172 from scap until it's properly set up

https://gerrit.wikimedia.org/r/266210

akosiaris triaged this task as Medium priority.Jan 25 2016, 11:50 AM
akosiaris subscribed.
jcrespo renamed this task from mw1172 is unresponsive, mgmt interface unreachable to mw1172, mw1217, mw1178, mw1257 are unresponsive, mgmt interface unreachable.Jan 25 2016, 3:32 PM
jcrespo updated the task description. (Show Details)
jcrespo set Security to None.
gerritbot added a comment.Jan 25 2016, 3:38 PM

Change 266238 had a related patch set uploaded (by Jcrespo):
Exclude mw1217, mw1178 and mw1257 from SCAP until they are brought up

https://gerrit.wikimedia.org/r/266238

gerritbot added a comment.Jan 25 2016, 3:55 PM

Change 266238 merged by Jcrespo:
Exclude mw1217, mw1178 and mw1257 from SCAP until they are brought up

https://gerrit.wikimedia.org/r/266238

gerritbot added a comment.Jan 25 2016, 4:30 PM

Change 266250 had a related patch set uploaded (by Jcrespo):
Fixing bug in wrong server being excluded from dsh (mw1257, not mw1157)

https://gerrit.wikimedia.org/r/266250

gerritbot added a comment.Jan 25 2016, 4:33 PM

Change 266250 merged by Jcrespo:
Fixing bug in wrong server being excluded from dsh (mw1257, not mw1157)

https://gerrit.wikimedia.org/r/266250

Joe renamed this task from mw1172, mw1217, mw1178, mw1257 are unresponsive, mgmt interface unreachable to mw1172, mw1178,mw1217, mw1228, mw1257 are unresponsive, mgmt interface unreachable.Jan 26 2016, 8:22 AM
Joe renamed this task from mw1172, mw1178,mw1217, mw1228, mw1257 are unresponsive, mgmt interface unreachable to mw1172, mw1178,mw1217, mw1257 are unresponsive, mgmt interface unreachable.
Joe updated the task description. (Show Details)
Joe updated the task description. (Show Details)
• Cmjohnson assigned this task to Joe.Jan 28 2016, 5:24 PM
• Cmjohnson subscribed.

Fixed all the servers and verified mgmt interface is reachable. assigning back @Joe

gerritbot added a comment.Jan 29 2016, 2:36 PM

Change 267252 had a related patch set uploaded (by Giuseppe Lavagetto):
scap: re-add servers to mediawiki-installation

https://gerrit.wikimedia.org/r/267252

gerritbot added a comment.Jan 29 2016, 2:37 PM

Change 267252 merged by Giuseppe Lavagetto:
scap: re-add servers to mediawiki-installation

https://gerrit.wikimedia.org/r/267252

gerritbot added a comment.Jan 29 2016, 2:52 PM

Change 267256 had a related patch set uploaded (by Giuseppe Lavagetto):
conftool: re-pool recovered servers

https://gerrit.wikimedia.org/r/267256

gerritbot added a comment.Jan 29 2016, 2:56 PM

Change 267256 merged by Giuseppe Lavagetto:
conftool: re-pool recovered servers

https://gerrit.wikimedia.org/r/267256

Joe closed this task as Resolved.Jan 29 2016, 3:23 PM
• MoritzMuehlenhoff mentioned this in T146263: mw1172 stuck after reboot.Sep 21 2016, 8:20 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits