Page MenuHomePhabricator
Create Task
Maniphest T198792

snapshot1005 does not power back up
Closed, ResolvedPublic
Actions

Description

Saw an icinga alert for snapshot1005 being done.
From the mgmt console, power was off. issuing power on command claimed to power it on, yet power status remained powered off, even after mutiple attempts.

Tried checking the logs: powersupply1 land powersupply2 both look ok, only thing in /system1/log1 records earlier than my typo of some command or other was from two months ago. Nothing useful in /map1/log1 records either, just logins and attempts to power on the dang thing.

Does this need a manual unplug/replug? Is there something I missed?

Details

SubjectRepoBranchLines +/-
switch en wiki dumps to run on snapshot1009 for nowoperations/puppetproduction+9 -3
Customize query in gerrit

Event Timeline

ArielGlenn created this task.Jul 4 2018, 12:57 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2018, 12:57 PM
ArielGlenn triaged this task as High priority.Jul 4 2018, 1:01 PM
ArielGlenn added projects: SRE, ops-eqiad, DC-Ops.
ArielGlenn updated the task description. (Show Details)

If we can't get more info about this before tomorrow morning (UTC morning), I'll move the services on this box to snapshot1009 then; if it turns out it just needs the right kind of kick, I won't bother to do that of course.

gerritbot subscribed.Jul 4 2018, 1:11 PM

Change 443828 had a related patch set uploaded (by ArielGlenn; owner: ArielGlenn):
[operations/puppet@production] switch en wiki dumps to run on snapshot1009 for now

https://gerrit.wikimedia.org/r/443828

gerritbot added a project: Patch-For-Review.Jul 4 2018, 1:11 PM
ArielGlenn added a project: Dumps-Generation.Jul 4 2018, 1:12 PM
ArielGlenn moved this task from Backlog to Active on the Dumps-Generation board.
ArielGlenn added a comment.Jul 4 2018, 1:17 PM

Because our dc folks are off today, I will likely go ahead and make this switch this evening in time for the next cron cycle to pick up the jobs.

gerritbot added a comment.Jul 4 2018, 1:37 PM

Change 443828 merged by ArielGlenn:
[operations/puppet@production] switch en wiki dumps to run on snapshot1009 for now

https://gerrit.wikimedia.org/r/443828

ArielGlenn added a comment.Jul 4 2018, 1:44 PM

Changed my mind and merged it through. I'll let the en wp dumps run to completion over here this month, regardless.

• Cmjohnson subscribed.Jul 10 2018, 2:21 PM

I attempted to power off, unplug and power the server back on, unfortunately it does not want to power on...i just get a flashing green led on the front. I am able to access the mgmt web portal and pulled the AHS log a requirement by HP tech support. Worth noting that this is a lease server and it expires in February 2019. We should still fix the server but is this a good opportunity to move the service?

ArielGlenn added a comment.Jul 10 2018, 2:23 PM

The service was moved to another host already (which had been the spare) but we do still want a spare. February is not that far away though, all things considered. I guess snapshot1006,7 will have the same expiry dates.

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Jul 10 2018, 2:26 PM
• Cmjohnson closed this task as Resolved.Jul 17 2018, 6:24 PM
• Cmjohnson claimed this task.

The system board has been replaced and the server is accessible now

ArielGlenn moved this task from Active to Done on the Dumps-Generation board.Jul 20 2018, 5:37 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL