Page MenuHomePhabricator
Create Task
Maniphest T223518

ms-be1033 not powering up
Closed, ResolvedPublic
Actions

Description

Yesterday during T223126: Install new PDUs into b5-eqiad ms-be1033 powered off at the beginning of the window (since 2019-05-16 12:54:56 according to icinga) and after work was completed it couldn't be powered back on by @Cmjohnson . Filing a task for tracking on further diagnosis and next steps.

Details

SubjectRepoBranchLines +/-
eqiad-prod: start depool ms-be1033operations/software/swift-ringmaster+1 K -1 K
Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.May 17 2019, 8:30 AM
Restricted Application added a project: SRE. · View Herald TranscriptMay 17 2019, 8:30 AM
fgiunchedi added a project: User-fgiunchedi.May 20 2019, 2:43 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.May 21 2019, 9:59 AM
fgiunchedi added a comment.May 21 2019, 10:02 AM

Since the host is not coming back for another week for sure I'm going to de-weight in swift

gerritbot added a comment.May 21 2019, 10:09 AM

Change 511670 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/software/swift-ring@master] eqiad-prod: start depool ms-be1033

https://gerrit.wikimedia.org/r/511670

gerritbot added a project: Patch-For-Review.May 21 2019, 10:09 AM
Stashbot added a comment.May 21 2019, 1:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-21T13:25:44Z] <godog> swift eqiad-prod: start depool ms-be1033 - T223518

gerritbot added a comment.May 21 2019, 1:26 PM

Change 511670 merged by Filippo Giunchedi:
[operations/software/swift-ring@master] eqiad-prod: start depool ms-be1033

https://gerrit.wikimedia.org/r/511670

Ladsgroup removed a project: Patch-For-Review.May 21 2019, 10:06 PM
Stashbot added a comment.May 23 2019, 1:00 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-23T13:00:26Z] <godog> swift eqiad-prod: ms-be1033 weight to 1500 - T223518

Stashbot added a comment.May 27 2019, 1:02 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-27T13:02:58Z] <godog> swift eqiad-prod: ms-be1033 weight to 0 - T223518

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.May 28 2019, 2:53 PM
• Cmjohnson added a comment.May 29 2019, 7:00 PM

Steps i have taken

  • I took the server down to the bare minimum operating condition 1CPU and 1DIMM and the server will still not boot. I created a support ticket with HP.

5338974069

jijiki subscribed.May 30 2019, 12:08 PM

I downtimed the host on icinga for another week

• Cmjohnson added a comment.Jun 6 2019, 2:54 PM

The HP technician will be her June 7 @1000 Ashburn time.

• Cmjohnson closed this task as Resolved.Jun 7 2019, 4:21 PM

The motherboard was replaced and the server is back up

Stashbot added a comment.Jun 8 2019, 11:58 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-08T11:58:20Z] <godog> stop swift processes on ms-be1033 - T223518

Stashbot added a comment.Jun 11 2019, 10:54 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-11T10:54:13Z] <godog> wipe fs on ms-be1033 data partitions - T223518

fgiunchedi reopened this task as Open.Jun 11 2019, 12:53 PM
fgiunchedi claimed this task.
Stashbot added a comment.Jun 11 2019, 12:54 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-11T12:54:13Z] <godog> swift eqiad-prod: put back ms-be1033 - T223518

• Cmjohnson moved this task from Hardware Failure / Troubleshoot to Blocked on the ops-eqiad board.Jun 11 2019, 4:16 PM
Stashbot added a comment.Jun 12 2019, 11:55 AM

Mentioned in SAL (#wikimedia-operations) [2019-06-12T11:55:43Z] <godog> swift eqiad-prod: put back ms-be1033 - T223518

Stashbot added a comment.Jun 25 2019, 12:48 PM

Mentioned in SAL (#wikimedia-operations) [2019-06-25T12:48:40Z] <godog> swift eqiad-prod: put back ms-be1033 - T223518

Stashbot added a comment.Jul 1 2019, 9:54 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-01T09:54:28Z] <godog> swift eqiad-prod eqiad-prod: put back ms-be1033 - T223518

fgiunchedi closed this task as Resolved.Jul 1 2019, 10:02 AM

The last rebalance is underway now to put ms-be1033 fully back in service. Resolving.

fgiunchedi mentioned this in T226937: Not possible to server-side upload certain images: "An unknown error occurred in storage backend "local-swift-eqiad"".Jul 2 2019, 12:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL