Page MenuHomePhabricator
Create Task
Maniphest T233273

labsdb1009 broken PSU
Closed, ResolvedPublic
Actions

Description

As spoken in IRC, probably caused by the a1-pdu issues (T233248), labsdb1009 has lost its PSU redundancy.

</system1/log1>hpiLO-> show record27

status=0
status_tag=COMMAND COMPLETED
Thu Sep 19 04:45:31 2019



/system1/log1/record27
  Targets
  Properties
    number=27
    severity=Caution
    date=09/18/2019
    time=18:02
    description=System Power Supply: Input Power Loss or Unplugged Power Cord, Verify Power Supply Input (Power Supply 1)
  Verbs
    cd version exit show


</system1/log1>hpiLO-> show record28

status=0
status_tag=COMMAND COMPLETED
Thu Sep 19 04:45:34 2019



/system1/log1/record28
  Targets
  Properties
    number=28
    severity=Caution
    date=09/18/2019
    time=18:02
    description=System Power Supplies Not Redundant

Details

SubjectRepoBranchLines +/-
dbproxy1011: Depool labsdb1009operations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Sep 19 2019, 4:49 AM
Restricted Application added a project: SRE. · View Herald TranscriptSep 19 2019, 4:49 AM
• Marostegui added a parent task: T233248: Power issue in eqiad A1.Sep 19 2019, 4:50 AM
• Marostegui moved this task from Triage to In progress on the DBA board.
wiki_willy assigned this task to Jclark-ctr.Sep 19 2019, 6:22 AM
wiki_willy added a subtask: Unknown Object (Task).Sep 19 2019, 6:29 AM
• Cmjohnson added a subscriber: RobH.Sep 19 2019, 12:08 PM

This server is out of warranty and @RobH has created a procurement task.

• Cmjohnson moved this task from Backlog to Blocked on the ops-eqiad board.Sep 19 2019, 12:08 PM
• Marostegui added a comment.Sep 19 2019, 12:09 PM
In T233273#5505877, @Cmjohnson wrote:

This server is out of warranty and @RobH has created a procurement task.

Indeed Chris - thanks! I caught up with Willy earlier today and he pointed me to the procurement task
Will leave this task open to handle the tech side of things related to this server
Thank you again!

• Marostegui triaged this task as Medium priority.Sep 19 2019, 12:09 PM
• Marostegui mentioned this in T226782: a1-eqiad pdu refresh (Tuesday 10/15 @11am UTC).Oct 1 2019, 6:16 AM
• Marostegui added a parent task: T226782: a1-eqiad pdu refresh (Tuesday 10/15 @11am UTC).
wiki_willy added a comment.Oct 11 2019, 4:42 PM

@Jclark-ctr - this arrived Thursday via https://www.fedex.com/en-us/home.html. Just a heads up, this will need to be replaced before the PDU upgrade next Tuesday, to retain redundant power on labsdb1009. Thanks, Willy

• Marostegui added a comment.Oct 11 2019, 4:43 PM

I believe we don't have to put the host down for the PSU replacement, do we?
However I would like to depool and stop mysql before, as a crash with mysql running could cause data corruption (and it has 7TB of data). So please ping me before any work on the host.

wiki_willy added a comment.Oct 11 2019, 4:45 PM

Yup, it should be a hot swap. So @Jclark-ctr - please reach out to @Marostegui before replacing it. Thanks, Willy

Jclark-ctr added a comment.Oct 11 2019, 11:41 PM

@Marostegui Received PSU. would like to replace Monday

• Marostegui added a comment.Oct 14 2019, 4:48 AM

Sounds good thanks - I will have this host ready.

gerritbot added a comment.Oct 14 2019, 4:53 AM

Change 542778 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/542778

gerritbot added a project: Patch-For-Review.Oct 14 2019, 4:53 AM
gerritbot added a comment.Oct 14 2019, 4:55 AM

Change 542778 merged by Marostegui:
[operations/puppet@production] dbproxy1011: Depool labsdb1009

https://gerrit.wikimedia.org/r/542778

Stashbot added a comment.Oct 14 2019, 4:56 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-14T04:56:53Z] <marostegui> Depool labsdb1009 for on-site maintenance - T233273

• Marostegui closed subtask Unknown Object (Task) as Resolved.Oct 14 2019, 5:04 AM
Maintenance_bot removed a project: Patch-For-Review.Oct 14 2019, 5:10 AM
Stashbot added a comment.Oct 14 2019, 7:16 AM

Mentioned in SAL (#wikimedia-operations) [2019-10-14T07:16:34Z] <marostegui> Stop MySQL on labsdb1009 for on-site maintenance - T233273

• Marostegui added a comment.Oct 14 2019, 9:22 AM

@Jclark-ctr you can proceed and change the PSU now. MySQL has been stopped.

Jclark-ctr added a comment.Oct 14 2019, 1:10 PM

Replaced failed PSU

• Marostegui added a comment.Oct 14 2019, 1:15 PM

Thanks John!
The alert recovered:

Sensor Type(s) Temperature, Power_Supply Status: OK
	
	This service is currently in a period of scheduled downtime	View Extra Service Notes	OK	2019-10-14 13:05:54	0d 0h 8m 15s	1/3	Sensor Type(s) Temperature, Power_Supply Status: OK

On the HW logs I see:

/system1/powersupply1
  Targets
  Properties
    ElementName=Power Supply
    OperationalStatus=Ok
    HealthState=Good, In Use
  Verbs
    cd version exit show


</system1>hpiLO-> show powersupply2

status=0
status_tag=COMMAND COMPLETED
Mon Oct 14 13:09:51 2019



/system1/powersupply2
  Targets
  Properties
    ElementName=Power Supply
    OperationalStatus=Ok
    HealthState=Good, In Use
  Verbs
    cd version exit show
Jclark-ctr closed this task as Resolved.Oct 14 2019, 1:23 PM
Stashbot added a comment.Oct 14 2019, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-14T13:42:51Z] <marostegui> Repool labsdb1009 after PSU replacement - T233273

wiki_willy added a comment.Oct 14 2019, 6:35 PM

@Jclark-ctr @Marostegui - thanks guys

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL