Page MenuHomePhabricator
Create Task
Maniphest T250602

db1140 (backup source) crashed
Closed, ResolvedPublic
Actions

Description

Times in UTC:

03:26:26 <+icinga-wm> PROBLEM - Host db1140 is DOWN: PING CRITICAL - Packet loss = 100%
iLO Standard 1.40 at  Feb 05 2019
Server Name:
Server Power: Off

</>hpiLO->

Looks like a HW issue:

/system1/log1/record21
  Targets
  Properties
    number=21
    severity=Critical
    date=04/19/2020
    time=03:22:39
    description=Server Critical Fault (Service Information: Runtime Fault, Processor(Intel),  Processor 1 (02h))
  Verbs
    cd version exit show

This server is under warranty.
The backups that run from this host are x1 and s2.

Details

SubjectRepoBranchLines +/-
Revert "db1140: Disable notifications"operations/puppetproduction+0 -1
db1140: Disable notificationsoperations/puppetproduction+1 -0
install_server: Update NIC hw address for db1140operations/puppetproduction+1 -1
mariadb-backups: Prepare db1140 for reimage to busteroperations/puppetproduction+3 -5
mariadb-backups: Enable notifications on db1095 after failover from db1140operations/puppetproduction+0 -1
mariadb-backups: Move s2, x1 eqiad backups to db1095operations/puppetproduction+4 -4
mariadb-backups: Add s2, x1 to db1095 (eqiad backup source)operations/puppetproduction+5 -2
Customize query in gerrit

Related Objects

Event Timeline

• Marostegui created this task.Apr 19 2020, 5:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 19 2020, 5:51 AM
Stashbot added a comment.Apr 19 2020, 5:51 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-19T05:51:57Z] <marostegui> Power back on db1140 T250602

• Marostegui assigned this task to jcrespo.Apr 19 2020, 6:00 AM
• Marostegui triaged this task as High priority.
• Marostegui added projects: ops-eqiad, DC-Ops.
• Marostegui added subscribers: jcrespo, wiki_willy.

This server is fully broken apparently, it is not powering ON :-( - I have tried multiple combinations:

</>hpiLO-> power on

status=0
status_tag=COMMAND COMPLETED
Sun Apr 19 05:52:58 2020



Server powering on .......


</>hpiLO-> power

status=0
status_tag=COMMAND COMPLETED
Sun Apr 19 05:54:29 2020



power: server power is currently: Off

</>hpiLO-> power reset

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND ERROR-UNSPECIFIED
Sun Apr 19 05:55:27 2020

Server power off.




</>hpiLO-> power

status=0
status_tag=COMMAND COMPLETED
Sun Apr 19 05:55:48 2020



power: server power is currently: Off

</>hpiLO-> power off hard

status=2
status_tag=COMMAND PROCESSING FAILED
error_tag=COMMAND ERROR-UNSPECIFIED
Sun Apr 19 05:56:51 2020

Server power already off.




</>hpiLO-> power on

status=0
status_tag=COMMAND COMPLETED
Sun Apr 19 05:56:57 2020



Server powering on .......



</>hpiLO-> power

status=0
status_tag=COMMAND COMPLETED
Sun Apr 19 05:58:22 2020



power: server power is currently: Off

We should probably contact HP and get a new mother board - assigning to @jcrespo so he decide on the next steps for this backup source host.

• Marostegui moved this task from Triage to In progress on the DBA board.Apr 19 2020, 6:00 AM
gerritbot added a comment.Apr 19 2020, 7:47 AM

Change 590255 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1140: Disable notifications

https://gerrit.wikimedia.org/r/590255

gerritbot added a project: Patch-For-Review.Apr 19 2020, 7:47 AM
gerritbot added a comment.Apr 19 2020, 7:50 AM

Change 590255 merged by Marostegui:
[operations/puppet@production] db1140: Disable notifications

https://gerrit.wikimedia.org/r/590255

Maintenance_bot removed a project: Patch-For-Review.Apr 19 2020, 8:10 AM
gerritbot added a comment.Apr 20 2020, 7:47 AM

Change 590995 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Add s2, x1 to db1095 (eqiad backup source)

https://gerrit.wikimedia.org/r/590995

gerritbot added a project: Patch-For-Review.Apr 20 2020, 7:47 AM
gerritbot added a comment.Apr 20 2020, 8:06 AM

Change 590995 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Add s2, x1 to db1095 (eqiad backup source)

https://gerrit.wikimedia.org/r/590995

Stashbot added a comment.Apr 20 2020, 8:09 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-20T08:09:44Z] <jynus> restarting s3 instance on db1095 to reduce its buffer pool T250602

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2020, 8:10 AM
gerritbot added a comment.Apr 20 2020, 8:20 AM

Change 590999 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Move s2, x1 eqiad backups to db1095

https://gerrit.wikimedia.org/r/590999

gerritbot added a project: Patch-For-Review.Apr 20 2020, 8:20 AM
gerritbot added a comment.Apr 20 2020, 12:52 PM

Change 590999 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Move s2, x1 eqiad backups to db1095

https://gerrit.wikimedia.org/r/590999

Maintenance_bot removed a project: Patch-For-Review.Apr 20 2020, 1:10 PM
jcrespo reassigned this task from jcrespo to wiki_willy.Apr 20 2020, 1:14 PM

Service failover done, this is now 100% on DC-Ops side for vendor handling, as per description and initial triage by Manuel. No production service is there now, and data is useless (will be reimaged) -can be handled freely by dcops (although please note there is private data on these disks).

gerritbot added a comment.Apr 20 2020, 1:18 PM

Change 591047 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Enable notifications on db1095 after failover from db1140

https://gerrit.wikimedia.org/r/591047

gerritbot added a project: Patch-For-Review.Apr 20 2020, 1:18 PM
• Cmjohnson claimed this task.Apr 20 2020, 1:55 PM
• Cmjohnson subscribed.

@Marostegui @jynus Before I can start a ticket with HPE some local troubleshooting has to be done.

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Apr 20 2020, 1:56 PM
Stashbot added a comment.Apr 21 2020, 5:34 AM

Mentioned in SAL (#wikimedia-operations) [2020-04-21T05:34:11Z] <marostegui> Add db1095:3312, db1095:3320 to tendril - T250602

gerritbot added a comment.Apr 21 2020, 8:48 AM

Change 591047 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Enable notifications on db1095 after failover from db1140

https://gerrit.wikimedia.org/r/591047

Maintenance_bot removed a project: Patch-For-Review.Apr 21 2020, 9:10 AM
• Cmjohnson added a comment.Apr 21 2020, 3:39 PM

I pulled all power plugs, reseated psu's, DIMM and CPU. The server will not power on, the LEDs are flashing orange and red.

• Cmjohnson added a comment.Apr 21 2020, 4:11 PM

@Marostegui this could be down for a bit, between HPE troubleshooting and getting a tech on-site.

RhinosF1 subscribed.Apr 21 2020, 4:13 PM
jcrespo added a comment.Apr 21 2020, 4:20 PM

I will be the first contact for this server (although manuel will be around if needed, ofc :-D) @Cmjohnson we are aware- that is why migrated the service away, as it couldn't wait for the repair/etc. So that is not immediate issue. However, the reduced redundancy is not ideal for long term- we will budget additional servers for next fiscal. Thanks for taking care of it!

• Cmjohnson reassigned this task from • Cmjohnson to Jclark-ctr.May 4 2020, 3:25 PM
• Cmjohnson added a subscriber: Jclark-ctr.

@Jclark-ctr can you start the process with HPE please.

Jclark-ctr added a comment.May 6 2020, 12:54 PM

led`s flashing in sequence of 2 most likely a processor error

Case Reference ID: 5346998524
Status: Case is generated and in Progress
Product: HPE ProLiant DL360 Gen10 8SFF Configure-to-order Server
Product number: 867959-B21
Serial number: MXQ91300JD

Jclark-ctr added a comment.May 14 2020, 2:57 PM

updated ticket with HP. Scheduled main board replacement.

• Marostegui added a comment.May 19 2020, 7:31 PM

Downtime expired - I have acked the alerts in Icinga

Jclark-ctr added a comment.May 19 2020, 9:22 PM

Main board replaced today entered password & management address into server

• Marostegui reassigned this task from Jclark-ctr to jcrespo.EditedMay 21 2020, 4:43 AM

Per my IRC chat with John, assigning this back to Jaime as the on-site part is done
Thank you John!

jcrespo added a comment.May 21 2020, 1:43 PM

Per my IRC chat with John

Could you tell me more, as before a processor error was mentioned, but then a board change?

• Marostegui added a comment.May 21 2020, 1:45 PM
In T250602#6155334, @jcrespo wrote:

Per my IRC chat with John

Could you tell me more, as before a processor error was mentioned, but then a board change?

My chat was more like a heads up from John that the mainboard was replaced and to check if we are able to connect back to the host.
I would assume that the vendor decided just to replace the whole mainboard rather than just the CPU (this is a guess)

gerritbot added a comment.May 21 2020, 2:00 PM

Change 597780 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb-backups: Prepare db1140 for reimage to buster

https://gerrit.wikimedia.org/r/597780

gerritbot added a project: Patch-For-Review.May 21 2020, 2:00 PM
gerritbot added a comment.May 21 2020, 3:20 PM

Change 597780 merged by Jcrespo:
[operations/puppet@production] mariadb-backups: Prepare db1140 for reimage to buster

https://gerrit.wikimedia.org/r/597780

jcrespo reassigned this task from jcrespo to Jclark-ctr.May 21 2020, 3:46 PM

Hi,

I cannot reinstall the server because the remote ipmi interface doesn't work (and the ssh or the https acesses, that are enabled, don't accept my password). It looks like the password wasn't setup correctly after reset, but common default user/password combinations doesn't work either.

I could restart the server and change it from post utilities, but I cannot connect to serial interface through ssh/https to be able to do so- needs local access :-(.

Maintenance_bot removed a project: Patch-For-Review.May 21 2020, 4:11 PM
jcrespo added a comment.May 27 2020, 8:18 AM

Hi, @wiki_willy I just want to ping you so your team is aware that the maintenance here didn't complete correctly and that we need more onsite help (I don't need this fast, just making sure it doesn't fall under the radar).

wiki_willy added a comment.May 27 2020, 2:56 PM

Thanks @jcrespo . I don't think @Jclark-ctr has been onsite at the data center since the last update, but I'll follow up with him on this when he's out there this week. Thanks, Willy

Jclark-ctr added a comment.May 28 2020, 2:17 PM

corrected user name. Jcrespo confirmed able to log in

jcrespo claimed this task.May 28 2020, 2:20 PM
jcrespo added a comment.Jun 1 2020, 8:10 AM

@Jclark-ctr One last thing- this was not an issue for me because as I had remote login so I could fix it myself, but may be interesting for you: remote IPMI was disabled, so I had to enable on management interface, following the documented procedure.

ops-monitoring-bot added a comment.Jun 1 2020, 8:10 AM

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['db1140.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006010810_jynus_86215.log.

jcrespo reassigned this task from jcrespo to Jclark-ctr.Jun 1 2020, 8:53 AM

@Jclark-ctr Serial port redirection doesn't work. This is a blocker because I cannot read the console output on restart, and understand why it is not booting from network when requested on ipmi (it just rebooted into hard drive).

One last thing- this was not an issue for me because as I had remote login so I could fix it myself, but may be interesting for you: remote IPMI was disabled, so I had to enable on management interface, following the documented procedure.

This host seems to not have the regular WMF setup (serial port, initial setup), could you please have a look so the net installer works?

ops-monitoring-bot added a comment.Jun 2 2020, 2:17 PM

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['db1140.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006021417_jynus_99411.log.

gerritbot added a comment.Jun 2 2020, 2:30 PM

Change 601744 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] install_server: Update NIC hw address for db1140

https://gerrit.wikimedia.org/r/601744

gerritbot added a project: Patch-For-Review.Jun 2 2020, 2:30 PM
gerritbot added a comment.Jun 2 2020, 2:35 PM

Change 601744 merged by Jcrespo:
[operations/puppet@production] install_server: Update NIC hw address for db1140

https://gerrit.wikimedia.org/r/601744

ops-monitoring-bot added a comment.Jun 2 2020, 2:38 PM

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['db1140.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202006021438_jynus_117969.log.

jcrespo claimed this task.Jun 2 2020, 2:55 PM

Jclark changed the serial port and we now have output on serial console, including post. The other issue was the mac address changing because of the board change. After the 2 changes, reimage went through successfully. I will take care of remaining sw steps.

I think the reason why this took so long and failure-correction steps is that there is no documented procedure/checklist for DC Ops to update on HP hosts after a board replacement (network updates, installer updates, bios changes, managemement changes, ipmi changes, passwords, etc.). Even if it is not a common operation, maybe this ticket could be used as a starting point to document that so future maintenances are smoother :-D. CC willy

ops-monitoring-bot added a comment.Jun 2 2020, 2:57 PM

Completed auto-reimage of hosts:

['db1140.eqiad.wmnet']

and were ALL successful.

Maintenance_bot removed a project: Patch-For-Review.Jun 2 2020, 3:11 PM
wiki_willy added a comment.Jun 2 2020, 6:13 PM

Thanks @jcrespo , our documentation looks to be a bit outdated, so we'll get this added in

In T250602#6185325, @jcrespo wrote:

Jclark changed the serial port and we now have output on serial console, including post. The other issue was the mac address changing because of the board change. After the 2 changes, reimage went through successfully. I will take care of remaining sw steps.

I think the reason why this took so long and failure-correction steps is that there is no documented procedure/checklist for DC Ops to update on HP hosts after a board replacement (network updates, installer updates, bios changes, managemement changes, ipmi changes, passwords, etc.). Even if it is not a common operation, maybe this ticket could be used as a starting point to document that so future maintenances are smoother :-D. CC willy

wiki_willy mentioned this in T254272: Update Documentation for dl360 Motherboard Swap.Jun 2 2020, 6:31 PM
jcrespo closed this task as Resolved.Jun 3 2020, 10:23 AM
jcrespo reassigned this task from jcrespo to Jclark-ctr.

db1140 has been repopulated from dbprov snapshots of s1 and s6 and upgraded to 10.4.

It has been added to tendril and zarcillo, it appears on prometheus.

Replication is catching up on s1; s6 already did.

I will do now try backups and snapshots of these sections, even if we still have backup sources on 10.1 for both s1 and s6.

gerritbot added a comment.Jun 3 2020, 10:37 AM

Change 602044 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Revert "db1140: Disable notifications"

https://gerrit.wikimedia.org/r/602044

gerritbot added a project: Patch-For-Review.Jun 3 2020, 10:37 AM
gerritbot added a comment.Jun 3 2020, 10:42 AM

Change 602044 merged by Jcrespo:
[operations/puppet@production] Revert "db1140: Disable notifications"

https://gerrit.wikimedia.org/r/602044

Maintenance_bot removed a project: Patch-For-Review.Jun 3 2020, 11:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL