Page MenuHomePhabricator
Create Task
Maniphest T193325

db2081 crashed/rebooted, probably due to hardware failure
Closed, ResolvedPublic
Actions

Description

Needs investigation by #DBAs first on why and mitigate the causes later.

Details

SubjectRepoBranchLines +/-
mariadb: Repool db2081 after crash & checkoperations/mediawiki-configmaster+2 -2
db2081: Reenable notificationsoperations/puppetproduction+0 -1
mariadb: Disable notifications for db2081, crashedoperations/puppetproduction+1 -0
mariadb: Depool db2081, crashedoperations/mediawiki-configmaster+2 -2
Customize query in gerrit

Related Objects

Event Timeline

jcrespo created this task.Apr 28 2018, 3:39 PM
Restricted Application added a project: SRE. · View Herald TranscriptApr 28 2018, 3:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot subscribed.Apr 28 2018, 3:40 PM

Change 429573 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool db2081, crashed

https://gerrit.wikimedia.org/r/429573

gerritbot added a project: Patch-For-Review.Apr 28 2018, 3:40 PM
jcrespo triaged this task as High priority.Apr 28 2018, 3:45 PM
gerritbot added a comment.Apr 28 2018, 3:46 PM

Change 429573 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool db2081, crashed

https://gerrit.wikimedia.org/r/429573

jcrespo removed a project: Patch-For-Review.Apr 28 2018, 3:48 PM
gerritbot added a comment.Apr 28 2018, 3:52 PM

Change 429575 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Disable notifications for db2081, crashed

https://gerrit.wikimedia.org/r/429575

gerritbot added a project: Patch-For-Review.Apr 28 2018, 3:52 PM

Change 429575 merged by Jcrespo:
[operations/puppet@production] mariadb: Disable notifications for db2081, crashed

https://gerrit.wikimedia.org/r/429575

jcrespo removed a project: Patch-For-Review.Apr 28 2018, 3:56 PM
• Marostegui assigned this task to Papaul.Apr 28 2018, 5:12 PM
• Marostegui moved this task from Triage to In progress on the DBA board.
• Marostegui added a subscriber: Papaul.

Same error we experienced at: T175973#3615656

PWR2262: The Intel Management Engine has reported an internal system error.
 2018-04-28T15:16:13-0500
Log Sequence Number: 154
Detailed Description:
The Intel Management Engine is unable to utilize the PECI over DMI facility.
Recommended Action:
Look for the PWR2264 "normal system operation" message in the Lifecycle Log after the PWR2262 entry. It may take 30 seconds for the message to be logged. If the PWR2264 message does not appear, do the following: Disconnect power from the server and reconnect power. Turn on the server. If the issue persists, contact your service provider.

That error was only visible via the idrac web interface. Nothing on the idrac shell. Nothing on the controller logs, syslog, remote syslog, kernel etc

@Papaul can we check and upgrade BIOS/firwmare and then do power disconnection+connection?

Papaul added a comment.Apr 28 2018, 5:21 PM

Okay will check that on Monday.

• Marostegui mentioned this in T193331: db1098 crashed and got rebooted.Apr 29 2018, 12:15 AM
• Marostegui added a comment.Apr 29 2018, 11:35 AM

As a side note. Either db1098 (T193331), db2081 (T193325) and db1100 (T175973) (they were all coming from the same batch of purchases (T162159 and T162233)) all crashed with the same errors.

PWR2262: The Intel Management Engine has reported an internal system error.
 2018-04-28T15:16:13-0500
Log Sequence Number: 154
Detailed Description:
The Intel Management Engine is unable to utilize the PECI over DMI facility.
Recommended Action:
Look for the PWR2264 "normal system operation" message in the Lifecycle Log after the PWR2262 entry. It may take 30 seconds for the message to be logged. If the PWR2264 message does not appear, do the following: Disconnect power from the server and reconnect power. Turn on the server. If the issue persists, contact your service provider.

db1100 never crashed again after doing the power drain.

Stashbot subscribed.Apr 30 2018, 2:26 PM

Mentioned in SAL (#wikimedia-operations) [2018-04-30T14:26:34Z] <marostegui> Power off db2081 for HW maintenance - T193325

gerritbot added a comment.Apr 30 2018, 2:49 PM

Change 429816 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db2081: Reenable notifications

https://gerrit.wikimedia.org/r/429816

gerritbot added a project: Patch-For-Review.Apr 30 2018, 2:49 PM
Papaul reassigned this task from Papaul to • Marostegui.Apr 30 2018, 3:15 PM

1- power disconnection+connection
2- update BIOS from 2.5.5 to 2.7.1
3- update IDRAC from 2.50 to 2.52

• Marostegui added a comment.Apr 30 2018, 3:21 PM

Thanks @Papaul - I have started MySQL to let it replicate for a couple of days before closing this.
I will leave the host depooled too, just in case.

MySQL and kernel have been upgraded too.

gerritbot added a comment.Apr 30 2018, 3:22 PM

Change 429816 merged by Marostegui:
[operations/puppet@production] db2081: Reenable notifications

https://gerrit.wikimedia.org/r/429816

jcrespo added a comment.May 2 2018, 3:48 PM

I am leaving a check ongoing on wikidatawiki on some codfw hosts to proof no data was lost.

jcrespo added a comment.May 3 2018, 7:33 AM

The check detected some difference, but they could be false positives, checking again.

• Marostegui added a comment.May 4 2018, 8:29 AM
In T193325#4177621, @jcrespo wrote:

The check detected some difference, but they could be false positives, checking again.

Was it a false positive in the end?

• Marostegui lowered the priority of this task from High to Medium.May 4 2018, 8:29 AM
jcrespo added a comment.May 4 2018, 9:10 AM

Yes, no differences were assured on a second run. I will repool the server now.

gerritbot added a comment.May 4 2018, 11:56 AM

Change 430894 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db2081 after crash & check

https://gerrit.wikimedia.org/r/430894

gerritbot added a comment.May 4 2018, 12:01 PM

Change 430894 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db2081 after crash & check

https://gerrit.wikimedia.org/r/430894

jcrespo closed this task as Resolved.May 4 2018, 1:49 PM
• Vvjjkkii renamed this task from db2081 crashed/rebooted, probably due to hardware failure to 60daaaaaaa.Jul 1 2018, 1:13 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed • Marostegui as the assignee of this task.
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
• Marostegui renamed this task from 60daaaaaaa to db2081 crashed/rebooted, probably due to hardware failure.Jul 1 2018, 7:29 AM
• Marostegui closed this task as Resolved.
• Marostegui claimed this task.
• Marostegui lowered the priority of this task from High to Medium.
• Marostegui removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
• Marostegui updated the task description. (Show Details)
• Marostegui added subscribers: Aklapper, GerritBot.
Aklapper edited subscribers, added: gerritbot; removed: GerritBot.May 16 2023, 10:18 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.May 16 2023, 10:29 AM
Maintenance_bot removed a project: Patch-For-Review.May 16 2023, 10:39 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL