Page MenuHomePhabricator
Create Task
Maniphest T358421

db2118 crashed and rebooted due to HW
Closed, ResolvedPublic
Actions

Description

-------------------------------------------------------------------------------
Record:      27
Date/Time:   02/24/2024 10:08:18
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
-------------------------------------------------------------------------------

Details

SubjectRepoBranchLines +/-
instances.yaml: Remove db2118 from dbctloperations/puppetproduction+0 -1
db2118: Notes about its crashoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

taavi triaged this task as Unbreak Now! priority.Feb 24 2024, 10:18 AM
taavi created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 24 2024, 10:18 AM
Superpes15 subscribed.Feb 24 2024, 10:20 AM
RhinosF1 subscribed.Feb 24 2024, 10:20 AM
Marostegui subscribed.Feb 24 2024, 10:21 AM

Started it - InnoDB doing recovery, leaving it on RO. Once it's caught up I am switching it

RhinosF1 added a subtask: T358423: Switchover s7 master (db2118 -> db2121).Feb 24 2024, 10:23 AM
DerHexer subscribed.Feb 24 2024, 10:25 AM
Marostegui added a comment.Feb 24 2024, 10:25 AM

Even though mariadb is up, it is all in RO. I don't want to risk the data.

Minorax subscribed.Feb 24 2024, 10:26 AM
Ameisenigel subscribed.Feb 24 2024, 10:27 AM
Marostegui added a comment.Feb 24 2024, 10:28 AM
-------------------------------------------------------------------------------
Record:      26
Date/Time:   02/24/2024 10:08:18
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      27
Date/Time:   02/24/2024 10:08:18
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
-------------------------------------------------------------------------------
Record:      28
Date/Time:   02/24/2024 10:08:18
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      29
Date/Time:   02/24/2024 10:08:18
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      30
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      31
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      32
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Critical
Description: CPU 2 machine check error detected.
-------------------------------------------------------------------------------
Record:      33
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      34
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      35
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      36
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      37
Date/Time:   02/24/2024 10:08:19
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      38
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      39
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      40
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      41
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      42
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      43
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      44
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      45
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      46
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      47
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      48
Date/Time:   02/24/2024 10:08:20
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      49
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      50
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      51
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      52
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      53
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      54
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      55
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      56
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      57
Date/Time:   02/24/2024 10:08:21
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      58
Date/Time:   02/24/2024 10:08:22
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      59
Date/Time:   02/24/2024 10:08:22
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Marostegui added a project: ops-eqiad.Feb 24 2024, 10:29 AM
Marostegui added a subscriber: wiki_willy.

@wiki_willy can we contact the vendor about this issue which caused a reboot?

Record:      27
Date/Time:   02/24/2024 10:08:18
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
Marostegui renamed this task from db2118 crashed to db2118 crashed and rebooted due to HW.Feb 24 2024, 10:29 AM
Marostegui updated the task description. (Show Details)
MdsShakil subscribed.Feb 24 2024, 10:47 AM
Marostegui lowered the priority of this task from Unbreak Now! to High.Feb 24 2024, 10:48 AM

Everything should be back to normal now.

Marostegui closed subtask T358423: Switchover s7 master (db2118 -> db2121) as Resolved.Feb 24 2024, 10:51 AM
Marostegui moved this task from Active investigation to Awaiting report on the Wikimedia-Incident board.Feb 24 2024, 11:36 AM
Marostegui moved this task from Triage to In progress on the DBA board.
LSobanski subscribed.Feb 26 2024, 8:22 AM
wiki_willy added subscribers: VRiley-WMF, Jclark-ctr.Feb 26 2024, 6:18 PM

++ @VRiley-WMF & @Jclark-ctr

In T358421#9574362, @Marostegui wrote:

@wiki_willy can we contact the vendor about this issue which caused a reboot?

Record:      27
Date/Time:   02/24/2024 10:08:18
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
Jhancock.wm edited projects, added ops-codfw; removed ops-eqiad.Feb 27 2024, 1:41 AM
Jhancock.wm subscribed.Feb 27 2024, 1:43 AM

This server is in codfw. I'll get a report sent to Dell asap to get a replacement cpu

Jhancock.wm added a comment.Feb 27 2024, 1:46 AM

actually, this server is not in warranty. I will try to find a viable replacement from the decommissioned inventory in the morning.

Jhancock.wm moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-codfw board.Feb 27 2024, 1:46 AM
wiki_willy added a comment.Feb 27 2024, 2:09 AM

Thanks for picking this up @Jhancock.wm. @Marostegui - since this host looks like it's close to being refreshed in T355350, do you want to just wait for the refreshed server to be setup instead of fixing this one? Thanks, Willy

Marostegui added a comment.Feb 27 2024, 6:03 AM
In T358421#9578961, @wiki_willy wrote:

Thanks for picking this up @Jhancock.wm. @Marostegui - since this host looks like it's close to being refreshed in T355350, do you want to just wait for the refreshed server to be setup instead of fixing this one? Thanks, Willy

Ah yes. I was not aware of this. We can wait for its new replacement server. Thanks!

Marostegui removed a project: ops-codfw.Feb 27 2024, 6:31 AM

Started data consistency check

Marostegui mentioned this in T355422: Productionize db2196-db2220.Feb 27 2024, 6:36 AM
gerritbot added a comment.Feb 27 2024, 6:42 AM

Change 1006750 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] db2118: Notes about its crash

https://gerrit.wikimedia.org/r/1006750

gerritbot added a project: Patch-For-Review.Feb 27 2024, 6:42 AM
gerritbot added a comment.Feb 27 2024, 6:44 AM

Change 1006750 merged by Marostegui:

[operations/puppet@production] db2118: Notes about its crash

https://gerrit.wikimedia.org/r/1006750

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2024, 7:30 AM
Marostegui added a comment.Feb 28 2024, 7:10 AM

The data looks correct. I am not going to repool this host for now, I am going to wait until its replacement in T355350 gets installed and simply clone that one and decommission this one.

Marostegui lowered the priority of this task from High to Medium.Feb 28 2024, 7:11 AM
Marostegui added a parent task: T355422: Productionize db2196-db2220.
Stashbot added a comment.Feb 29 2024, 6:26 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-29T06:26:01Z] <marostegui@cumin1002> dbctl commit (dc=all): 'Pool db2218 with 1% weight only T358421 T355422', diff saved to https://phabricator.wikimedia.org/P58171 and previous config saved to /var/cache/conftool/dbconfig/20240229-062601-marostegui.json

gerritbot added a comment.Feb 29 2024, 6:27 AM

Change 1007481 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] instances.yaml: Remove db2118 from dbctl

https://gerrit.wikimedia.org/r/1007481

gerritbot added a project: Patch-For-Review.Feb 29 2024, 6:27 AM

Change 1007481 merged by Marostegui:

[operations/puppet@production] instances.yaml: Remove db2118 from dbctl

https://gerrit.wikimedia.org/r/1007481

Marostegui closed this task as Resolved.Feb 29 2024, 6:28 AM
Marostegui claimed this task.

This host will no longer come back to production. I will decommission it in a couple of days.

Maintenance_bot moved this task from In progress to Done on the DBA board.Feb 29 2024, 6:29 AM
Maintenance_bot removed a project: Patch-For-Review.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits