Page MenuHomePhabricator
Create Task
Maniphest T338354

db1135 has crashed
Closed, ResolvedPublic
Actions

Description

Can't ssh into it.

getsel:

-------------------------------------------------------------------------------
Record:      2
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.
-------------------------------------------------------------------------------
Record:      3
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      4
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      5
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      6
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      7
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      8
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      9
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      10
Date/Time:   06/07/2023 17:29:39
Source:      system
Severity:    Ok
Description: An OEM diagnostic event occurred.
-------------------------------------------------------------------------------
Record:      11
Date/Time:   06/07/2023 17:32:54
Source:      system
Severity:    Ok
Description: A problem was detected related to the previous server boot.
-------------------------------------------------------------------------------
Record:      12
Date/Time:   06/07/2023 17:32:54
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B6.
-------------------------------------------------------------------------------

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
db1135: Disable notificationoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects

Event Timeline

Ladsgroup created this task.Jun 7 2023, 5:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 7 2023, 5:39 PM
Ladsgroup updated the task description. (Show Details)Jun 7 2023, 5:43 PM
cmooney subscribed.EditedJun 7 2023, 5:49 PM

iDRAC reporting the same error message, paused waiting for user input OS is down.

image.png (771×889 px, 64 KB)

Ladsgroup added a project: ops-eqiad.Jun 7 2023, 5:51 PM
gerritbot added a comment.Jun 7 2023, 5:54 PM

Change 928127 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] db1135: Disable notification

https://gerrit.wikimedia.org/r/928127

gerritbot added a project: Patch-For-Review.Jun 7 2023, 5:54 PM

Change 928127 merged by Ladsgroup:

[operations/puppet@production] db1135: Disable notification

https://gerrit.wikimedia.org/r/928127

Maintenance_bot removed a project: Patch-For-Review.Jun 7 2023, 6:10 PM
Stashbot added a comment.Jun 7 2023, 6:27 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T18:27:04Z] <ladsgroup@cumin1001> START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1135.eqiad.wmnet with reason: T338354

Stashbot added a comment.Jun 7 2023, 6:27 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T18:27:17Z] <ladsgroup@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1135.eqiad.wmnet with reason: T338354

Maintenance_bot added a project: SRE.Jun 7 2023, 6:29 PM
Jclark-ctr subscribed.EditedJun 7 2023, 6:57 PM

This server is out of warranty i can pull a dimm from decom server if needed

jcrespo subscribed.Jun 8 2023, 3:17 PM
In T338354#8911065, @Jclark-ctr wrote:

This server is out of warranty i can pull a dimm from decom server if needed

I think this is the only thing we can do, please go ahead, the server has disabled notifications (although I think it is stuck in POST).

jcrespo added a subscriber: Marostegui.Jun 8 2023, 3:25 PM

@Ladsgroup @Marostegui As you will be back before I am, remember to (in case you want to do it, if not you can wait for me):

  • Resetup data (can be done from the lastest snapshot, as documented)
  • Remove the manual disabled notifications (had to be done because puppet hadn't run there)
  • Remove the pupper disabled notifications
  • Remove or let expire the downtime
  • Start replication, etc.
  • Repool
Jclark-ctr claimed this task.Jun 12 2023, 1:28 PM
Jclark-ctr added a comment.Jun 12 2023, 6:54 PM

Replaced Failed Dimm DIMM_B6

Jclark-ctr closed this task as Resolved.Jun 12 2023, 6:54 PM
Ladsgroup added a comment.Jun 12 2023, 8:58 PM

Thanks! I'm setting the mysql up and making sure it's getting replicated.

Stashbot added a comment.Jun 16 2023, 10:38 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-16T10:38:34Z] <Amir1> root@cumin1001:/home/ladsgroup/software2/dbtools# cat s1.dblist | grep -v "#" | while read db; do cat tables_to_check.txt | while read table index; do echo "$db.$table"; db-compare $db $table $index db1135.eqiad.wmnet:3306 db1118 db1139:3311 || break 2; done ; done (T338354)

Stashbot added a comment.Jun 19 2023, 10:16 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-19T10:16:53Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 10%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49446 and previous config saved to /var/cache/conftool/dbconfig/20230619-101653-ladsgroup.json

Ladsgroup added a comment.Jun 19 2023, 10:17 AM

The data check didn't bring any difference. Repooling

Stashbot added a comment.Jun 19 2023, 10:31 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-19T10:31:58Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 25%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49447 and previous config saved to /var/cache/conftool/dbconfig/20230619-103157-ladsgroup.json

Stashbot added a comment.Jun 19 2023, 10:47 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-19T10:47:03Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 75%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49448 and previous config saved to /var/cache/conftool/dbconfig/20230619-104702-ladsgroup.json

Stashbot added a comment.Jun 19 2023, 11:02 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-19T11:02:07Z] <ladsgroup@cumin1001> dbctl commit (dc=all): 'db1135 (re)pooling @ 100%: Maint over (T338354)', diff saved to https://phabricator.wikimedia.org/P49449 and previous config saved to /var/cache/conftool/dbconfig/20230619-110207-ladsgroup.json

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits