Page MenuHomePhabricator
Create Task
Maniphest T326119

hw troubleshooting: CPU1 machine check error on parse1002.eqiad.wmnet
Closed, ResolvedPublic
Actions

Description

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
  • - Put system into a failed state in Netbox.
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)
  • - Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

parse1002.eqiad.wmnet is down and per SEL it has a broken CPU:

Record:      165
Date/Time:   01/03/2023 02:20:24
Source:      system
Severity:    Critical
Description: CPU 1 machine check error detected.

I've set it to inactive in conftool, it can be powered down for analysis/replacement any time.

Host is downtimed for 7 days.

Details

SubjectRepoBranchLines +/-
Revert "dsh: Remove parse1002 from parsoid dsh group"operations/puppetproduction+1 -1
dsh: Remove parse1002 from parsoid dsh groupoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects

Event Timeline

MoritzMuehlenhoff created this task.Jan 3 2023, 8:19 AM
Maintenance_bot added a project: SRE.Jan 3 2023, 8:29 AM
ops-monitoring-bot added a comment.Jan 3 2023, 10:37 AM

Icinga downtime and Alertmanager silence (ID=8a35d570-5625-4e3d-a6ff-eb737a303711) set by cgoubert@cumin1001 for 7 days, 0:00:00 on 1 host(s) and their services with reason: CPU1 machine check error

parse1002.eqiad.wmnet
Clement_Goubert renamed this task from Broken CPU on parse1002 to hw troubleshooting: CPU1 machine check error on parse1002.eqiad.wmnet.Jan 3 2023, 10:38 AM
Clement_Goubert assigned this task to • Cmjohnson.
Clement_Goubert triaged this task as High priority.
Clement_Goubert lowered the priority of this task from High to Medium.
Clement_Goubert updated Other Assignee, added: Jclark-ctr.
Clement_Goubert edited projects, added DC-Ops; removed SRE.
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops board.
Clement_Goubert reassigned this task from • Cmjohnson to Jclark-ctr.Jan 3 2023, 10:47 AM
Clement_Goubert updated Other Assignee, removed: Jclark-ctr.
Clement_Goubert added a subscriber: • Cmjohnson.
wiki_willy moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jan 3 2023, 7:30 PM
gerritbot added a comment.Jan 4 2023, 4:38 PM

Change 875360 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] dsh: Remove parse1002 from parsoid dsh group

https://gerrit.wikimedia.org/r/875360

gerritbot added a project: Patch-For-Review.Jan 4 2023, 4:38 PM
gerritbot added a comment.Jan 4 2023, 4:58 PM

Change 875360 merged by Clément Goubert:

[operations/puppet@production] dsh: Remove parse1002 from parsoid dsh group

https://gerrit.wikimedia.org/r/875360

Clement_Goubert subscribed.Jan 4 2023, 4:59 PM
In T326119#8499305, @gerritbot wrote:

Change 875360 merged by Clément Goubert:

[operations/puppet@production] dsh: Remove parse1002 from parsoid dsh group

https://gerrit.wikimedia.org/r/875360

Note not to forget to revert this once fixed

Maintenance_bot removed a project: Patch-For-Review.Jan 4 2023, 5:30 PM
Jclark-ctr added a comment.Jan 6 2023, 2:38 PM

Sorry did not give update. Case# 159648923 was submitted 1/4/2023
Idrac was not reachable remotely. Reset Idrac with crash cart 1/6/2023
TSR report sent to Dell 1/6/2023

Clement_Goubert added a comment.Jan 6 2023, 4:04 PM

Thanks for the update. I will extend the downtime to two weeks from now, will revisit if necessary.

ops-monitoring-bot added a comment.Jan 6 2023, 4:05 PM

Icinga downtime and Alertmanager silence (ID=5c4b686a-9560-44c1-acb3-c16978d72b37) set by cgoubert@cumin1001 for 14 days, 0:00:00 on 1 host(s) and their services with reason: CPU1 machine check error

parse1002.eqiad.wmnet
Jclark-ctr added a comment.EditedJan 9 2023, 2:51 PM

Cleared SEL Dell requested set the system profile to performance

Clement_Goubert added a comment.Jan 9 2023, 3:48 PM
In T326119#8509402, @Jclark-ctr wrote:

Cleared SEL Dell requested set the system profile to performance

The cpu governor is already performance, are Dell talking about a different kind of system profile?

Jclark-ctr closed this task as Resolved.Jan 9 2023, 5:59 PM

@Clement_Goubert. I updated this morning. Dell has said this will resolve our issue I am closing this ticket and hope it isn’t going to return

gerritbot added a comment.Jan 9 2023, 6:09 PM

Change 877207 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] Revert "dsh: Remove parse1002 from parsoid dsh group"

https://gerrit.wikimedia.org/r/877207

gerritbot added a project: Patch-For-Review.Jan 9 2023, 6:09 PM
Clement_Goubert added a comment.Jan 9 2023, 6:09 PM

@Jclark-ctr Thank you :) I'll repool the machine and remove the downtimes tomorrow.

gerritbot added a comment.Jan 10 2023, 9:52 AM

Change 877207 merged by Clément Goubert:

[operations/puppet@production] Revert "dsh: Remove parse1002 from parsoid dsh group"

https://gerrit.wikimedia.org/r/877207

Stashbot added a comment.Jan 10 2023, 10:14 AM

Mentioned in SAL (#wikimedia-operations) [2023-01-10T10:14:35Z] <claime> repooled parse1002.eqiad.wmnet - T326119

Maintenance_bot removed a project: Patch-For-Review.Jan 10 2023, 10:30 AM
akosiaris mentioned this in T339340: hw troubleshooting: CPU machine check failure for parse1002.eqiad.wmnet.Jun 16 2023, 9:00 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL