Page MenuHomePhabricator
Create Task
Maniphest T221502

db1099 memory issues
Closed, ResolvedPublic
Actions

Description

db1099 is having memory issues:

racadm>>racadm getsel

racadm getsel
Record:      1
Date/Time:   03/30/2017 16:15:55
Source:      system
Severity:    Ok
Description: Log cleared.
-------------------------------------------------------------------------------
Record:      2
Date/Time:   04/20/2019 13:33:36
Source:      system
Severity:    Non-Critical
Description: Correctable memory error rate exceeded for DIMM_A5.
-------------------------------------------------------------------------------
Record:      3
Date/Time:   04/20/2019 13:56:56
Source:      system
Severity:    Critical
Description: Correctable memory error rate exceeded for DIMM_A5.
-------------------------------------------------------------------------------

Should we exchange this module with another existing one to see if it start reporting errors on a different location to discard memory slot issues?

Details

SubjectRepoBranchLines +/-
db-eqiad.php: Depool db1099operations/mediawiki-configmaster+12 -12
db-eqiad.php: Slowly repool db1099operations/mediawiki-configmaster+2 -2
db-eqiad.php: Depool db1099operations/mediawiki-configmaster+12 -12
Customize query in gerrit

Event Timeline

• Marostegui created this task.Apr 20 2019, 5:54 PM
Restricted Application added a project: SRE. · View Herald TranscriptApr 20 2019, 5:54 PM
• Marostegui triaged this task as Medium priority.Apr 20 2019, 5:55 PM
• Marostegui moved this task from Triage to In progress on the DBA board.Apr 21 2019, 6:09 AM
gerritbot added a comment.Apr 22 2019, 5:21 AM

Change 505481 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099

https://gerrit.wikimedia.org/r/505481

gerritbot added a project: Patch-For-Review.Apr 22 2019, 5:21 AM
gerritbot added a comment.Apr 22 2019, 5:23 AM

Change 505481 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099

https://gerrit.wikimedia.org/r/505481

Stashbot added a comment.Apr 22 2019, 5:25 AM

Mentioned in SAL (#wikimedia-operations) [2019-04-22T05:25:52Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099 T221502 (duration: 01m 15s)

Stashbot added a comment.Apr 22 2019, 5:26 AM

Mentioned in SAL (#wikimedia-operations) [2019-04-22T05:25:59Z] <marostegui> Stop MySQL and reboot db1099 to see if memory errors clear up T221502

• Marostegui added a comment.Apr 22 2019, 5:33 AM

I rebooted the host to see if the memory errors would clear up, but it didn't happen, so I guess we have to either contact Dell or move the DIMM to a different slot and wait to see if it happens again on a different location
@Cmjohnson please advise

Dzahn moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Apr 30 2019, 6:34 PM
• Marostegui closed this task as Resolved.May 6 2019, 1:09 PM

This host recovered itself, so closing for now as nothing is to be done.

gerritbot added a comment.May 30 2019, 12:59 PM

Change 513278 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099

https://gerrit.wikimedia.org/r/513278

gerritbot added a comment.May 30 2019, 1:01 PM

Change 513278 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1099

https://gerrit.wikimedia.org/r/513278

Stashbot added a comment.May 30 2019, 1:02 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-30T13:02:58Z] <marostegui@deploy1001> Synchronized wmf-config/db-eqiad.php: Depool db1099 T221502 (duration: 00m 56s)

Stashbot added a comment.May 30 2019, 1:03 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-30T13:03:20Z] <marostegui> Stop MySQL on db1099 for onsite maintenance - T221502

• Marostegui reopened this task as Open.May 30 2019, 1:06 PM

re-opening as this is going to be worked out.
MySQL is stopped on s1 and s8, host downtimed and OS upgraded. It can be taken by @Cmjohnson anytime.
Please power it back on when done and comment here so I can repool

Thanks!

Maintenance_bot removed a project: Patch-For-Review.May 30 2019, 1:10 PM
Stashbot added a comment.May 30 2019, 3:26 PM

Mentioned in SAL (#wikimedia-operations) [2019-05-30T15:26:54Z] <cmjohnson1> shutting down db1099 to swap DIMM T221502

• Cmjohnson added a comment.May 30 2019, 3:42 PM

Swapped DIMM A5 with DIMM B5 and cleared the racadm log.

• Marostegui added a comment.May 30 2019, 3:44 PM

Thanks - I will take it from here

• Marostegui claimed this task.May 30 2019, 3:46 PM

mysql started and replication catching up

gerritbot added a comment.May 30 2019, 3:49 PM

Change 513304 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1099

https://gerrit.wikimedia.org/r/513304

gerritbot added a project: Patch-For-Review.May 30 2019, 3:49 PM
gerritbot added a comment.May 31 2019, 7:46 AM

Change 513304 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1099

https://gerrit.wikimedia.org/r/513304

Maintenance_bot removed a project: Patch-For-Review.May 31 2019, 8:10 AM
jcrespo added a comment.May 31 2019, 8:43 AM

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/513561 is pending.

• Marostegui closed this task as Resolved.May 31 2019, 2:58 PM

Server has been fully repooled by Jaime by pushing T221502#5225565

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL