Memory errors on clouddb1019
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Marostegui
	Jan 15 2021, 9:09 AM

Description

When trying to reboot clouddb1019, this host was getting stuck on boot, not sure if it is related, but I have seen this happening at the same time of the reboot (times do match):

-------------------------------------------------------------------------------
Record:      2
Date/Time:   01/15/2021 08:56:59
Source:      system
Severity:    Critical
Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A4.
-------------------------------------------------------------------------------

I am trying to issue a hardreset from the idrac, but it doesn't seem to be doing anything. Can we get this host looked at on-site? Maybe move the DIMM to another slot and check if the host boots and/or if the error moves to a different place?

Related Objects
Search...

Status	Assigned	Task
Resolved	Marostegui	T233766 labsdb1011 mariadb crashed
		Restricted Task
		Restricted Task
Open	None	T204950 Public Edit Data Lake: Mediawiki history snapshots available in SQL data store to cloud (labs) users
Open	None	T215858 Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema
Resolved	fnegri	T280152 Mitigate breaking changes from the new Wiki Replicas architecture
		Unknown Object (Task)
Resolved	RobH	T260441 (Need By: ASAP) rack/setup/install clouddb10[13-20]
Resolved	• Bstorm	T260389 Redesign and rebuild the wikireplicas service using a multi-instance architecture
Resolved	• Bstorm	T260843 Set up roles for new wiki replicas layout
Resolved	Marostegui	T265135 wikireplicas: Define MW sections per host
Resolved	Marostegui	T267090 Productionize clouddb10[13-20]
Stalled	None	T302086 Set scap minimum python version to 3.7
Resolved	None	T247045 Migrate all of production metal and VMs to Buster or later
Resolved	Marostegui	T250666 Upgrade WMF database-and-backup-related hosts to buster
		Unknown Object (Task)
Resolved	Marostegui	T258361 Productionize db1155-db1175 and refresh and decommission db1074-db1095 (22 servers)
Resolved	RobH	T267043 (Need By: 2020-11-29) rack/setup/install db11[51-76]
Resolved	Marostegui	T268742 Test upgrading sanitarium hosts to Buster + 10.4
Resolved	Marostegui	T272008 Move wikireplicas under the new sanitarium hosts (db1154, db1155)
		Restricted Task
		Restricted Task
Resolved	• Cmjohnson	T272125 Memory errors on clouddb1019
Resolved	dcaro	T272127 2021-01-15: PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL

Event Timeline

Marostegui created this task.Jan 15 2021, 9:09 AM

Restricted Application added a project: SRE. · View Herald TranscriptJan 15 2021, 9:09 AM

Setting to high as we are trying to finish up the new wiki replicas infra

Marostegui moved this task from Triage to In progress on the DBA board.Jan 15 2021, 9:12 AM

There's something going on with this host:

racadm>>serveraction powerstatus
Server power status: OFF
racadm>>serveraction powerup
Server power operation initiated successfully
racadm>>serveraction powerstatus
Server power status: ON
racadm>>serveraction powerstatus
Server power status: ON
racadm>>serveraction powerstatus
Server power status: OFF
racadm>>

aborrero added a subtask: T272127: 2021-01-15: ** PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL **.Jan 15 2021, 9:33 AM

And without doing anything again:

racadm>>serveraction powerstatus
Server power status: ON

Interestingly, I cannot see anything on the console, so I have no idea what it is doing and if it is rebooting or doing something else.

aborrero mentioned this in T272127: 2021-01-15: ** PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL **.Jan 15 2021, 9:36 AM

Mentioned in SAL (#wikimedia-cloud) [2021-01-15T09:47:36Z] <arturo> labstore1004 maintain-dbusers affected by T272127 and T272125

Mentioned in SAL (#wikimedia-cloud) [2021-01-15T13:41:57Z] <arturo> icinga downtime labstore1004 maintain-dbuser alert until 2021-01-19 (T272125)

taavi subscribed.Jan 15 2021, 1:44 PM

wiki_willy assigned this task to • Cmjohnson.Jan 15 2021, 4:03 PM

• Bstorm closed subtask T272127: 2021-01-15: ** PROBLEM alert - labstore1004/Ensure mysql credential creation for tools users is running is CRITICAL ** as Resolved.Jan 15 2021, 4:19 PM

Record: 1
Date/Time: 08/31/2020 17:37:02
Source: system
Severity: Ok

Description: Log cleared.

Record: 2
Date/Time: 01/15/2021 08:56:59
Source: system
Severity: Critical

Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_A4.

Record: 3
Date/Time: 01/20/2021 18:44:27
Source: system
Severity: Critical

Description: The chassis is open while the power is off.

Swapped DIMM A4 with DIMM B4, cleared the system log and powered on. Let's see if the error returns, stays the same or changes.

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jan 20 2021, 6:46 PM

Fast response by the server, after swapping the DIMM, the server was stuck in a continuous reboot. connected the console and see that the server is failing during post at the memory check. Not sure if it's both DIMM for B4 and A4 but I removed them both, used the DIMM from A9 and A10 to populate A4/B4 to keep the correct DIMM sequence. Server booted with no issues. A ticket will need to be created with Dell to have new DIMM sent.

RhinosF1 subscribed.Jan 20 2021, 7:06 PM

@Cmjohnson unfortunately the server isn't accessible yet - I cannot even reach its idrac :-(

root@cumin1001:~# ping clouddb1019.eqiad.wmnet -c5
PING clouddb1019.eqiad.wmnet (10.64.48.9) 56(84) bytes of data.

--- clouddb1019.eqiad.wmnet ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 87ms

root@cumin1001:~# ping clouddb1019.mgmt.eqiad.wmnet -c5
PING clouddb1019.mgmt.eqiad.wmnet (10.65.0.131) 56(84) bytes of data.

--- clouddb1019.mgmt.eqiad.wmnet ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 95ms

Marostegui mentioned this in T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).Jan 21 2021, 11:34 AM

Marostegui added a parent task: T272008: Move wikireplicas under the new sanitarium hosts (db1154, db1155).

Could we have an ETA on when this server can be worked on? We were aiming to open a new infrastructure this server is part of to users 1st of Feb

I am working on it, I am dependant on Dell. I do need to update all the f/w and idrac today.

I failed to re-connect the mgmt cable after getting it to power on and was not able to remotely access the server to get the logs for the Dell tech. I connected everything, updated the bios and submitting a task to Dell now.

Thanks Chris, any chances that we can get the host to boot up at least so MySQL replication can catch up a bit.
Thank you!

@Marostegui it should be accessible now

There is just let memory at the moment

Dell ticket number SR1049824647

Thanks Chris - I can now access the server and will start mysql so it can catch up on replication!. Let's coordinate to install the new memory once it arrives.
Thanks again

New DIMM has been dispatched for the server I will coordinate a time with you to power down to restore the original configuration.

Sounds good @Cmjohnson let me know when it arrives and you plan to change it so I can stop mysql
Thank you

Mentioned in SAL (#wikimedia-operations) [2021-01-28T15:49:08Z] <marostegui> Power off clouddb1019 for memory replacement T272125

This has been fixed.

DELL Return tracking #
USPS 9202394653012447257126

Thank you so much

Memory errors on clouddb1019Closed, ResolvedPublicActions