Page MenuHomePhabricator
Create Task
Maniphest T426109

Reboot lvs1019 for memory self-healing
Closed, ResolvedPublic
Actions

Description

I came across this warning in lvs1019's iDRAC interface, set since Sun Apr 12 2026 04:32:43:

The memory health monitor feature has detected a degradation in the DIMM installed in DIMM_B2. Reboot system to initiate self-heal process.

I'm not seeing any kernel errors related to memory... Regardless, let's depool/reboot lvs1019 so it can self-heal.

Further reading: https://www.dell.com/support/kbdoc/en-us/000053203/what-is-ddr4-self-healing-on-dell-poweredge-servers-with-intel-xeon-scalable-processors

Dell recommends updating BIOS before rebooting:

update BIOS to the latest revision that includes many memory Self-healing capabilities and ongoing enhancements

Event Timeline

BCornwall created this task.May 12 2026, 6:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2026, 6:05 PM
BCornwall triaged this task as Medium priority.May 12 2026, 6:05 PM
BCornwall updated the task description. (Show Details)May 12 2026, 6:07 PM
BCornwall updated the task description. (Show Details)May 12 2026, 6:10 PM
ssingh moved this task from Backlog to Actively Servicing on the Traffic board.Fri, May 22, 3:55 PM
ssingh added a comment.Tue, May 26, 2:26 PM

Once we reboot for T426585, we can consider this resolved as well.

BCornwall added a comment.Tue, May 26, 10:20 PM

@ssingh The Dell docs mention updating the BIOS:

update BIOS to the latest revision that includes many memory Self-healing capabilities and ongoing enhancements

However, that would put our version ahead of the rest of the fleet. Is that acceptable or should I just keep it as-is?

ssingh added a comment.Wed, May 27, 1:29 PM
In T426109#11957496, @BCornwall wrote:

@ssingh The Dell docs mention updating the BIOS:

update BIOS to the latest revision that includes many memory Self-healing capabilities and ongoing enhancements

However, that would put our version ahead of the rest of the fleet. Is that acceptable or should I just keep it as-is?

I think it should be fine as a one-off I think? Also we will be doing away with these in Q1/Q2 as part of the eqiad LVS refresh, so they will be eventually upgraded.

Stashbot added a comment.Thu, May 28, 6:34 PM

Mentioned in SAL (#wikimedia-operations) [2026-05-28T18:34:42Z] <brett> Stopping pybal/puppet/downtiming lvs1019.eqiad.wmnet for reboot and BIOS update/memory self-healing - T426109

BCornwall closed this task as Resolved.Thu, May 28, 7:00 PM

BIOS updated to 2.27.0.

After a reboot:

The self-heal operation successfully completed at DIMM DIMM_B2.
A problem was detected during Power-On Self-Test (POST).
The self-heal operation successfully completed at DIMM DIMM_B2.

I rebooted a second time cold just in case of the scary message in the middle there but all seems well.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits