While doing unrelated work onsite at ulsfo, I noticed that cp4032 had a memory error on the LCD. I've filed T183176 to fix that particular server's hardware.
However, these errors should show up in icinga, and not rely for onsite visits to notice the issue, or for system crashes.
At the time of the memory error on cp4032, icinga showed that system as all green. System is still online, but with one of its dimm slots disabled, which should show some non-optimal status on icinga.