After hard resetting it and checking how it boots:

Enumerating Boot options...
Enumerating Boot options... Done

UEFI0067: A PCIe link training failure is observed in Embedded Network Device
and the link is disabled.
Do one of the following: 1) Turn off the input power to the system and turn on
again. 2) Update the PCIe device firmware. If the issue persists, contact your
service provider.



Available Actions:
F1 to Continue and Retry Boot Order
F2 for System Setup (BIOS)
F10 for LifeCycle Controller
- Enable/Configure iDRAC
- Update or Backup/Restore Server Firmware
- Help Install an Operating System
F11 for Boot Manager

Change 437194 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Depool pc1005, hardware issues

https://gerrit.wikimedia.org/r/437194

Change 437194 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Depool pc2005, hardware issues

https://gerrit.wikimedia.org/r/437194

@Papaul Please try any trivial thing you want, but these hosts are under warranty/support, and hw provider should take care of any hw issues.

Will be receiving a main board replacement .

Enterprise Service Request 966671345

Good morning Papaul,

I set up a dispatch for a replacement motherboard and daughter card set for parts only as requested.

I would recommend replacing the daughter card first then testing, if issue persists replace the motherboard.

Let me know if you have any questions or concerns.

Jackie Barnard

Senior Analyst, Enterprise Tech Support

Linux & Virtualization Technology

Dell EMC | Enterprise Support Services

Office + 1 800 456 3355 x4341804

My work schedule is 8:00am – 5:00pm Monday – Friday

yay! for the replacements, sorry to create you more work.

@jcrespo you didn't create me more work, it is my work lol

I got the server back with network up. We will take it from here. Thanks a lot @Papaul

So, MySQL is up, but unfortunately the master's binlog where pc2005 was replicating from is gone

I guess as this data is just be erased when the number of days expires, we can probably start replication from the current position?

I guess as this data is just be erased when the number of days expires, we can probably start replication from the current position?

Yes, or replicating from the first pc1005 position available- so no purges are lost. all actions should be idempotent (not to the internal state, to replication working). This was perpared so that they can run in a master-master way.

If you run it from the last position, we will (maybe, not sure if that is row or statement) have to manually purge older events.

But in general, consistency is not an issue.

Sounds like a plan, I will start replication from the first available position

Replication has started. It is not ROW:

+------------+
| @@hostname |
+------------+
| pc1005     |
+------------+
+---------------+-----------+
| Variable_name | Value     |
+---------------+-----------+
| binlog_format | STATEMENT |
+---------------+-----------+

pc2005 caught up - I will wait until Monday to make sure it is estable before repooling it and closing this ticket

Mentioned in SAL (#wikimedia-operations) [2018-06-11T05:17:33Z] <marostegui> Stop MySQL and reboot pc2005 for intel-microcode update and final HW check - T196339

Mentioned in SAL (#wikimedia-operations) [2018-06-11T05:28:24Z] <marostegui@deploy1001> Synchronized wmf-config/db-codfw.php: Repool pc2005 - T196339 (duration: 00m 52s)