I've downtimed db1098 on Icinga until Wed mid EU day and disabled notifications.

This is the same error as db2081 earlier today: T193325

The Intel Management Engine has recovered the ability to utilize the PECI over DMI facility.

If the PWR2262 "internal system error" was logged within the previous 30 seconds, ignore the message. No response action required.

T175973#3615656 db1100 suffered it too which is the same batch as db1098

Change 429652 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] db1098.yaml: Disable notifications

https://gerrit.wikimedia.org/r/429652

Change 429652 merged by Marostegui:
[operations/puppet@production] db1098.yaml: Disable notifications

https://gerrit.wikimedia.org/r/429652

I have started MySQL on db1098 to:

• Make sure nothing is corrupted and replication can flow
• Avoid leaving the host to fall behind replication for 2 days.

I will stop MySQL before @Cmjohnson wakes up, so it is all ready for him to take over on Monday without waiting on us to stop MySQL again.

It will remain depooled till maintenance has happened on it.

As a side note. Either db1098 (T193331), db2081 (T193325) and db1100 (T175973) (they were all coming from the same batch of purchases (T162159 and T162233)) all crashed with the same errors.

PWR2262: The Intel Management Engine has reported an internal system error.
 2018-04-28T15:16:13-0500
Log Sequence Number: 154
Detailed Description:
The Intel Management Engine is unable to utilize the PECI over DMI facility.
Recommended Action:
Look for the PWR2264 "normal system operation" message in the Lifecycle Log after the PWR2262 entry. It may take 30 seconds for the message to be logged. If the PWR2264 message does not appear, do the following: Disconnect power from the server and reconnect power. Turn on the server. If the issue persists, contact your service provider.

db1100 never crashed again after doing the power drain.

Mentioned in SAL (#wikimedia-operations) [2018-04-30T13:26:04Z] <marostegui> Stop MySQL on db1098 - T193331

Mentioned in SAL (#wikimedia-operations) [2018-04-30T13:30:17Z] <marostegui> Poweroff db1098 for HW maintenance - T193331

drained flea power, updated bios and idrac f/w to and powered back on

BIOS Version 2.7.1
Firmware Version 2.52.52.52

Thank you @Cmjohnson
I have started MySQL to let it catch up and replicate for a day before repooling it.

MySQL and kernel have been upgraded too

This can be merged once we are ready to repool: https://gerrit.wikimedia.org/r/#/c/430032/

s6 main_tables.txt have been checked, no errors found, now checking s7 instance:

$ cat s7.dblist | while read db; do cat main_tables.txt | while read table index; do echo "$db.$table..."; ./compare.py $db $table $index db1062.eqiad.wmnet db1098.eqiad.wmnet:3317 db1090.eqiad.wmnet:3317 || break; done; done
arwiki.archive...
Starting comparison between id 1 and 3593418
2018-05-02T10:38:08.069550: row id 990001/3593418, ETA: 00m31s, 0 chunk(s) found different
2018-05-02T10:38:19.449771: row id 1990001/3593418, ETA: 00m18s, 0 chunk(s) found different
2018-05-02T10:38:29.727766: row id 2990001/3593418, ETA: 00m06s, 0 chunk(s) found different
Execution ended, no differences found.
arwiki.logging...

Change 430391 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/mediawiki-config@master] mariadb: Repool db1098 after being checked for consistancy issues

https://gerrit.wikimedia.org/r/430391

Change 430391 merged by jenkins-bot:
[operations/mediawiki-config@master] mariadb: Repool db1098 after being checked for consistancy issues

https://gerrit.wikimedia.org/r/430391

No consistency issues found on s6 and s7, repooling.

@RobH the specific incident for this host has been taken care, should we centralize the recurring issue into a separate task? If yes, I would close this as resolved for now.

I vote for closing this and if it happens again on any other host - open a general task and a case with the vendor.

db1100 crashed half a year ago and never again after doing the upgrade (which I still think it is a consequence).