AFAICT from https://wikitech.wikimedia.org/wiki/Alertmanager alerts are (now?) meant to go into operations/alerts rather than into puppet directly. So I think we want a data-persistence routing for alerts, and then a suitable alert defined in operations/alerts (I've checked and mysql_exporter_last_scrape_error is in prometheus)

[I was pointed at this task from IRC, I'm new in data persistence team, used to do quite a bit of Ceph at the Sanger]

Marking this as resolved - as we deploy 10.4.21-2 everywhere, the fix will get rolled out.

I've also checked the stop/start/restart behaviour, which is as expected.
Also, that on reboot PME isn't started, but when you start mariadb, it does then get started for you.

Hi @Papaul this system seems to have had a hardware fault(s), and is (just) still within its warranty, could you get the hardware checked out, please? Thanks :)

On reboot, the disks came back, but many of the filesystems are unhappy:
mvernon@ms-be2045:~$ sudo dmesg | grep 'Shutting down filesystem'
[ 18.244602] XFS (sda3): Corruption of in-memory data detected. Shutting down filesystem
[ 18.724649] XFS (sdf1): Corruption of in-memory data detected. Shutting down filesystem
[ 19.448076] XFS (sdg1): Corruption of in-memory data detected. Shutting down filesystem
[ 20.610420] XFS (sdj1): I/O Error Detected. Shutting down filesystem
[ 20.745769] XFS (sdn1): I/O Error Detected. Shutting down filesystem
[ 20.938081] XFS (sdi1): I/O Error Detected. Shutting down filesystem
[ 23.719222] XFS (sdh1): I/O Error Detected. Shutting down filesystem
[ 24.802161] XFS (sde1): I/O Error Detected. Shutting down filesystem
[ 30.091057] XFS (sdm1): I/O Error Detected. Shutting down filesystem
[ 31.761276] XFS (sdc1): I/O Error Detected. Shutting down filesystem

This host is ready for DC-Ops to decommission

This host is ready for DC-Ops to decommission

This host is ready for DC-Ops to decommission

This host is ready for DC-Ops to decommission

This host is ready for DC-Ops to decommission

This host is ready for DC-Ops to decommission

This host is ready for DC-Ops to decommission

I think this could be achieved by setting a grafana alert on the MySQL Aggregated dashboard? But I don't really know much about how alerts are set up at WMF, or when one should use a Grafana alerts vs an icinga one, or...

I think we concluded that the mariadb.target idea isn't all that useful (since mostly folk don't stop and start >1 instance at once I think @Kormat said).

In T289488#7303806, @Marostegui wrote:

I know it is not totally related to this task, but maybe this can be also looked at as part of this? T257056: Add alert for prometheus-mysql-exporter failing to scrape mysql

I've split the systemd bits into a separate task - T289488

Another thought here - if we want the exporter to be automatically restarted if mysqld is restarted, then we should be able to get systemd to do this for us.