Event Timeline
Comment Actions
05:15:01 <+icinga-wm> RECOVERY - snapshot of s6 in eqiad on db1115 is OK: snapshot for s6 at eqiad taken less than 4 days ago and larger than 90 GB: Last one 2019-09-30 04:17:00 from db1139.eqiad.wmnet:3316 (503 GB) https://wikitech.wikimedia.org/wiki/MariaDB/Backups
Comment Actions
There is some amount of "self healing", but I will rerun manually some backups so not to miss the window. Thanks @jijiki for the report. These alerts are like RAID alerts- they are actionable and very worrying if on for a long time, but if they happen to go off during the weekend they can wait until the week for them to be corrected.
Will close when everything is fixed.
Comment Actions
root@db1115.eqiad.wmnet[zarcillo]> SELECT * FROM backups WHERE section in ('s6', 's7') ORDER BY id desc LIMIT 5;
+------+----------------------------------+----------+-------------------------+------------------------+----------+---------+---------------------+---------------------+--------------+
| id | name | status | source | host | type | section | start_date | end_date | total_size |
+------+----------------------------------+----------+-------------------------+------------------------+----------+---------+---------------------+---------------------+--------------+
| 2980 | snapshot.s7.2019-09-30--11-40-55 | finished | db1116.eqiad.wmnet:3317 | dbprov1002.eqiad.wmnet | snapshot | s7 | 2019-09-30 13:05:21 | 2019-09-30 14:15:43 | 927424383014 |
| 2977 | snapshot.s6.2019-09-30--03-30-02 | finished | db1139.eqiad.wmnet:3316 | dbprov1001.eqiad.wmnet | snapshot | s6 | 2019-09-30 04:17:00 | 2019-09-30 05:04:01 | 539671316158 |
| 2976 | snapshot.s7.2019-09-30--02-53-40 | finished | db1116.eqiad.wmnet:3317 | dbprov1002.eqiad.wmnet | snapshot | s7 | 2019-09-30 04:15:08 | 2019-09-30 06:37:21 | 926946625574 |
| 2973 | snapshot.s7.2019-09-30--00-24-45 | finished | db2100.codfw.wmnet:3317 | dbprov2002.codfw.wmnet | snapshot | s7 | 2019-09-30 01:45:05 | 2019-09-30 04:40:41 | 939256940768 |
| 2971 | snapshot.s6.2019-09-30--00-48-20 | finished | db2097.codfw.wmnet:3316 | dbprov2001.codfw.wmnet | snapshot | s6 | 2019-09-30 01:34:45 | 2019-09-30 02:24:45 | 543299025716 |
+------+----------------------------------+----------+-------------------------+------------------------+----------+---------+---------------------+---------------------+--------------+
5 rows in set (0.00 sec)