Change Details

db1074 went down: 05:18 <+icinga-wm> PROBLEM - Host db1074 is DOWN: PING CRITICAL - Packet loss = 100% This caused slave lag on db1125 <+icinga-wm> PROBLEM - MariaDB Slave IO: s2 on db1125 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, The console showed nothing, so ema powercycled it. Then jynus depooled it and pointed out this hits tendril which is annoying but does not break wikis. 06:10 <+logmsgbot> !log jynus@deploy1001 Synchronized wmf-config/db-eqiad.php: Mirror dbctl depool of db1074 (duration: 00m 55s) After the depool we saw recovery of the slave lage alert.

db1074 went down: 05:18 <+icinga-wm> PROBLEM - Host db1074 is DOWN: PING CRITICAL - Packet loss = 100% This caused slave lag on db1125 <+icinga-wm> PROBLEM - MariaDB Slave IO: s2 on db1125 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, The console showed nothing, so ema powercycled it. Then jynus depooled it and pointed out this hits tendril which is annoying but does not break wikis. 06:10 <+logmsgbot> !log jynus@deploy1001 Synchronized wmf-config/db-eqiad.php: Mirror dbctl depool of db1074 (duration: 00m 55s) After the depool we saw recovery of the slave lag alert. 06:12 <+icinga-wm> RECOVERY - snapshot of s3 in eqiad on db1115 is OK: snapshot for s3 at eqiad taken less than 4 days ago ..

db1074 went down: 05:18 <+icinga-wm> PROBLEM - Host db1074 is DOWN: PING CRITICAL - Packet loss = 100% This caused slave lag on db1125 <+icinga-wm> PROBLEM - MariaDB Slave IO: s2 on db1125 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, The console showed nothing, so ema powercycled it. Then jynus depooled it and pointed out this hits tendril which is annoying but does not break wikis. 06:10 <+logmsgbot> !log jynus@deploy1001 Synchronized wmf-config/db-eqiad.php: Mirror dbctl depool of db1074 (duration: 00m 55s) After the depool we saw recovery of the slave lage alert alert. 06:12 <+icinga-wm> RECOVERY - snapshot of s3 in eqiad on db1115 is OK: snapshot for s3 at eqiad taken less than 4 days ago ..