Page MenuHomePhabricator

apply events killing long running queries to db1070; any other production server
Closed, ResolvedPublic

Details

Event Timeline

jcrespo created this task.Oct 20 2016, 8:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2016, 8:20 PM
jcrespo triaged this task as Unbreak Now! priority.Oct 20 2016, 8:21 PM
Restricted Application added subscribers: Jay8g, Luke081515, TerraCodes. · View Herald TranscriptOct 20 2016, 8:21 PM
jcrespo claimed this task.Oct 21 2016, 8:18 AM
jcrespo moved this task from Triage to In progress on the DBA board.

Applied to db1070- the reason why it was not active is because it was a former master; so it had the configuration for a normal master.

Trying to detect other potentially problematic servers:

root@neodymium:~$ sudo salt -C 'G@mysql_role:slave and G@site:eqiad and G@mysql_group:core' cmd.run 'mysql --defaults-file=/root/.my.cnf --batch --skip-column-names -e "SHOW EVENTS FROM ops"'
db1071.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978778	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978778	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978778	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978778	utf8	utf8_general_ci	binary
db1042.eqiad.wmnet:
db1086.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101622	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101622	utf8	utf8_general_ci	binary
db1070.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2016-10-21 00:00:01NULL	ENABLED	171978777	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2016-10-21 00:00:00	NULL	ENABLED	171978777	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2016-10-21 00:00:05	NULL	ENABLED	171978777	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2016-10-21 00:00:03NULL	ENABLED	171978777	utf8	utf8_general_ci	binary
db1077.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
db1094.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101622	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101622	utf8	utf8_general_ci	binary
db1081.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101629	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101629	utf8	utf8_general_ci	binary
db1090.eqiad.wmnet:
    ops	wmf_master_purge	root@208.80.154.151	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@208.80.154.151	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
db1084.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101629	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101629	utf8	utf8_general_ci	binary
db1019.eqiad.wmnet:
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory")
db1082.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	103228	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	103228	utf8	utf8_general_ci	binary
db1092.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	103228	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	103228	utf8	utf8_general_ci	binary
db1089.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171974771	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171974771	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171974771	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171974771	utf8	utf8_general_ci	binary
db1078.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970576	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970576	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970576	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970576	utf8	utf8_general_ci	binary
db1093.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101612	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101612	utf8	utf8_general_ci	binary
db1074.eqiad.wmnet:
    ops	wmf_master_purge	root@208.80.154.151	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@208.80.154.151	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
db1083.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970661	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970661	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970661	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970661	utf8	utf8_general_ci	binary
db1079.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101622	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101622	utf8	utf8_general_ci	binary
db1076.eqiad.wmnet:
    ops	wmf_master_purge	root@208.80.154.151	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@208.80.154.151	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
db1080.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171966556	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171966556	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171966556	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171966556	utf8	utf8_general_ci	binary
db1064.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
db1087.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	103228	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	103228	utf8	utf8_general_ci	binary
db1088.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101612	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101612	utf8	utf8_general_ci	binary
db1073.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978780	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978780	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978780	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978780	utf8	utf8_general_ci	binary
db1091.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101629	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101629	utf8	utf8_general_ci	binary
db1053.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978771	utf8	utf8_general_ci	binary
db1085.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101612	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101612	utf8	utf8_general_ci	binary
db1068.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978775	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978775	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978775	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978775	utf8	utf8_general_ci	binary
db1021.eqiad.wmnet:
db1059.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171974685	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171974685	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171974685	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171974685	utf8	utf8_general_ci	binary
db1060.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171974686	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171974686	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171974686	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171974686	utf8	utf8_general_ci	binary
db1067.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978774	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978774	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978774	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978774	utf8	utf8_general_ci	binary
db1061.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978766	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978766	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978766	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978766	utf8	utf8_general_ci	binary
db1038.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101627	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101627	utf8	utf8_general_ci	binary
db1063.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978768	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978768	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978768	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978768	utf8	utf8_general_ci	binary
db1062.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978767	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978767	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978767	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978767	utf8	utf8_general_ci	binary
db1028.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970577	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970577	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970577	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970577	utf8	utf8_general_ci	binary
db1036.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970585	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970585	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970585	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970585	utf8	utf8_general_ci	binary
db1072.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978779	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978779	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978779	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978779	utf8	utf8_general_ci	binary
db1054.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171974680	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171974680	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171974680	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171974680	utf8	utf8_general_ci	binary
db1029.eqiad.wmnet:
db1026.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970575	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970575	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970575	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970575	utf8	utf8_general_ci	binary
db1056.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171974682	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171974682	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171974682	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171974682	utf8	utf8_general_ci	binary
db1052.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	103222	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	103222	utf8	utf8_general_ci	binary
es1013.eqiad.wmnet:
db1066.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978773	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978773	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978773	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978773	utf8	utf8_general_ci	binary
db1024.eqiad.wmnet:
    ops	wmf_master_purge	root@208.80.154.151	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@208.80.154.151	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101613	utf8	utf8_general_ci	binary
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970573	utf8	utf8_general_ci	binary
db1044.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970593	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970593	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970593	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970593	utf8	utf8_general_ci	binary
es1011.eqiad.wmnet:
es1014.eqiad.wmnet:
es1017.eqiad.wmnet:
db1015.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171966483	utf8	utf8_general_ci	binary
db1039.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970588	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970588	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970588	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970588	utf8	utf8_general_ci	binary
db1055.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171974681	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171974681	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171974681	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171974681	utf8	utf8_general_ci	binary
db1022.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970571	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970571	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970571	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970571	utf8	utf8_general_ci	binary
db1045.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970594	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970594	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970594	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970594	utf8	utf8_general_ci	binary
db1035.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970584	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970584	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970584	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970584	utf8	utf8_general_ci	binary
db1023.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101612	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101612	utf8	utf8_general_ci	binary
db1033.eqiad.wmnet:
    ops	wmf_master_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-26 00:00:00NULL	ENABLED	101622	utf8	utf8_general_ci	binary
    ops	wmf_master_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-26 00:00:05	NULL	ENABLED	101622	utf8	utf8_general_ci	binary
db1037.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970586	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970586	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970586	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970586	utf8	utf8_general_ci	binary
db1051.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171974677	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171974677	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171974677	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171974677	utf8	utf8_general_ci	binary
db1034.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970583	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970583	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970583	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970583	utf8	utf8_general_ci	binary
db1030.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171970579	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171970579	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171970579	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171970579	utf8	utf8_general_ci	binary
db1065.eqiad.wmnet:
    ops	wmf_slave_overload	root@localhost	SYSTEM	RECURRING	NULL	10	SECOND	2014-08-13 00:00:01NULL	ENABLED	171978772	utf8	utf8_general_ci	binary
    ops	wmf_slave_purge	root@localhost	SYSTEM	RECURRING	NULL	15	MINUTE	2014-08-13 00:00:00	NULL	ENABLED	171978772	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_sleep	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:05	NULL	ENABLED	171978772	utf8	utf8_general_ci	binary
    ops	wmf_slave_wikiuser_slow	root@localhost	SYSTEM	RECURRING	NULL	30	SECOND	2014-08-13 00:00:03NULL	ENABLED	171978772	utf8	utf8_general_ci	binary

Change 317122 had a related patch set uploaded (by Jcrespo):
dbtools: binarly log and drop fixes on the events

https://gerrit.wikimedia.org/r/317122

Change 317122 merged by Jcrespo:
dbtools: binary log and drop fixes on the events

https://gerrit.wikimedia.org/r/317122

Pending hosts:

  • es hosts- it will not be applied
  • Old masters: needs to change its events to slaves
  • New masters: needs to change its events to masters
  • db1019.eqiad.wmnet: it is decomissioned- needs to reset its salt grains
  • Many of the >db1073, which were cloned from old masters
  • In addition, db1029.eqiad.wmnet and db1021.eqiad.wmnet

Change 317123 had a related patch set uploaded (by Jcrespo):
dbtools: disable binary log for all events (not only on creation)

https://gerrit.wikimedia.org/r/317123

Change 317124 had a related patch set uploaded (by Jcrespo):
dbtools: explicitly set definer as root@localhost

https://gerrit.wikimedia.org/r/317124

Change 317123 merged by Jcrespo:
dbtools: disable binary log for all events (not only on creation)

https://gerrit.wikimedia.org/r/317123

Change 317124 merged by Jcrespo:
dbtools: explicitly set definer as root@localhost

https://gerrit.wikimedia.org/r/317124

Mentioned in SAL (#wikimedia-operations) [2016-10-21T09:13:31Z] <jynus> reviewing and applying new watchdog events to all core dbs T148790

jcrespo closed this task as Resolved.Oct 21 2016, 9:49 AM

I have applied master events to all masters, eqiad and codfw, slave events to all slaves (on the core s1-s7 shards). For x1, I have applied the slave ones to all, as all handle lots of slave connections. es shards should continue having no events.