Another big delete in the s51698__yetkin database is causing a replication lag
fnegri@cloudcumin1001:~$ sudo cookbook wmcs.toolforge.toolsdb.get_cluster_status
START - Cookbook wmcs.toolforge.toolsdb.get_cluster_status
{
"primary": {
"fqdn": "tools-db-1.tools.eqiad1.wikimedia.cloud",
"nodeid": "2886731301",
"replication_state": {
"status": "Running",
"replica_ids": [
"2886730873"
]
},
"host_status": "Up",
"mariadb_status": "Running"
},
"replicas": [
{
"fqdn": "tools-db-2.tools.eqiad1.wikimedia.cloud",
"nodeid": "2886730873",
"replication_state": {
"status": "Running",
"slave_io_state": "Waiting for master to send event",
"master_host": "tools-db-1.tools.eqiad1.wikimedia.cloud",
"master_log_file": "log.072204",
"read_master_log_pos": "68383356",
"relay_master_log_file": "log.072009",
"slave_io_running": "Yes",
"slave_sql_running": "Yes",
"replicate_wild_ignore_table": "s51698\\_\\_yetkin.wanted\\_items",
"last_error": "",
"last_errno": "0",
"exec_master_log_pos": "91952448",
"master_server_id": "2886731301",
"seconds_behind_master": "66237"
},
"host_status": "Up",
"mariadb_status": "Running"
}
]
}
END (PASS) - Cookbook wmcs.toolforge.toolsdb.get_cluster_status (exit_code=0)fnegri@cloudcumin1001:~$ sudo cookbook wmcs.toolforge.toolsdb.get_current_replica_transaction
START - Cookbook wmcs.toolforge.toolsdb.get_current_replica_transaction
Suspicious tables:
Table_map: `s51698__yetkin`.`visited_pages_agg` mapped
Suspicious queries:
#Q> DELETE FROM visited_pages_agg WHERE vpa_year = '2024' AND vpa_month = '2' AND vpa_day = '18'
END (PASS) - Cookbook wmcs.toolforge.toolsdb.get_current_replica_transaction (exit_code=0)