Page MenuHomePhabricator

[toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2024-02-20
Closed, ResolvedPublic

Description

Another big delete in the s51698__yetkin database is causing a replication lag

fnegri@cloudcumin1001:~$ sudo cookbook wmcs.toolforge.toolsdb.get_cluster_status
START - Cookbook wmcs.toolforge.toolsdb.get_cluster_status
{
    "primary": {
        "fqdn": "tools-db-1.tools.eqiad1.wikimedia.cloud",
        "nodeid": "2886731301",
        "replication_state": {
            "status": "Running",
            "replica_ids": [
                "2886730873"
            ]
        },
        "host_status": "Up",
        "mariadb_status": "Running"
    },
    "replicas": [
        {
            "fqdn": "tools-db-2.tools.eqiad1.wikimedia.cloud",
            "nodeid": "2886730873",
            "replication_state": {
                "status": "Running",
                "slave_io_state": "Waiting for master to send event",
                "master_host": "tools-db-1.tools.eqiad1.wikimedia.cloud",
                "master_log_file": "log.072204",
                "read_master_log_pos": "68383356",
                "relay_master_log_file": "log.072009",
                "slave_io_running": "Yes",
                "slave_sql_running": "Yes",
                "replicate_wild_ignore_table": "s51698\\_\\_yetkin.wanted\\_items",
                "last_error": "",
                "last_errno": "0",
                "exec_master_log_pos": "91952448",
                "master_server_id": "2886731301",
                "seconds_behind_master": "66237"
            },
            "host_status": "Up",
            "mariadb_status": "Running"
        }
    ]
}
END (PASS) - Cookbook wmcs.toolforge.toolsdb.get_cluster_status (exit_code=0)
fnegri@cloudcumin1001:~$ sudo cookbook wmcs.toolforge.toolsdb.get_current_replica_transaction
START - Cookbook wmcs.toolforge.toolsdb.get_current_replica_transaction
Suspicious tables:
    Table_map: `s51698__yetkin`.`visited_pages_agg` mapped
Suspicious queries:
    #Q> DELETE FROM visited_pages_agg WHERE vpa_year = '2024' AND vpa_month = '2' AND vpa_day = '18'
END (PASS) - Cookbook wmcs.toolforge.toolsdb.get_current_replica_transaction (exit_code=0)