Page MenuHomePhabricator

Clean up heartbeat table on clouddb hosts
Closed, ResolvedPublic

Description

Coming from: T272523#6794131
We need to clean up those old entries and just leave the concrete shard for that instance.

root@clouddb1013.eqiad.wmnet[heartbeat]> select * from heartbeat;
+----------------------------+-----------+-------------------+------------+-----------------------+-------------------
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_po
+----------------------------+-----------+-------------------+------------+-----------------------+-------------------
| 2012-08-22T23:48:33.001080 |     10645 | db35-bin.000112   |  489937000 | NULL                  |                NUL
| 2013-04-16T19:58:27.000830 |     10655 | db45-bin.000563   |  587023799 | db1039-bin.000567     |           56809380
| 2013-04-17T19:16:14.000660 |    101628 | db1039-bin.000572 |  650180541 | NULL                  |                NUL
| 2016-04-19T14:36:37.000700 |    103228 | db1058-bin.004408 |  510114119 | db2023-bin.002385     |           26314121
| 2018-07-18T06:01:44.001480 | 171970637 | db1052-bin.005945 |  479413784 | NULL                  |                NUL
| 2021-02-02T10:21:32.001480 | 171970661 | db1083-bin.006268 |  351669379 | NULL                  |                NUL
| 2017-04-26T09:51:46.001190 | 171970704 | db1049-bin.004713 |  788123851 | db2023-bin.004329     |           88114806
| 2019-11-14T06:01:21.000750 | 171974720 | db1067-bin.003024 |  445994072 | NULL                  |                NUL
| 2017-11-16T17:15:23.000810 | 171974884 | db1063-bin.001382 |  234441239 | NULL                  |                NUL
| 2017-04-28T19:30:15.000990 | 171978768 | db1063-bin.000042 | 1002606543 | db2023-bin.004338     |           78476004
| 2018-03-19T08:11:36.000620 | 171978777 | db1070-bin.001815 |  432216346 | NULL                  |                NUL
| 2018-10-22T07:53:05.002070 | 180355171 | db2048-bin.004286 |  172403944 | db1067-bin.001687     |           37167004
| 2017-05-09T11:45:28.001080 | 180359179 | db2023-bin.004393 |  358980382 | db1063-bin.000097     |           89978013
| 2020-10-29T08:02:21.000720 | 180363268 | db2112-bin.002375 |  568497678 | db1083-bin.005621     |            3405317
+----------------------------+-----------+-------------------+------------+-----------------------+-------------------
14 rows in set (0.001 sec)

root@clouddb1013.eqiad.wmnet[heartbeat]> use heartbeat_p
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
root@clouddb1013.eqiad.wmnet[heartbeat_p]> select * from heartbeat;
+-------+----------------------------+----------------+
| shard | last_updated               | lag            |
+-------+----------------------------+----------------+
| NULL  | 2013-04-17T19:16:14.000660 | 246035142.9993 |
| s1    | 2021-02-02T10:21:57.001170 |         0.0000 |
| s5    | 2018-03-19T08:11:36.000620 |  90814220.9994 |
+-------+----------------------------+----------------+
3 rows in set (0.001 sec)

Hosts to clean:

  • clouddb1013:3311
  • clouddb1013:3313
  • clouddb1014:3312
  • clouddb1014:3317
  • clouddb1015:3314
  • clouddb1015:3316
  • clouddb1016:3315
  • clouddb1016:3318
  • clouddb1017:3311
  • clouddb1017:3313
  • clouddb1018:3312
  • clouddb1018:3317
  • clouddb1019:3314
  • clouddb1019:3316
  • clouddb1020:3315
  • clouddb1020:3318

Event Timeline

Marostegui triaged this task as Medium priority.Feb 2 2021, 10:23 AM
Marostegui moved this task from Triage to Ready on the DBA board.

I will try to get this done this week

s1 clouddb replicas cleaned:

mysqlroot@cumin1001:/home/marostegui# for i in clouddb1017:3311 clouddb1013:3311; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"
> done
clouddb1017:3311
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s1    | 2021-02-03T10:15:44.001030 | 0.0000 |
+-------+----------------------------+--------+
clouddb1013:3311
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s1    | 2021-02-03T10:15:44.001030 | 0.0000 |
+-------+----------------------------+--------+

s3 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1017:3313 clouddb1013:3313; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1017:3313
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s3    | 2021-02-03T10:18:09.001030 | 0.0000 |
+-------+----------------------------+--------+
clouddb1013:3313
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s3    | 2021-02-03T10:18:09.001030 | 0.0000 |
+-------+----------------------------+--------+

s2 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1014:3312 clouddb1018:3312; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1014:3312
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s2    | 2021-02-03T10:28:45.001090 | 0.0000 |
+-------+----------------------------+--------+
clouddb1018:3312
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s2    | 2021-02-03T10:28:45.001090 | 0.0000 |
+-------+----------------------------+--------+

s7 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1014:3317 clouddb1018:3317; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1014:3317
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s7    | 2021-02-03T10:48:52.001190 | 0.0000 |
+-------+----------------------------+--------+
clouddb1018:3317
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s7    | 2021-02-03T10:48:52.001190 | 0.0000 |
+-------+----------------------------+--------+

s6 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1015:3316 clouddb1019:3316; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1015:3316
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s6    | 2021-02-03T11:03:49.000900 | 0.0000 |
+-------+----------------------------+--------+
clouddb1019:3316
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s6    | 2021-02-03T11:03:49.000900 | 0.0000 |
+-------+----------------------------+--------+

s5 clouddb replicas cleaned:

`
root@cumin1001:/home/marostegui# for i in clouddb1016:3315 clouddb1020:3315; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1016:3315
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s5    | 2021-02-03T11:13:59.001210 | 0.0000 |
+-------+----------------------------+--------+
clouddb1020:3315
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s5    | 2021-02-03T11:14:00.001080 | 0.0000 |
+-------+----------------------------+--------+
Marostegui claimed this task.

s8 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1016:3318 clouddb1020:3318; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1016:3318
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s8    | 2021-02-03T11:26:06.000850 | 0.0000 |
+-------+----------------------------+--------+
clouddb1020:3318
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s8    | 2021-02-03T11:26:06.000850 | 0.0000 |
+-------+----------------------------+--------+

All hosts are now done.