Page MenuHomePhabricator
Create Task
Maniphest T273593

Clean up heartbeat table on clouddb hosts
Closed, ResolvedPublic
Actions

Description

Coming from: T272523#6794131
We need to clean up those old entries and just leave the concrete shard for that instance.

root@clouddb1013.eqiad.wmnet[heartbeat]> select * from heartbeat;
+----------------------------+-----------+-------------------+------------+-----------------------+-------------------
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_po
+----------------------------+-----------+-------------------+------------+-----------------------+-------------------
| 2012-08-22T23:48:33.001080 |     10645 | db35-bin.000112   |  489937000 | NULL                  |                NUL
| 2013-04-16T19:58:27.000830 |     10655 | db45-bin.000563   |  587023799 | db1039-bin.000567     |           56809380
| 2013-04-17T19:16:14.000660 |    101628 | db1039-bin.000572 |  650180541 | NULL                  |                NUL
| 2016-04-19T14:36:37.000700 |    103228 | db1058-bin.004408 |  510114119 | db2023-bin.002385     |           26314121
| 2018-07-18T06:01:44.001480 | 171970637 | db1052-bin.005945 |  479413784 | NULL                  |                NUL
| 2021-02-02T10:21:32.001480 | 171970661 | db1083-bin.006268 |  351669379 | NULL                  |                NUL
| 2017-04-26T09:51:46.001190 | 171970704 | db1049-bin.004713 |  788123851 | db2023-bin.004329     |           88114806
| 2019-11-14T06:01:21.000750 | 171974720 | db1067-bin.003024 |  445994072 | NULL                  |                NUL
| 2017-11-16T17:15:23.000810 | 171974884 | db1063-bin.001382 |  234441239 | NULL                  |                NUL
| 2017-04-28T19:30:15.000990 | 171978768 | db1063-bin.000042 | 1002606543 | db2023-bin.004338     |           78476004
| 2018-03-19T08:11:36.000620 | 171978777 | db1070-bin.001815 |  432216346 | NULL                  |                NUL
| 2018-10-22T07:53:05.002070 | 180355171 | db2048-bin.004286 |  172403944 | db1067-bin.001687     |           37167004
| 2017-05-09T11:45:28.001080 | 180359179 | db2023-bin.004393 |  358980382 | db1063-bin.000097     |           89978013
| 2020-10-29T08:02:21.000720 | 180363268 | db2112-bin.002375 |  568497678 | db1083-bin.005621     |            3405317
+----------------------------+-----------+-------------------+------------+-----------------------+-------------------
14 rows in set (0.001 sec)

root@clouddb1013.eqiad.wmnet[heartbeat]> use heartbeat_p
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
root@clouddb1013.eqiad.wmnet[heartbeat_p]> select * from heartbeat;
+-------+----------------------------+----------------+
| shard | last_updated               | lag            |
+-------+----------------------------+----------------+
| NULL  | 2013-04-17T19:16:14.000660 | 246035142.9993 |
| s1    | 2021-02-02T10:21:57.001170 |         0.0000 |
| s5    | 2018-03-19T08:11:36.000620 |  90814220.9994 |
+-------+----------------------------+----------------+
3 rows in set (0.001 sec)

Hosts to clean:

  • clouddb1013:3311
  • clouddb1013:3313
  • clouddb1014:3312
  • clouddb1014:3317
  • clouddb1015:3314
  • clouddb1015:3316
  • clouddb1016:3315
  • clouddb1016:3318
  • clouddb1017:3311
  • clouddb1017:3313
  • clouddb1018:3312
  • clouddb1018:3317
  • clouddb1019:3314
  • clouddb1019:3316
  • clouddb1020:3315
  • clouddb1020:3318

Related Objects
Search...

Event Timeline

Marostegui created this task.Feb 2 2021, 10:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 2 2021, 10:22 AM
Marostegui triaged this task as Medium priority.Feb 2 2021, 10:23 AM
Marostegui moved this task from Triage to Ready on the DBA board.

I will try to get this done this week

Marostegui updated the task description. (Show Details)Feb 2 2021, 10:24 AM
Marostegui added a parent task: T272523: Early testing of the new Wiki Replicas multi-instance architecture.
LSobanski subscribed.Feb 2 2021, 10:27 AM
Marostegui updated the task description. (Show Details)Feb 2 2021, 10:33 AM
Marostegui added a comment.Feb 3 2021, 10:15 AM

s1 clouddb replicas cleaned:

mysqlroot@cumin1001:/home/marostegui# for i in clouddb1017:3311 clouddb1013:3311; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"
> done
clouddb1017:3311
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s1    | 2021-02-03T10:15:44.001030 | 0.0000 |
+-------+----------------------------+--------+
clouddb1013:3311
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s1    | 2021-02-03T10:15:44.001030 | 0.0000 |
+-------+----------------------------+--------+
Marostegui updated the task description. (Show Details)Feb 3 2021, 10:16 AM
Marostegui added a comment.Feb 3 2021, 10:26 AM

s3 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1017:3313 clouddb1013:3313; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1017:3313
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s3    | 2021-02-03T10:18:09.001030 | 0.0000 |
+-------+----------------------------+--------+
clouddb1013:3313
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s3    | 2021-02-03T10:18:09.001030 | 0.0000 |
+-------+----------------------------+--------+
Marostegui updated the task description. (Show Details)Feb 3 2021, 10:27 AM
Marostegui added a comment.Feb 3 2021, 10:46 AM

s2 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1014:3312 clouddb1018:3312; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1014:3312
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s2    | 2021-02-03T10:28:45.001090 | 0.0000 |
+-------+----------------------------+--------+
clouddb1018:3312
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s2    | 2021-02-03T10:28:45.001090 | 0.0000 |
+-------+----------------------------+--------+
Marostegui updated the task description. (Show Details)Feb 3 2021, 10:46 AM
Marostegui added a comment.Feb 3 2021, 11:02 AM

s7 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1014:3317 clouddb1018:3317; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1014:3317
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s7    | 2021-02-03T10:48:52.001190 | 0.0000 |
+-------+----------------------------+--------+
clouddb1018:3317
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s7    | 2021-02-03T10:48:52.001190 | 0.0000 |
+-------+----------------------------+--------+
Marostegui updated the task description. (Show Details)Feb 3 2021, 11:02 AM
Marostegui added a comment.Feb 3 2021, 11:11 AM

s6 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1015:3316 clouddb1019:3316; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1015:3316
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s6    | 2021-02-03T11:03:49.000900 | 0.0000 |
+-------+----------------------------+--------+
clouddb1019:3316
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s6    | 2021-02-03T11:03:49.000900 | 0.0000 |
+-------+----------------------------+--------+
Marostegui updated the task description. (Show Details)Feb 3 2021, 11:12 AM
Marostegui added a comment.Feb 3 2021, 11:24 AM

s5 clouddb replicas cleaned:

`
root@cumin1001:/home/marostegui# for i in clouddb1016:3315 clouddb1020:3315; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1016:3315
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s5    | 2021-02-03T11:13:59.001210 | 0.0000 |
+-------+----------------------------+--------+
clouddb1020:3315
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s5    | 2021-02-03T11:14:00.001080 | 0.0000 |
+-------+----------------------------+--------+
Marostegui updated the task description. (Show Details)Feb 3 2021, 11:25 AM
Marostegui closed this task as Resolved.Feb 3 2021, 11:35 AM
Marostegui claimed this task.

s8 clouddb replicas cleaned:

root@cumin1001:/home/marostegui# for i in clouddb1016:3318 clouddb1020:3318; do echo $i; mysql.py -h$i heartbeat_p -e "select * from heartbeat;"; done
clouddb1016:3318
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s8    | 2021-02-03T11:26:06.000850 | 0.0000 |
+-------+----------------------------+--------+
clouddb1020:3318
+-------+----------------------------+--------+
| shard | last_updated               | lag    |
+-------+----------------------------+--------+
| s8    | 2021-02-03T11:26:06.000850 | 0.0000 |
+-------+----------------------------+--------+

All hosts are now done.

Marostegui updated the task description. (Show Details)Feb 3 2021, 11:35 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits