Page MenuHomePhabricator
Paste P13346

(An Untitled Masterwork)
ActivePublic

Authored by Kormat on Nov 20 2020, 10:31 AM.
Tags
None
Referenced Files
F33920281: raw-paste-data.txt
Nov 20 2020, 10:31 AM
Subscribers
None
root@db2093.codfw.wmnet[heartbeat]> SELECT ts FROM heartbeat.heartbeat ORDER BY ts DESC LIMIT 1;
+----------------------------+
| ts |
+----------------------------+
| 2020-11-20T10:31:30.000710 |
+----------------------------+
1 row in set (0.034 sec)
root@db2093.codfw.wmnet[heartbeat]> describe heartbeat;
+-----------------------+---------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-----------------------+---------------------+------+-----+---------+-------+
| ts | varbinary(26) | NO | | NULL | |
| server_id | int(10) unsigned | NO | PRI | NULL | |
| file | varbinary(255) | YES | | NULL | |
| position | bigint(20) unsigned | YES | | NULL | |
| relay_master_log_file | varbinary(255) | YES | | NULL | |
| exec_master_log_pos | bigint(20) unsigned | YES | | NULL | |
| shard | varbinary(30) | YES | | NULL | |
| datacenter | binary(5) | YES | | NULL | |
+-----------------------+---------------------+------+-----+---------+-------+
8 rows in set (0.035 sec)

Event Timeline

root@db2093.codfw.wmnet[heartbeat]> SELECT TIMEDIFF(ts,NOW()) FROM heartbeat.heartbeat ORDER BY ts DESC LIMIT 1;
+--------------------+
| TIMEDIFF(ts,NOW()) |
+--------------------+
| 00:00:00.000880    |
+--------------------+
1 row in set (0.034 sec)
root@db1139.eqiad.wmnet[(none)]> SELECT NOW(),ts,TIMEDIFF(NOW(),ts) FROM heartbeat.heartbeat ORDER BY ts DESC LIMIT 1;
+---------------------+----------------------------+--------------------+
| NOW()               | ts                         | TIMEDIFF(NOW(),ts) |
+---------------------+----------------------------+--------------------+
| 2020-11-20 10:44:16 | 2020-11-19T02:54:04.001120 | 31:50:11.998880    |
+---------------------+----------------------------+--------------------+
root@pc2010.codfw.wmnet[(none)]> SELECT NOW(),server_id,ts,TIMEDIFF(NOW(),ts) FROM heartbeat.heartbeat WHERE server_id!=@@server_id ORDER BY ts DESC LIMIT 1;
+---------------------+-----------+----------------------------+--------------------+
| NOW()               | server_id | ts                         | TIMEDIFF(NOW(),ts) |
+---------------------+-----------+----------------------------+--------------------+
| 2020-11-20 10:50:20 | 171966644 | 2020-11-20T10:49:57.000820 | 00:00:22.999180    |
+---------------------+-----------+----------------------------+--------------------+
1 row in set (0.034 sec)
root@db2094.codfw.wmnet[(none)]> select * from heartbeat.heartbeat;
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_pos | shard | datacen
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------
| 2018-04-25T13:12:01.000970 | 171966669 | db1075-bin.002735 |  145906585 | NULL                  |                NULL | s3    | eqiad  
| 2017-04-20T16:15:01.000960 | 171970589 | db1040-bin.002987 |  587360761 | db2019-bin.002532     |           773446073 | s4    | eqiad  
| 2018-07-18T06:01:44.001480 | 171970637 | db1052-bin.005945 |  479413784 | NULL                  |                NULL | s1    | eqiad  
| 2020-11-20T12:53:24.001090 | 171970661 | db1083-bin.005754 |  610841608 | NULL                  |                NULL | s1    | eqiad  
| 2017-01-26T08:16:47.000790 | 171974683 | db1057-bin.002885 |  576316184 | db1052-bin.004556     |           210334207 | s1    | eqiad  
| 2019-11-14T06:01:21.000750 | 171974720 | db1067-bin.003024 |  445994072 | NULL                  |                NULL | s1    | eqiad  
| 2017-11-16T17:15:23.000810 | 171974884 | db1063-bin.001382 |  234441239 | NULL                  |                NULL | s5    | eqiad  
| 2017-08-22T07:34:57.000440 | 171978775 | db1068-bin.000516 |  439889287 | NULL                  |                NULL | s4    | eqiad  
| 2018-04-25T13:12:01.000630 | 171978777 | db1070-bin.001894 |   46969145 | NULL                  |                NULL | s5    | eqiad  
| 2018-04-25T13:12:01.000630 | 171978778 | db1071-bin.006549 |  147776760 | NULL                  |                NULL | s8    | eqiad  
| 2019-09-11T05:33:39.001920 | 180355171 | db2048-bin.005384 |  980454332 | db1067-bin.002779     |           346367189 | s1    | codfw  
| 2017-05-10T13:29:31.001050 | 180359172 | db2016-bin.002977 |  503605243 | db1052-bin.004793     |           401461576 | s1    | codfw  
| 2017-05-16T06:34:43.000930 | 180359174 | db2018-bin.003211 | 1032717323 | db1075-bin.001439     |           632039344 | s3    | codfw  
| 2017-05-09T09:24:39.001160 | 180359175 | db2019-bin.002580 |  364384048 | db1068-bin.000049     |           373607821 | s4    | codfw  
|                            | 180359179 | NULL              |       NULL | NULL                  |                NULL | NULL  | NULL   
| 2020-11-20T12:53:24.000700 | 180363268 | db2112-bin.002509 |  488527459 | db1083-bin.005754     |           610841608 | s1    | codfw  
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+--------