Page MenuHomePhabricator
Paste P12559

(An Untitled Masterwork)
ActivePublic

Authored by Marostegui on Thu, Sep 10, 1:50 PM.
root@labsdb1012.eqiad.wmnet[heartbeat]> SELECT shard, greatest(0, TIMESTAMPDIFF(MICROSECOND, max(ts), UTC_TIMESTAMP(6)) - 500000)/1000000 AS lag FROM heartbeat.heartbeat WHERE datacenter='eqiad' GROUP BY shard HAVING shard like 's%'
-> ;
+-------+--------+
| shard | lag |
+-------+--------+
| s1 | 0.0000 |
| s2 | 0.0000 |
| s3 | 0.0000 |
| s4 | 0.0000 |
| s5 | 0.0000 |
| s6 | 0.0000 |
| s7 | 0.0000 |
| s8 | 0.0000 |
+-------+--------+
8 rows in set (0.001 sec)
root@labsdb1012.eqiad.wmnet[heartbeat]> SELECT shard, greatest(0, TIMESTAMPDIFF(MICROSECOND, max(ts), UTC_TIMESTAMP(6)) - 500000)/1000000 AS lag FROM heartbeat.heartbeat WHERE datacenter='codfw' GROUP BY shard HAVING shard like 's%'
-> ;
+-------+-------------+
| shard | lag |
+-------+-------------+
| s1 | 277237.2624 |
| s2 | 0.2624 |
| s3 | 7.2623 |
| s4 | 0.2623 |
| s5 | 0.2624 |
| s6 | 0.2624 |
| s7 | 0.2622 |
| s8 | 278333.2622 |
+-------+-------------+
8 rows in set (0.001 sec)

Event Timeline

root@labsdb1012.eqiad.wmnet[heartbeat]> select * from heartbeat where shard='s1' and ts like '2020%';
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| 2020-09-10T13:55:21.001020 | 171970661 | db1083-bin.005408 |   54728787 | db2112-bin.002159     |          1007695415 | s1    | eqiad      |
| 2020-09-07T08:45:43.000920 | 180363268 | db2112-bin.002159 | 1007687371 | NULL                  |                NULL | s1    | codfw      |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
2 rows in set (0.001 sec)