Page MenuHomePhabricator
Paste P13049

2020-10 dc switchback codfw->eqiad
ActivePublic
Actions

Authored by • Kormat on Oct 22 2020, 8:28 AM.
Tags
None
Referenced Files
F32410815: 2020-10 dc switchback codfw->eqiad
Oct 22 2020, 8:42 AM
F32410807: 2020-10 dc switchback codfw->eqiad
Oct 22 2020, 8:30 AM
F32410806: 2020-10 dc switchback codfw->eqiad
Oct 22 2020, 8:28 AM
F32410805: 2020-10 dc switchback codfw->eqiad
Oct 22 2020, 8:28 AM
F32410804: 2020-10 dc switchback codfw->eqiad
Oct 22 2020, 8:28 AM
Subscribers
None
# s6
kormat@x1c:~(0:0)$ master-pos db1131.eqiad.wmnet
# at 570303690
#201022 8:29:47 server id 180367475 end_log_pos 570304001 Query thread_id=14 exec_time=0 error_code=0
SET TIMESTAMP=1603355387/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T08:29:47.000890', '180367475', 'db2129-bin.001126', '265047915', NULL, NULL, 's6', 'codfw')
=======> Host: db1131.eqiad.wmnet
=======> File: db1131-bin.001248
=======> Pos: 570304001
change master to master_host='db1131.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1131-bin.001248', master_log_pos=570304001;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s6 eqiad codfw
Source: db1131.eqiad.wmnet 10.64.32.6
Dest: db2129
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T08:42:08.000950 | 171974662 | db1131-bin.001248 | 595609706 | db2129-bin.001126 | 290373509 | s6 | eqiad |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+

Event Timeline

• Kormat created this paste.Oct 22 2020, 8:28 AM
• Kormat updated the paste's language from autodetect to shell.
• Kormat edited the content of this paste. (Show Details)
• Kormat edited the content of this paste. (Show Details)
• Kormat edited the content of this paste. (Show Details)
• Kormat edited the content of this paste. (Show Details)Oct 22 2020, 8:42 AM
• Kormat added a comment.EditedOct 22 2020, 8:47 AM

s5

kormat@x1c:~(0:0)$ master-pos db1100.eqiad.wmnet
# at 769335177
#201022  8:47:40 server id 180359180  end_log_pos 769335488 	Query	thread_id=12	exec_time=0	error_code=0
SET TIMESTAMP=1603356460/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T08:47:40.000790', '180359180', 'db2123-bin.000948', '483420466', NULL, NULL, 's5', 'codfw')

=======> Host: db1100.eqiad.wmnet
=======> File: db1100-bin.002795
=======> Pos: 769335488

change master to master_host='db1100.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1100-bin.002795', master_log_pos=769335488;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s5 eqiad codfw
Source: db1100.eqiad.wmnet 10.64.32.197
Dest: db2123
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T08:50:15.000940 | 171974853 | db1100-bin.002795 | 775231419 | db2123-bin.000948     |           489359512 | s5    | eqiad      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
• Kormat added a comment.Oct 22 2020, 8:56 AM

s2

kormat@x1c:~(0:0)$ master-pos db1122.eqiad.wmnet
# at 56555850
#201022  8:52:09 server id 180359271  end_log_pos 56556161 	Query	thread_id=11	exec_time=0	error_code=0
SET TIMESTAMP=1603356729/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T08:52:09.000810', '180359271', 'db2107-bin.001785', '965641595', NULL, NULL, 's2', 'codfw')

=======> Host: db1122.eqiad.wmnet
=======> File: db1122-bin.002343
=======> Pos: 56556161

change master to master_host='db1122.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1122-bin.002343', master_log_pos=56556161;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s2 eqiad codfw
Source: db1122.eqiad.wmnet 10.64.48.34
Dest: db2107
+----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T08:56:00.001120 | 171978786 | db1122-bin.002343 | 62769740 | db2107-bin.001785     |           971885025 | s2    | eqiad      |
+----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+
• Kormat added a comment.EditedOct 22 2020, 8:59 AM

s7

kormat@x1c:~(0:0)$ master-pos db1086.eqiad.wmnet
# at 135473207
#201022  8:59:39 server id 180367395  end_log_pos 135473519 	Query	thread_id=4	exec_time=0	error_code=0
SET TIMESTAMP=1603357179/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T08:59:39.000760', '180367395', 'db2118-bin.001742', '1002488800', NULL, NULL, 's7', 'codfw')

=======> Host: db1086.eqiad.wmnet
=======> File: db1086-bin.004958
=======> Pos: 135473519

change master to master_host='db1086.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1086-bin.004958', master_log_pos=135473519;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s7 eqiad codfw
Source: db1086.eqiad.wmnet 10.64.16.104
Dest: db2118
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:01:03.001180 | 171970664 | db1086-bin.004958 | 137842000 | db2118-bin.001742     |          1004868007 | s7    | eqiad      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
• Kormat added a comment.EditedOct 22 2020, 9:03 AM

s4

kormat@x1c:~(0:0)$ master-pos db1081.eqiad.wmnet
# at 50776976
#201022  9:02:40 server id 180363436  end_log_pos 50777286 	Query	thread_id=4	exec_time=0	error_code=0
SET TIMESTAMP=1603357360/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T09:02:40.000810', '180363436', 'db2090-bin.005631', '22688982', NULL, NULL, 's4', 'codfw')

=======> Host: db1081.eqiad.wmnet
=======> File: db1081-bin.006153
=======> Pos: 50777286

change master to master_host='db1081.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1081-bin.006153', master_log_pos=50777286;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s4 eqiad codfw
Source: db1081.eqiad.wmnet 10.64.0.93
Dest: db2090
+----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:04:15.001170 | 171966557 | db1081-bin.006153 | 66953812 | db2090-bin.005631     |            38926189 | s4    | eqiad      |
+----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+
• Kormat added a comment.EditedOct 22 2020, 9:06 AM

s8

kormat@x1c:~(0:0)$ master-pos db1104.eqiad.wmnet
# at 623492805
#201022  9:05:56 server id 180355078  end_log_pos 623493116 	Query	thread_id=4	exec_time=0	error_code=0
SET TIMESTAMP=1603357556/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T09:05:56.001060', '180355078', 'db2079-bin.007114', '385911629', NULL, NULL, 's8', 'codfw')

=======> Host: db1104.eqiad.wmnet
=======> File: db1104-bin.005360
=======> Pos: 623493116

change master to master_host='db1104.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1104-bin.005360', master_log_pos=623493116;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s8 eqiad codfw
Source: db1104.eqiad.wmnet 10.64.16.85
Dest: db2079
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:07:30.000960 | 171970645 | db1104-bin.005360 | 627060539 | db2079-bin.007114     |           389479805 | s8    | eqiad      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
• Kormat added a comment.EditedOct 22 2020, 9:09 AM

s3

kormat@x1c:~(0:0)$ master-pos db1123.eqiad.wmnet
# at 1033273351
#201022  9:08:55 server id 171978787  end_log_pos 1033273685 	Query	thread_id=6	exec_time=0	error_code=0
SET TIMESTAMP=1603357735/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T09:08:55.001080', '171978787', 'db1123-bin.003549', '1033273313', 'db2105-bin.002177', '160826719', 's3', 'eqiad')

=======> Host: db1123.eqiad.wmnet
=======> File: db1123-bin.003549
=======> Pos: 1033273685

change master to master_host='db1123.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1123-bin.003549', master_log_pos=1033273685;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s3 eqiad codfw
Source: db1123.eqiad.wmnet 10.64.48.35
Dest: db2105
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:10:00.001080 | 171978787 | db1123-bin.003549 | 1043770774 | db2105-bin.002177     |           171389135 | s3    | eqiad      |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
• Kormat added a comment.EditedOct 22 2020, 9:11 AM

s1

kormat@x1c:~(0:0)$ master-pos db1083.eqiad.wmnet
# at 483880065
#201022  9:10:55 server id 180363268  end_log_pos 483880376 	Query	thread_id=39	exec_time=0	error_code=0
SET TIMESTAMP=1603357855/*!*/;
REPLACE INTO `heartbeat`.`heartbeat` (ts, server_id, file, position, relay_master_log_file, exec_master_log_pos, shard, datacenter) VALUES ('2020-10-22T09:10:55.001040', '180363268', 'db2112-bin.002348', '715978160', NULL, NULL, 's1', 'codfw')

=======> Host: db1083.eqiad.wmnet
=======> File: db1083-bin.005594
=======> Pos: 483880376

change master to master_host='db1083.eqiad.wmnet', master_user='repl', master_password='x', master_ssl=1, master_log_file='db1083-bin.005594', master_log_pos=483880376;
kormat@cumin1001:~(0:0)$ software/dbtools/check-master-heartbeat.sh s1 eqiad codfw
Source: db1083.eqiad.wmnet 10.64.16.101
Dest: db2112
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:12:02.000980 | 171970661 | db1083-bin.005594 | 485611419 | db2112-bin.002348     |           717729986 | s1    | eqiad      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
• Kormat mentioned this in T261914: Enable replication eqiad -> codfw and other checks.Oct 22 2020, 9:12 AM

Final check:

kormat@cumin1001:~(0:0)$ date -u --rfc-3339=sec; for i in s{1..8}; do echo "=======> $i"; ./software/dbtools/check-master-heartbeat.sh $i codfw eqiad; done                                                                                  
2020-10-22 09:12:58+00:00                                                                                                                                                                                                                    
=======> s1                                                                                                                                                                                                                                  
Source: db2112.codfw.wmnet 10.192.32.4                                                                                                                                                                                                       
Dest: db1083                                                                                                                                                                                                                                 
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+                                                                                                
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |                                                                                                
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:12:59.000890 | 180363268 | db2112-bin.002348 | 720257919 | db1083-bin.005594     |           488095728 | s1    | codfw      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
=======> s2
Source: db2107.codfw.wmnet 10.192.16.103
Dest: db1122
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:13:00.000880 | 180359271 | db2107-bin.001785 | 1000031286 | db1122-bin.002343     |            90797715 | s2    | codfw      |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
=======> s3
Source: db2105.codfw.wmnet 10.192.0.120
Dest: db1123
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:13:01.002390 | 180355192 | db2105-bin.002177 | 190857138 | db1123-bin.003550     |            14552511 | s3    | codfw      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
=======> s4
Source: db2090.codfw.wmnet 10.192.32.172
Dest: db1081
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:13:01.000860 | 180363436 | db2090-bin.005631 | 115222718 | db1081-bin.006153     |           142909905 | s4    | codfw      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
=======> s5
Source: db2123.codfw.wmnet 10.192.16.12
Dest: db1100
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:13:02.000950 | 180359180 | db2123-bin.000948 | 547036346 | db1100-bin.002795     |           832494131 | s5    | codfw      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
=======> s6
Source: db2129.codfw.wmnet 10.192.48.115
Dest: db1131
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:13:03.000990 | 180367475 | db2129-bin.001126 | 353964347 | db1131-bin.001248     |           659115266 | s6    | codfw      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
=======> s7
Source: db2118.codfw.wmnet 10.192.48.35
Dest: db1086
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position   | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:13:04.000800 | 180367395 | db2118-bin.001742 | 1026230768 | db1086-bin.004958     |           159122731 | s7    | codfw      |
+----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+
=======> s8
Source: db2079.codfw.wmnet 10.192.0.6
Dest: db1104
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| ts                         | server_id | file              | position  | relay_master_log_file | exec_master_log_pos | shard | datacenter |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
| 2020-10-22T09:13:05.001140 | 180355078 | db2079-bin.007114 | 403729191 | db1104-bin.005360     |           641307417 | s8    | codfw      |
+----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits