Page Menu
Home
Phabricator
Search
Configure Global Search
Log In
Files
F25752751
switchdc DB steps
No One
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Authored By
Volans
Sep 10 2018, 4:36 PM
2018-09-10 16:36:28 (UTC+0)
Size
35 KB
Referenced Files
None
Subscribers
None
switchdc DB steps
View Options
###########################
# 03-set-db-readonly OUTPUT
###########################
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly
Setting in read-only mode all the core DB masters in codfw and verify those in eqiad
Skip verifying core DB masters in eqiad are in read-only mode
===== NO OUTPUT =====
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 5.92hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=1"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
===== NODE GROUP =====
(11) db[2034-2035,2039-2040,2043,2045,2048,2051-2052].codfw.wmnet,es[2016-2017].codfw.wmnet
----- OUTPUT of 'mysql --skip-ssl...lobal.read_only"' -----
1
================
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 5.11hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Check that all core masters in eqiad are in sync with the core masters in codfw.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:28:58.001220
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.97hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:28:58.001150
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.96hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:28:59.001580
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.06hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:00.001120
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.93hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:01.000940
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.02hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:02.001800
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.36hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:03.001030
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.98hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:04.001190
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.04hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:05.001020
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.95hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:05.000890
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.31hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:06.000920
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.54hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:07.001030
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.30hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:09.001920
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.37hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:10.001270
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.54hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:11.000850
================
PASS: |#######################################################################################| 100% (1/1) [00:01<00:00, 1.01s/hosts]
FAIL: | | 0% (0/1) [00:01<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:12.001080
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.55hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:14.001130
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.56hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:15.001650
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.52hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:16.001060
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.54hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:17.001490
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.62hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:18.001680
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.39hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:19.001190
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.36hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0)
###########################
# 03-set-db-readonly LOGS
###########################
2018-09-10 10:28:54,725 [INFO] START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly
2018-09-10 10:28:54,767 [INFO] Setting in read-only mode all the core DB masters in codfw and verify those in eqiad
2018-09-10 10:28:54,767 [INFO] Skip verifying core DB masters in eqiad are in read-only mode
2018-09-10 10:28:55,400 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=1"')] on '11' hosts: db[2034-2035,2039-2040,2043,2045,2048,2051-2052].codfw.wmnet,es[2016-2017].codfw.wmnet
2018-09-10 10:28:55,948 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=1"'
2018-09-10 10:28:55,949 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=1"'.
2018-09-10 10:28:55,950 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:56,574 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"')] on '11' hosts: db[2034-2035,2039-2040,2043,2045,2048,2051-2052].codfw.wmnet,es[2016-2017].codfw.wmnet
2018-09-10 10:28:56,958 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"'
2018-09-10 10:28:56,963 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
2018-09-10 10:28:56,963 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:56,965 [INFO] Check that all core masters in eqiad are in sync with the core masters in codfw.
2018-09-10 10:28:57,690 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2048.codfw.wmnet
2018-09-10 10:28:58,027 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:28:58,029 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:28:58,029 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:58,557 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2035.codfw.wmnet
2018-09-10 10:28:58,895 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:28:58,896 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:28:58,897 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:59,415 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2043.codfw.wmnet
2018-09-10 10:28:59,741 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:28:59,743 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:28:59,744 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:00,243 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s4\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2051.codfw.wmnet
2018-09-10 10:29:00,584 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's4' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:00,586 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:00,587 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:01,316 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s5\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2052.codfw.wmnet
2018-09-10 10:29:01,648 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's5' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:01,650 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:01,651 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:02,307 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s6\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2039.codfw.wmnet
2018-09-10 10:29:02,605 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's6' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:02,606 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:02,607 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:03,059 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s7\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2040.codfw.wmnet
2018-09-10 10:29:03,394 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's7' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:03,396 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:03,397 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:03,868 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s8\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2045.codfw.wmnet
2018-09-10 10:29:04,196 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's8' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:04,198 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:04,199 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:04,687 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'x1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2034.codfw.wmnet
2018-09-10 10:29:05,025 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'x1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:05,027 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:05,028 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:05,539 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es2016.codfw.wmnet
2018-09-10 10:29:05,840 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:05,842 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:05,843 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:06,298 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es2017.codfw.wmnet
2018-09-10 10:29:06,581 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:06,583 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:06,584 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:07,097 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es1017.eqiad.wmnet
2018-09-10 10:29:07,865 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:07,867 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:07,868 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:08,416 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'x1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1069.eqiad.wmnet
2018-09-10 10:29:09,146 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'x1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:09,147 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:09,148 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:09,637 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s7\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1062.eqiad.wmnet
2018-09-10 10:29:10,289 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's7' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:10,290 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:10,291 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:10,776 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es1015.eqiad.wmnet
2018-09-10 10:29:11,782 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:11,784 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:11,785 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:12,289 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s5\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1070.eqiad.wmnet
2018-09-10 10:29:12,933 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's5' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:12,935 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:12,936 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:13,422 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s4\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1068.eqiad.wmnet
2018-09-10 10:29:14,062 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's4' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:14,063 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:14,064 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:14,579 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s8\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1071.eqiad.wmnet
2018-09-10 10:29:15,237 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's8' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:15,239 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:15,239 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:15,752 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1075.eqiad.wmnet
2018-09-10 10:29:16,399 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:16,401 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:16,402 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:16,949 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s6\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1061.eqiad.wmnet
2018-09-10 10:29:17,568 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's6' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:17,569 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:17,570 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:18,060 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1066.eqiad.wmnet
2018-09-10 10:29:18,782 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:18,783 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:18,784 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:19,283 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1067.eqiad.wmnet
2018-09-10 10:29:20,018 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:20,019 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:20,020 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:20,021 [INFO] END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0)
############################
# 06-set-db-readwrite OUTPUT
############################
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite
Setting in read-write mode all the core DB masters in eqiad
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
===== NO OUTPUT =====
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 2.22hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=0"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
===== NODE GROUP =====
(11) db[1061-1062,1066-1071,1075].eqiad.wmnet,es[1015,1017].eqiad.wmnet
----- OUTPUT of 'mysql --skip-ssl...lobal.read_only"' -----
0
================
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 14.75hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0)
###########################
# 06-set-db-readwrite LOGS
###########################
2018-09-10 10:57:54,396 [INFO] START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite
2018-09-10 10:57:54,437 [INFO] Setting in read-write mode all the core DB masters in eqiad
2018-09-10 10:57:54,457 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:54,554 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:54,992 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:55,070 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=0"')] on '11' hosts: db[1061-1062,1066-1071,1075].eqiad.wmnet,es[1015,1017].eqiad.wmnet
2018-09-10 10:57:55,862 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=0"'
2018-09-10 10:57:55,863 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=0"'.
2018-09-10 10:57:55,864 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:57:55,884 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:55,981 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:56,434 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:56,522 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"')] on '11' hosts: db[1061-1062,1066-1071,1075].eqiad.wmnet,es[1015,1017].eqiad.wmnet
2018-09-10 10:57:57,261 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"'
2018-09-10 10:57:57,268 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
2018-09-10 10:57:57,268 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:57:57,270 [INFO] END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0)
File Metadata
Details
Attached
Mime Type
text/plain; charset=utf-8
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
6400091
Default Alt Text
switchdc DB steps (35 KB)
Attached To
Mode
P7519 switchdc DB steps
Attached
Detach File
Event Timeline
Log In to Comment