#==== EXAMPLE FOR THE codfw -> eqiad SWITCH ====###########################
# 03-set-db-readonly OUTPUT
###########################
Executing commands ('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shardSTART - Cookbook sre.switchdc.mediawiki.03-set-db-readonly
Setting in read-only mode all the core DB masters in codfw and verify those in eqiad
Skip verifying core DB masters in eqiad are in read-only mode
===== NO OUTPUT =====
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 5.92hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=1"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
===== NODE GROUP =====
(11) db[2034-2035,2039-2040,2043,2045,2048,2051-2052].codfw.wmnet,es[2016-2017].codfw.wmnet
----- OUTPUT of 'mysql --skip-ssl...lobal.read_only"' -----
1
================
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 5.11hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Check that all core masters in eqiad are in sync with the core masters in codfw.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:28:58.001220
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.97hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:28:58.001150
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.96hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:28:59.001580
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.06hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:00.001120
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.93hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:01.000940
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.02hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:02.001800
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.36hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:03.001030
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.98hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:04.001190
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.04hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:05.001020
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 2.95hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:05.000890
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.31hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:06.000920
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 3.54hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:07.001030
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.30hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:09.001920
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.37hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:10.001270
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.54hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:11.000850
================
PASS: |#######################################################################################| 100% (1/1) [00:01<00:00, 1.01s/hosts]
FAIL: | | 0% (0/1) [00:01<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:12.001080
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.55hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
= \'s1\' ORDER BY ts DESC LIMIT 1"',) on 1 hosts: db2048.codfw.wmnet100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-06T16:42:3710T10:29:14.001790130
================
PASS: |####################################################################################################| 100% (1/1) [00:00<00:00, 3.01.56hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Executing commands ('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard
= \'s1\' ORDER BY ts DESC LIMIT 1"',) on 1 hosts: db1067.eqiad.wmnet----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:15.001650
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.52hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-06T16:42:5010T10:29:16.001530060
================
PASS: |####################################################################################################| 100% (1/1) [00:00<00:00, 1.4354hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
And the later check passed without retries. This because we do first gather all the positions from dc_from and then check dc_to, so some time passes between one and the other.----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:17.001490
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.62hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:18.001680
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.39hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
----- OUTPUT of 'mysql --skip-ssl...ts DESC LIMIT 1"' -----
2018-09-10T10:29:19.001190
================
PASS: |#######################################################################################| 100% (1/1) [00:00<00:00, 1.36hosts/s]
FAIL: | | 0% (0/1) [00:00<?, ?hosts/s]
100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0)
###########################
# 03-set-db-readonly LOGS
###########################
2018-09-10 10:28:54,725 [INFO] START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly
2018-09-10 10:28:54,767 [INFO] Setting in read-only mode all the core DB masters in codfw and verify those in eqiad
2018-09-10 10:28:54,767 [INFO] Skip verifying core DB masters in eqiad are in read-only mode
2018-09-10 10:28:55,400 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=1"')] on '11' hosts: db[2034-2035,2039-2040,2043,2045,2048,2051-2052].codfw.wmnet,es[2016-2017].codfw.wmnet
2018-09-10 10:28:55,948 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=1"'
2018-09-10 10:28:55,949 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=1"'.
2018-09-10 10:28:55,950 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:56,574 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"')] on '11' hosts: db[2034-2035,2039-2040,2043,2045,2048,2051-2052].codfw.wmnet,es[2016-2017].codfw.wmnet
2018-09-10 10:28:56,958 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"'
2018-09-10 10:28:56,963 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
2018-09-10 10:28:56,963 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:56,965 [INFO] Check that all core masters in eqiad are in sync with the core masters in codfw.
2018-09-10 10:28:57,690 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2048.codfw.wmnet
2018-09-10 10:28:58,027 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:28:58,029 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:28:58,029 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:58,557 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2035.codfw.wmnet
2018-09-10 10:28:58,895 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:28:58,896 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:28:58,897 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:28:59,415 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2043.codfw.wmnet
2018-09-10 10:28:59,741 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:28:59,743 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:28:59,744 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:00,243 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s4\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2051.codfw.wmnet
2018-09-10 10:29:00,584 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's4' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:00,586 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:00,587 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:01,316 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s5\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2052.codfw.wmnet
2018-09-10 10:29:01,648 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's5' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:01,650 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:01,651 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:02,307 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s6\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2039.codfw.wmnet
2018-09-10 10:29:02,605 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's6' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:02,606 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:02,607 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:03,059 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s7\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2040.codfw.wmnet
2018-09-10 10:29:03,394 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's7' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:03,396 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:03,397 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:03,868 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s8\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2045.codfw.wmnet
2018-09-10 10:29:04,196 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's8' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:04,198 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:04,199 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:04,687 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'x1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db2034.codfw.wmnet
2018-09-10 10:29:05,025 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'x1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:05,027 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:05,028 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:05,539 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es2016.codfw.wmnet
2018-09-10 10:29:05,840 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:05,842 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:05,843 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:06,298 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es2017.codfw.wmnet
2018-09-10 10:29:06,581 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:06,583 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:06,584 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:07,097 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es1017.eqiad.wmnet
2018-09-10 10:29:07,865 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:07,867 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:07,868 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:08,416 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'x1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1069.eqiad.wmnet
2018-09-10 10:29:09,146 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'x1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:09,147 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:09,148 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:09,637 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s7\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1062.eqiad.wmnet
2018-09-10 10:29:10,289 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's7' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:10,290 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:10,291 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:10,776 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'es2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: es1015.eqiad.wmnet
2018-09-10 10:29:11,782 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 'es2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:11,784 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:11,785 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:12,289 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s5\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1070.eqiad.wmnet
2018-09-10 10:29:12,933 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's5' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:12,935 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:12,936 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:13,422 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s4\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1068.eqiad.wmnet
2018-09-10 10:29:14,062 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's4' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:14,063 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:14,064 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:14,579 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s8\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1071.eqiad.wmnet
2018-09-10 10:29:15,237 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's8' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:15,239 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:15,239 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:15,752 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s3\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1075.eqiad.wmnet
2018-09-10 10:29:16,399 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's3' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:16,401 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:16,402 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:16,949 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s6\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1061.eqiad.wmnet
2018-09-10 10:29:17,568 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's6' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:17,569 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:17,570 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:18,060 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s2\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1066.eqiad.wmnet
2018-09-10 10:29:18,782 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's2' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:18,783 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:18,784 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:19,283 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = \'codfw\' and shard = \'s1\' ORDER BY ts DESC LIMIT 1"')] on '1' hosts: db1067.eqiad.wmnet
2018-09-10 10:29:20,018 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT ts FROM heartbeat.heartbeat WHERE datacenter = 'codfw' and shard = 's1' ORDER BY ts DESC LIMIT 1"'
2018-09-10 10:29:20,019 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...ts DESC LIMIT 1"'.
2018-09-10 10:29:20,020 [INFO] 100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:29:20,021 [INFO] END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0)
############################
# 06-set-db-readwrite OUTPUT
############################
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite
Setting in read-write mode all the core DB masters in eqiad
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
===== NO OUTPUT =====
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 2.22hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=0"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
===== NODE GROUP =====
(11) db[1061-1062,1066-1071,1075].eqiad.wmnet,es[1015,1017].eqiad.wmnet
----- OUTPUT of 'mysql --skip-ssl...lobal.read_only"' -----
0
================
PASS: |#####################################################################################| 100% (11/11) [00:00<00:00, 14.75hosts/s]
FAIL: | | 0% (0/11) [00:00<?, ?hosts/s]
100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0)
###########################
# 06-set-db-readwrite LOGS
###########################
2018-09-10 10:57:54,396 [INFO] START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite
2018-09-10 10:57:54,437 [INFO] Setting in read-write mode all the core DB masters in eqiad
2018-09-10 10:57:54,457 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:54,554 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:54,992 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:55,070 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=0"')] on '11' hosts: db[1061-1062,1066-1071,1075].eqiad.wmnet,es[1015,1017].eqiad.wmnet
2018-09-10 10:57:55,862 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SET GLOBAL read_only=0"'
2018-09-10 10:57:55,863 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...BAL read_only=0"'.
2018-09-10 10:57:55,864 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:57:55,884 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:55,981 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:56,434 [INFO] Starting new HTTPS connection (1): puppetdb2001.codfw.wmnet
2018-09-10 10:57:56,522 [INFO] Executing commands [cumin.transports.Command('mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"')] on '11' hosts: db[1061-1062,1066-1071,1075].eqiad.wmnet,es[1015,1017].eqiad.wmnet
2018-09-10 10:57:57,261 [INFO] Completed command 'mysql --skip-ssl --skip-column-names --batch -e "SELECT @@global.read_only"'
2018-09-10 10:57:57,268 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) for command: 'mysql --skip-ssl...lobal.read_only"'.
2018-09-10 10:57:57,268 [INFO] 100.0% (11/11) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
2018-09-10 10:57:57,270 [INFO] END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0)