Page MenuHomePhabricator

Set up TLS for MariaDB replication
Closed, ResolvedPublic

Description

MariaDB replication flows are currently not encrypted. They should be, as sometimes they cross datacenter boundaries. I think TLS is supported by MySQL/MariaDB and it would probably be the easiest way forward for this.

Related Objects

StatusAssignedTask
OpenNone
Openaaron
StalledNone
OpenNone
Resolvedjcrespo
Resolvedjcrespo
ResolvedCmjohnson
ResolvedGehel
Resolvedjcrespo
Resolvedjcrespo
Resolvedjcrespo
ResolvedMarostegui
DuplicateNone
Resolvedjcrespo
OpenNone
OpenNone
Resolvedjcrespo
ResolvedCmjohnson
ResolvedCmjohnson
ResolvedCmjohnson
Resolvedjcrespo
ResolvedMarostegui
ResolvedRobH
ResolvedAndrew
ResolvedCmjohnson
Resolvedjcrespo
ResolvedCmjohnson
ResolvedCmjohnson
Resolvedjcrespo
ResolvedCmjohnson
Resolvedjcrespo
ResolvedPapaul
ResolvedMarostegui
ResolvedRobH
ResolvedRobH

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 315049 had a related patch set uploaded (by Jcrespo):
Change phabricator misc dbs to use puppet TLS certificates

https://gerrit.wikimedia.org/r/315049

Above commands as of now:

$ sudo salt -C 'G@cluster:mysql and G@site:eqiad' cmd.run 'grep -l 'server\.key' /etc/my.cnf' | grep -c '/etc/my\.cnf'
102
$ sudo salt -C 'G@cluster:mysql and G@site:eqiad' cmd.run 'pt-config-diff --defaults-file=/root/.my.cnf --report-width=200 h=localhost /etc/my.cnf | grep "server\.key"' | grep -c 'server\.key'
20

Change 315049 merged by Jcrespo:
Change phabricator misc dbs to use puppet TLS certificates

https://gerrit.wikimedia.org/r/315049

Change 315051 had a related patch set uploaded (by Jcrespo):
Update phabricator my.cnf config template to include TLS config

https://gerrit.wikimedia.org/r/315051

Change 315051 merged by Jcrespo:
Update phabricator my.cnf config template to include TLS config

https://gerrit.wikimedia.org/r/315051

Change 319806 had a related patch set uploaded (by Jcrespo):
Allow SSL (TLS) and performance_schema on misc servers

https://gerrit.wikimedia.org/r/319806

Change 319806 merged by Jcrespo:
Allow SSL (TLS) and performance_schema on misc servers

https://gerrit.wikimedia.org/r/319806

Change 319831 had a related patch set uploaded (by Jcrespo):
Enable ssl (TLS) on misc database servers

https://gerrit.wikimedia.org/r/319831

Change 319831 merged by Jcrespo:
Enable ssl (TLS) on misc database servers

https://gerrit.wikimedia.org/r/319831

jcrespo moved this task from Backlog to Meta/Epic on the DBA board.Nov 10 2016, 12:26 PM
jcrespo lowered the priority of this task from High to Normal.Nov 24 2016, 11:43 AM

Out of 157 active hosts responding to salt, 15 host with no TLS deployed, 42 with the old certificate, 100 with the puppet one:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
15
$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
42
$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
100

List of eqiad hosts with the old cert:

db1015.eqiad.wmnet
db1021.eqiad.wmnet
db1022.eqiad.wmnet
db1036.eqiad.wmnet
db1054.eqiad.wmnet
db1060.eqiad.wmnet
db1063.eqiad.wmnet
db1067.eqiad.wmnet
db1074.eqiad.wmnet
db1076.eqiad.wmnet

db1046.eqiad.wmnet
db1047.eqiad.wmnet
dbstore1002.eqiad.wmnet
dbstore1001.eqiad.wmnet
labsdb1009.eqiad.wmnet
labsdb1010.eqiad.wmnet
labsdb1011.eqiad.wmnet

MySQLs wit no SSL

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
14

MySQL with expired TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
30

MySQL with latest TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
114

I have enabled TLS on neodymium and sarin, but because the mysql clients there are not using OpenSSL, clients will fail with:

ERROR 2026 (HY000): SSL connection error: unknown error number

I said we shouldn't, but we may have to create client packages after all, to allow for TLS 1.2 clients beyond the mysql servers.

Change 327703 had a related patch set uploaded (by Marostegui):
osc_host.sh: Added skip-ssl for the connection

https://gerrit.wikimedia.org/r/327703

Change 327703 merged by jenkins-bot:
osc_host.sh: Add skip-ssl for the connection

https://gerrit.wikimedia.org/r/327703

MySQLs with no SSL

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
13

MySQL with expired TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
26

MySQL with latest TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
120

Change 335227 had a related patch set uploaded (by Jcrespo):
sanitarium2: Enable TLS, disable Toku-specific config

https://gerrit.wikimedia.org/r/335227

Change 335227 merged by Jcrespo:
sanitarium2: Enable TLS, disable Toku-specific config

https://gerrit.wikimedia.org/r/335227

Change 335233 had a related patch set uploaded (by Jcrespo):
mariadb: Add TLS support for tendril

https://gerrit.wikimedia.org/r/335233

Change 335233 merged by Jcrespo:
mariadb: Add TLS support for tendril

https://gerrit.wikimedia.org/r/335233

Mentioned in SAL (#wikimedia-operations) [2017-01-31T17:37:10Z] <jynus> stopping mysql, upgrading and restarting db1011- temporary outage of tendril & dbtree T111654

36 pending hosts:

db1030.eqiad.wmnet: NULL
db1045.eqiad.wmnet: NULL
db1020.eqiad.wmnet: NULL
db1001.eqiad.wmnet: NULL
db1039.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
labsdb1001.eqiad.wmnet: NULL
labsdb1003.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
db1009.eqiad.wmnet: NULL
db1016.eqiad.wmnet: NULL
db1069.eqiad.wmnet: NULL
db1067.eqiad.wmnet: cacert
db1022.eqiad.wmnet: cacert
db1021.eqiad.wmnet: cacert
db1015.eqiad.wmnet: cacert
db1036.eqiad.wmnet: cacert
db2044.codfw.wmnet: cacert
db2063.codfw.wmnet: cacert
db2051.codfw.wmnet: cacert
db2046.codfw.wmnet: cacert
db2059.codfw.wmnet: cacert
db2065.codfw.wmnet: cacert
db2053.codfw.wmnet: cacert
db2039.codfw.wmnet: cacert
db2054.codfw.wmnet: cacert
db2061.codfw.wmnet: cacert
db2050.codfw.wmnet: cacert
db2041.codfw.wmnet: cacert
db2036.codfw.wmnet: cacert
db2037.codfw.wmnet: cacert
db2045.codfw.wmnet: cacert
db2052.codfw.wmnet: cacert
db2058.codfw.wmnet: cacert
db2064.codfw.wmnet: cacert
db2043.codfw.wmnet: cacert
jcrespo added a comment.EditedFeb 1 2017, 11:50 PM
sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
130

18 with expired certs:

db1021.eqiad.wmnet: cacert
db1022.eqiad.wmnet: cacert
db1036.eqiad.wmnet: cacert
db1015.eqiad.wmnet: cacert
db2052.codfw.wmnet: cacert
db2059.codfw.wmnet: cacert
db2063.codfw.wmnet: cacert
db2053.codfw.wmnet: cacert
db2054.codfw.wmnet: cacert
db2041.codfw.wmnet: cacert
db2061.codfw.wmnet: cacert
db2039.codfw.wmnet: cacert
db2046.codfw.wmnet: cacert
db2045.codfw.wmnet: cacert
db2036.codfw.wmnet: cacert
db2064.codfw.wmnet: cacert
db2050.codfw.wmnet: cacert
db2043.codfw.wmnet: cacert

Mentioned in SAL (#wikimedia-operations) [2017-02-02T17:43:54Z] <jynus> upgrade & restart of db2052 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-02T17:56:56Z] <jynus> upgrade & restart of db2059 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T09:54:08Z] <jynus> upgrade & restart of db2063 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T10:54:49Z] <jynus> preparing to reimage db2053 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T11:21:13Z] <jynus> preparing to reimage db2054 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T13:58:04Z] <jynus> restarting and upgrading db2041 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T14:30:58Z] <jynus> upgrade and restart db2061 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T15:01:47Z] <jynus> preparing to reimage db2039 T111654

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2039.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702031641_jynus_2666.log.

Completed auto-reimage of hosts:

['db2039.codfw.wmnet']

and were ALL successful.

After resolving T152188, pending hosts:

$ sudo salt --output=txt -C 'G@cluster:mysql' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"' | grep NULL
db1020.eqiad.wmnet: NULL
db1001.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
labsdb1003.eqiad.wmnet: NULL
db1009.eqiad.wmnet: NULL
db1016.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
db1030.eqiad.wmnet: NULL
labsdb1001.eqiad.wmnet: NULL
db1045.eqiad.wmnet: NULL
db1069.eqiad.wmnet: NULL

Some of those will be decomissioned very soon and probably never deployed TLS. Counting only core dbs:

$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"' | grep NULL
db1045.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
db1030.eqiad.wmnet: NULL

Change 336601 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1045 for maintenance

https://gerrit.wikimedia.org/r/336601

Change 336601 merged by jenkins-bot:
mariadb: Depool db1045 for maintenance

https://gerrit.wikimedia.org/r/336601

Mentioned in SAL (#wikimedia-operations) [2017-02-08T10:39:54Z] <jynus> upgrading and restarting db1045 T111654

Change 336609 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1037 for maintenance

https://gerrit.wikimedia.org/r/336609

Change 336609 merged by Jcrespo:
mariadb: Depool db1037 for maintenance

https://gerrit.wikimedia.org/r/336609

Mentioned in SAL (#wikimedia-operations) [2017-02-08T12:17:30Z] <jynus> upgrading and restarting db1037 T111654

Change 336620 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1026 for maintenance

https://gerrit.wikimedia.org/r/336620

Change 336620 merged by jenkins-bot:
mariadb: Depool db1026 for maintenance

https://gerrit.wikimedia.org/r/336620

Mentioned in SAL (#wikimedia-operations) [2017-02-08T14:17:32Z] <jynus> upgrading and restarting db1026 T111654

Change 336636 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1023 for maintenance

https://gerrit.wikimedia.org/r/336636

Change 336636 merged by jenkins-bot:
mariadb: Depool db1030 for maintenance

https://gerrit.wikimedia.org/r/336636

Mentioned in SAL (#wikimedia-operations) [2017-02-08T16:19:06Z] <jynus> upgrading and restarting db1030 T111654

TLS is now deployed on all core servers:

root@neodymium:~$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"'
db1071.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1041.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1030.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1045.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1037.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1076.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1022.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1067.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1092.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1074.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1063.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1029.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1084.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1082.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1091.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1018.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1015.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1077.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1080.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1012.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1090.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1061.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1089.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1083.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1052.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1013.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1068.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1060.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1081.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1011.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1094.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1053.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1086.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1087.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1054.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1051.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1070.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1085.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1073.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1078.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1079.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1093.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1075.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1015.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1088.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1016.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1065.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1033.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1028.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1057.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1056.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1062.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1050.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1024.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1035.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1059.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1014.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1026.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1023.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1018.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1039.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1019.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1064.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1040.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1031.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1021.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1036.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1044.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1034.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1066.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2029.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2011.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2018.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2067.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2038.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2019.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2041.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2019.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2013.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2063.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2045.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2017.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2065.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2064.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1049.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2048.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2058.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2042.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2039.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2012.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2033.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2034.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2054.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2043.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2061.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2014.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2057.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2070.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2049.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2046.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2040.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2060.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2037.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2069.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2066.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2047.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2044.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2053.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2036.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2056.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2068.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2050.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2055.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2062.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2059.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2035.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1017.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2016.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2018.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2028.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1072.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2051.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2052.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1055.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1038.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2016.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2023.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2017.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2015.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem

Enabling it on all pending hosts:

db1028.eqiad.wmnet:            Master_SSL_Allowed: No
db1077.eqiad.wmnet:            Master_SSL_Allowed: No
db1034.eqiad.wmnet:            Master_SSL_Allowed: No
db1067.eqiad.wmnet:            Master_SSL_Allowed: No
db1044.eqiad.wmnet:            Master_SSL_Allowed: No
es1013.eqiad.wmnet:            Master_SSL_Allowed: No
db1065.eqiad.wmnet:            Master_SSL_Allowed: No
es1017.eqiad.wmnet:            Master_SSL_Allowed: No
db1078.eqiad.wmnet:            Master_SSL_Allowed: No
db1055.eqiad.wmnet:            Master_SSL_Allowed: No
db2067.codfw.wmnet:            Master_SSL_Allowed: No
db2065.codfw.wmnet:            Master_SSL_Allowed: No
db2044.codfw.wmnet:            Master_SSL_Allowed: No
db2057.codfw.wmnet:            Master_SSL_Allowed: No
db2038.codfw.wmnet:            Master_SSL_Allowed: No
db2052.codfw.wmnet:            Master_SSL_Allowed: No
db2035.codfw.wmnet:            Master_SSL_Allowed: No
db2059.codfw.wmnet:            Master_SSL_Allowed: No
db2048.codfw.wmnet:            Master_SSL_Allowed: No
db2069.codfw.wmnet:            Master_SSL_Allowed: No
db2051.codfw.wmnet:            Master_SSL_Allowed: No
db2037.codfw.wmnet:            Master_SSL_Allowed: No
db2070.codfw.wmnet:            Master_SSL_Allowed: No
db2058.codfw.wmnet:            Master_SSL_Allowed: No
db2062.codfw.wmnet:            Master_SSL_Allowed: No
db2068.codfw.wmnet:            Master_SSL_Allowed: No
db2066.codfw.wmnet:            Master_SSL_Allowed: No
db2056.codfw.wmnet:            Master_SSL_Allowed: No
db2055.codfw.wmnet:            Master_SSL_Allowed: No

Mentioned in SAL (#wikimedia-operations) [2017-02-08T17:04:01Z] <jynus> rolling restart of replication thread of 29 mysql hosts T111654

Enabled everywhere except on db1034 and db2057, which probably require a package upgrade.

$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql --skip-ssl -e "SHOW SLAVE STATUS\G" | grep Master_SSL_Allowed'
db1040.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1037.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1077.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1084.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1045.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1078.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2018.codfw.wmnet:            Master_SSL_Allowed: Yes
db1094.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2037.codfw.wmnet:            Master_SSL_Allowed: Yes
db1036.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1083.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2048.codfw.wmnet:            Master_SSL_Allowed: Yes
db2046.codfw.wmnet:            Master_SSL_Allowed: Yes
db2069.codfw.wmnet:            Master_SSL_Allowed: Yes
db1072.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1051.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2019.codfw.wmnet:            Master_SSL_Allowed: Yes
db1056.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1071.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1091.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1029.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1052.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1031.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1068.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1050.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1082.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1055.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2065.codfw.wmnet:            Master_SSL_Allowed: Yes
db2044.codfw.wmnet:            Master_SSL_Allowed: Yes
db1076.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1092.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1061.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1059.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2060.codfw.wmnet:            Master_SSL_Allowed: Yes
db2053.codfw.wmnet:            Master_SSL_Allowed: Yes
db1086.eqiad.wmnet:            Master_SSL_Allowed: Yes
es1017.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2070.codfw.wmnet:            Master_SSL_Allowed: Yes
db2040.codfw.wmnet:            Master_SSL_Allowed: Yes
db2055.codfw.wmnet:            Master_SSL_Allowed: Yes
db2063.codfw.wmnet:            Master_SSL_Allowed: Yes
db2023.codfw.wmnet:            Master_SSL_Allowed: Yes
db2059.codfw.wmnet:            Master_SSL_Allowed: Yes
db1018.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1074.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2061.codfw.wmnet:            Master_SSL_Allowed: Yes
db1060.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2014.codfw.wmnet:            Master_SSL_Allowed: Yes
db1067.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2051.codfw.wmnet:            Master_SSL_Allowed: Yes
db1026.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2047.codfw.wmnet:            Master_SSL_Allowed: Yes
db1028.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1034.eqiad.wmnet:            Master_SSL_Allowed: No
db1015.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2064.codfw.wmnet:            Master_SSL_Allowed: Yes
db2054.codfw.wmnet:            Master_SSL_Allowed: Yes
db2019.codfw.wmnet:            Master_SSL_Allowed: Yes
db1030.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1088.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1038.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1063.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1080.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1049.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1053.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1079.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1089.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2067.codfw.wmnet:            Master_SSL_Allowed: Yes
db1064.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1073.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1057.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2045.codfw.wmnet:            Master_SSL_Allowed: Yes
es2016.codfw.wmnet:            Master_SSL_Allowed: Yes
db1024.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2015.codfw.wmnet:            Master_SSL_Allowed: Yes
db1093.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1085.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2029.codfw.wmnet:            Master_SSL_Allowed: Yes
db2038.codfw.wmnet:            Master_SSL_Allowed: Yes
db2058.codfw.wmnet:            Master_SSL_Allowed: Yes
db2049.codfw.wmnet:            Master_SSL_Allowed: Yes
db2052.codfw.wmnet:            Master_SSL_Allowed: Yes
db1065.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1033.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1070.eqiad.wmnet:            Master_SSL_Allowed: Yes
es1013.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1023.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1081.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1054.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1066.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1022.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2033.codfw.wmnet:            Master_SSL_Allowed: Yes
db2057.codfw.wmnet:            Master_SSL_Allowed: No
db2034.codfw.wmnet:            Master_SSL_Allowed: Yes
es1019.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1062.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1087.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2050.codfw.wmnet:            Master_SSL_Allowed: Yes
db2066.codfw.wmnet:            Master_SSL_Allowed: Yes
db2035.codfw.wmnet:            Master_SSL_Allowed: Yes
db2028.codfw.wmnet:            Master_SSL_Allowed: Yes
db2017.codfw.wmnet:            Master_SSL_Allowed: Yes
db2016.codfw.wmnet:            Master_SSL_Allowed: Yes
db2036.codfw.wmnet:            Master_SSL_Allowed: Yes
db2018.codfw.wmnet:            Master_SSL_Allowed: Yes
db1041.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1021.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2043.codfw.wmnet:            Master_SSL_Allowed: Yes
es2017.codfw.wmnet:            Master_SSL_Allowed: Yes
db2056.codfw.wmnet:            Master_SSL_Allowed: Yes
db2062.codfw.wmnet:            Master_SSL_Allowed: Yes
db2039.codfw.wmnet:            Master_SSL_Allowed: Yes
db1090.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1044.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1035.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2041.codfw.wmnet:            Master_SSL_Allowed: Yes
es1015.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2068.codfw.wmnet:            Master_SSL_Allowed: Yes
db1075.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1039.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2042.codfw.wmnet:            Master_SSL_Allowed: Yes

Change 336644 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db2057 for mariadb upgrade

https://gerrit.wikimedia.org/r/336644

TLS is now deployed on all core servers:

Congratulations, that was a massive and tedious effort.

Change 336644 merged by jenkins-bot:
mariadb: Depool db2057 for mariadb upgrade

https://gerrit.wikimedia.org/r/336644

Mentioned in SAL (#wikimedia-operations) [2017-02-08T18:02:21Z] <jynus> upgrading and restarting db2057 T111654

Change 336661 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1034 for maintenance

https://gerrit.wikimedia.org/r/336661

jcrespo added a comment.EditedFeb 8 2017, 7:07 PM

db1034 is left, pending of the reimage marked above^.

Of the non core hosts, only the following are left (all to be decommed, marking only as such the ones that already have replacements):

db1020.eqiad.wmnet: NULL - m2 master
db1009.eqiad.wmnet: NULL - m5 master
db1001.eqiad.wmnet: NULL - m1 slave
labsdb1001.eqiad.wmnet: NULL - to be decommed
labsdb1003.eqiad.wmnet: NULL - to be decommed
db1016.eqiad.wmnet: NULL - m1 master
db1069.eqiad.wmnet: NULL - to be decommed

Change 336661 merged by jenkins-bot:
mariadb: Depool db1034 for maintenance

https://gerrit.wikimedia.org/r/336661

Mentioned in SAL (#wikimedia-operations) [2017-02-09T09:38:56Z] <jynus> upgrading and restarting db1034 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-09T10:26:58Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1034 - T111654 (duration: 00m 41s)

Change 336774 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db2040 for maintenance

https://gerrit.wikimedia.org/r/336774

Change 336774 merged by jenkins-bot:
mariadb: Depool db2040 for maintenance

https://gerrit.wikimedia.org/r/336774

Mentioned in SAL (#wikimedia-operations) [2017-02-09T10:42:39Z] <jynus> preparing to reimage db2040 T111654

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2040.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702091223_jynus_7278.log.

Completed auto-reimage of hosts:

['db2040.codfw.wmnet']

and were ALL successful.

All core servers/server with core data now support TLS connections and use it for replication (except labs- the new server suport it, but are not accesible remotely for security, and the old ones, to be decommissioned, do not support it ):

$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -BN -h $host -P $port $db -e "SELECT 1"; done
db2034.codfw.wmnet 3306: 1
db2042.codfw.wmnet 3306: 1
db2048.codfw.wmnet 3306: 1
db2055.codfw.wmnet 3306: 1
db2062.codfw.wmnet 3306: 1
db2069.codfw.wmnet 3306: 1
db2070.codfw.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2016.codfw.wmnet 3306: 1
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1080.eqiad.wmnet 3306: 1
db1083.eqiad.wmnet 3306: 1
db1089.eqiad.wmnet 3306: 1
db1073.eqiad.wmnet 3306: 1
db1072.eqiad.wmnet 3306: 1
db1066.eqiad.wmnet 3306: 1
db1065.eqiad.wmnet 3306: 1
db1055.eqiad.wmnet 3306: 1
db1051.eqiad.wmnet 3306: 1
db1047.eqiad.wmnet 3306: 1
db1057.eqiad.wmnet 3306: 1
db1052.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2035.codfw.wmnet 3306: 1
db2041.codfw.wmnet 3306: 1
db2049.codfw.wmnet 3306: 1
db2056.codfw.wmnet 3306: 1
db2063.codfw.wmnet 3306: 1
db2064.codfw.wmnet 3306: 1
db2017.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306: 1
dbstore1001.eqiad.wmnet 3306: 1
db1021.eqiad.wmnet 3306: 1
db1024.eqiad.wmnet 3306: 1
db1036.eqiad.wmnet 3306: 1
db1047.eqiad.wmnet 3306: 1
db1054.eqiad.wmnet 3306: 1
db1060.eqiad.wmnet 3306: 1
db1063.eqiad.wmnet 3306: 1
db1067.eqiad.wmnet 3306: 1
db1074.eqiad.wmnet 3306: 1
db1076.eqiad.wmnet 3306: 1
db1090.eqiad.wmnet 3306: 1
db1018.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2036.codfw.wmnet 3306: 1
db2043.codfw.wmnet 3306: 1
db2050.codfw.wmnet 3306: 1
db2057.codfw.wmnet 3306: 1
db2018.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306: 1
db1035.eqiad.wmnet 3306: 1
db1038.eqiad.wmnet 3306: 1
db1044.eqiad.wmnet 3306: 1
db1077.eqiad.wmnet 3306: 1
db1078.eqiad.wmnet 3306: 1
db1075.eqiad.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
db2065.codfw.wmnet 3306: 1
db2058.codfw.wmnet 3306: 1
db2051.codfw.wmnet 3306: 1
db2044.codfw.wmnet 3306: 1
db2037.codfw.wmnet 3306: 1
db2019.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306: 1
db1056.eqiad.wmnet 3306: 1
db1059.eqiad.wmnet 3306: 1
db1064.eqiad.wmnet 3306: 1
db1068.eqiad.wmnet 3306: 1
db1081.eqiad.wmnet 3306: 1
db1084.eqiad.wmnet 3306: 1
db1091.eqiad.wmnet 3306: 1
db1040.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2038.codfw.wmnet 3306: 1
db2045.codfw.wmnet 3306: 1
db2052.codfw.wmnet 3306: 1
db2059.codfw.wmnet 3306: 1
db2066.codfw.wmnet 3306: 1
db2023.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1026.eqiad.wmnet 3306: 1
db1045.eqiad.wmnet 3306: 1
db1070.eqiad.wmnet 3306: 1
db1071.eqiad.wmnet 3306: 1
db1082.eqiad.wmnet 3306: 1
db1087.eqiad.wmnet 3306: 1
db1092.eqiad.wmnet 3306: 1
db1049.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2039.codfw.wmnet 3306: 1
db2046.codfw.wmnet 3306: 1
db2053.codfw.wmnet 3306: 1
db2060.codfw.wmnet 3306: 1
db2067.codfw.wmnet 3306: 1
db2028.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306: 1
db1023.eqiad.wmnet 3306: 1
db1030.eqiad.wmnet 3306: 1
db1037.eqiad.wmnet 3306: 1
db1061.eqiad.wmnet 3306: 1
db1085.eqiad.wmnet 3306: 1
db1088.eqiad.wmnet 3306: 1
db1093.eqiad.wmnet 3306: 1
db1050.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2040.codfw.wmnet 3306: 1
db2047.codfw.wmnet 3306: 1
db2054.codfw.wmnet 3306: 1
db2061.codfw.wmnet 3306: 1
db2068.codfw.wmnet 3306: 1
db2029.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1028.eqiad.wmnet 3306: 1
db1033.eqiad.wmnet 3306: 1
db1034.eqiad.wmnet 3306: 1
db1039.eqiad.wmnet 3306: 1
db1062.eqiad.wmnet 3306: 1
db1079.eqiad.wmnet 3306: 1
db1086.eqiad.wmnet 3306: 1
db1094.eqiad.wmnet 3306: 1
db1041.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2033.codfw.wmnet 3306: 1
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1029.eqiad.wmnet 3306: 1
db1031.eqiad.wmnet 3306: 1
root@neodymium:~/software/dbtools$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -BN -h $host -P $port $db -e "SHOW STATUS like 'Ssl_cipher'"; done
db2034.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2042.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2048.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2055.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2062.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2069.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2070.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2016.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1080.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1083.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1089.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1073.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1072.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1066.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1065.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1055.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1051.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1047.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1057.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1052.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2035.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2041.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2049.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2056.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2063.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2064.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2017.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1021.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1024.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1036.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1047.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1054.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1060.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1063.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1067.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1074.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1076.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1090.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1018.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2036.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2043.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2050.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2057.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2018.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1035.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1038.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1044.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1077.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1078.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1075.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2065.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2058.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2051.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2044.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2037.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2019.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1056.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1059.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1064.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1068.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1081.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1084.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1091.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1040.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2038.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2045.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2052.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2059.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2066.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2023.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1026.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1045.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1070.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1071.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1082.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1087.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1092.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1049.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2039.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2046.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2053.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2060.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2067.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2028.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1023.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1030.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1037.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1061.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1085.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1088.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1093.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1050.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2040.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2047.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2054.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2061.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2068.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2029.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1028.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1033.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1034.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1039.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1062.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1079.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1086.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1094.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1041.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2033.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1029.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1031.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384

All single-shard hosts use it for replication, the dbstores and other multi-source replication's host have to restart its io thread to enable TLS:

$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -B -h $host -P $port $db -e "SHOW ALL SLAVES STATUS\G" | grep Master_SSL_Allowed; done
db2034.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2042.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2048.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2055.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2062.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2069.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2070.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2016.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1080.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1083.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1089.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1073.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1072.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1066.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1065.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1055.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1051.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1047.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
db1057.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1052.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2035.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2041.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2049.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2056.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2063.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2064.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2017.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
db1021.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1024.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1036.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1047.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
db1054.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1060.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1063.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1067.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1074.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1076.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1090.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1018.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2036.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2043.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2050.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2057.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2018.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1035.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1038.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1044.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1077.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1078.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1075.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2065.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2058.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2051.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2044.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2037.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2019.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1056.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1059.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1064.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1068.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1081.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1084.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1091.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1040.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2038.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2045.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2052.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2059.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2066.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2023.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1026.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1045.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1070.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1071.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1082.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1087.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1092.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1049.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2039.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2046.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2053.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2060.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2067.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2028.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1023.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1030.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1037.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1061.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1085.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1088.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1093.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1050.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2040.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2047.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2054.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2061.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2068.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2029.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1028.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1033.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1034.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1039.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1062.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1079.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1086.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1094.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1041.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2033.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1029.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1031.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes

Mentioned in SAL (#wikimedia-operations) [2017-02-09T16:06:23Z] <jynus> rolling restart of replication threads for dbstore1002/2001/2002 T111654

jcrespo closed this task as Resolved.Feb 9 2017, 5:08 PM
jcrespo claimed this task.

I have restarted all replication channels of dbstore1002/2001/2002 and db1047.

I consider this task resolved, with some follow-ups, less critical that I will handle on a separate ticket, with a different priority.