Set up TLS for MariaDB replication
Closed, ResolvedPublic

Description

MariaDB replication flows are currently not encrypted. They should be, as sometimes they cross datacenter boundaries. I think TLS is supported by MySQL/MariaDB and it would probably be the easiest way forward for this.

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 315049 had a related patch set uploaded (by Jcrespo):
Change phabricator misc dbs to use puppet TLS certificates

https://gerrit.wikimedia.org/r/315049

Above commands as of now:

$ sudo salt -C 'G@cluster:mysql and G@site:eqiad' cmd.run 'grep -l 'server\.key' /etc/my.cnf' | grep -c '/etc/my\.cnf'
102
$ sudo salt -C 'G@cluster:mysql and G@site:eqiad' cmd.run 'pt-config-diff --defaults-file=/root/.my.cnf --report-width=200 h=localhost /etc/my.cnf | grep "server\.key"' | grep -c 'server\.key'
20

Change 315049 merged by Jcrespo:
Change phabricator misc dbs to use puppet TLS certificates

https://gerrit.wikimedia.org/r/315049

Change 315051 had a related patch set uploaded (by Jcrespo):
Update phabricator my.cnf config template to include TLS config

https://gerrit.wikimedia.org/r/315051

Change 315051 merged by Jcrespo:
Update phabricator my.cnf config template to include TLS config

https://gerrit.wikimedia.org/r/315051

Change 319806 had a related patch set uploaded (by Jcrespo):
Allow SSL (TLS) and performance_schema on misc servers

https://gerrit.wikimedia.org/r/319806

Change 319806 merged by Jcrespo:
Allow SSL (TLS) and performance_schema on misc servers

https://gerrit.wikimedia.org/r/319806

Change 319831 had a related patch set uploaded (by Jcrespo):
Enable ssl (TLS) on misc database servers

https://gerrit.wikimedia.org/r/319831

Change 319831 merged by Jcrespo:
Enable ssl (TLS) on misc database servers

https://gerrit.wikimedia.org/r/319831

jcrespo moved this task from Backlog to Meta/Epic on the DBA board.Nov 10 2016, 12:26 PM
jcrespo lowered the priority of this task from High to Normal.Nov 24 2016, 11:43 AM

Out of 157 active hosts responding to salt, 15 host with no TLS deployed, 42 with the old certificate, 100 with the puppet one:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
15
$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
42
$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
100

List of eqiad hosts with the old cert:

db1015.eqiad.wmnet
db1021.eqiad.wmnet
db1022.eqiad.wmnet
db1036.eqiad.wmnet
db1054.eqiad.wmnet
db1060.eqiad.wmnet
db1063.eqiad.wmnet
db1067.eqiad.wmnet
db1074.eqiad.wmnet
db1076.eqiad.wmnet

db1046.eqiad.wmnet
db1047.eqiad.wmnet
dbstore1002.eqiad.wmnet
dbstore1001.eqiad.wmnet
labsdb1009.eqiad.wmnet
labsdb1010.eqiad.wmnet
labsdb1011.eqiad.wmnet

MySQLs wit no SSL

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
14

MySQL with expired TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
30

MySQL with latest TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
114

I have enabled TLS on neodymium and sarin, but because the mysql clients there are not using OpenSSL, clients will fail with:

ERROR 2026 (HY000): SSL connection error: unknown error number

I said we shouldn't, but we may have to create client packages after all, to allow for TLS 1.2 clients beyond the mysql servers.

Change 327703 had a related patch set uploaded (by Marostegui):
osc_host.sh: Added skip-ssl for the connection

https://gerrit.wikimedia.org/r/327703

Change 327703 merged by jenkins-bot:
osc_host.sh: Add skip-ssl for the connection

https://gerrit.wikimedia.org/r/327703

MySQLs with no SSL

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
13

MySQL with expired TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
26

MySQL with latest TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
120

Change 335227 had a related patch set uploaded (by Jcrespo):
sanitarium2: Enable TLS, disable Toku-specific config

https://gerrit.wikimedia.org/r/335227

Change 335227 merged by Jcrespo:
sanitarium2: Enable TLS, disable Toku-specific config

https://gerrit.wikimedia.org/r/335227

Change 335233 had a related patch set uploaded (by Jcrespo):
mariadb: Add TLS support for tendril

https://gerrit.wikimedia.org/r/335233

Change 335233 merged by Jcrespo:
mariadb: Add TLS support for tendril

https://gerrit.wikimedia.org/r/335233

Mentioned in SAL (#wikimedia-operations) [2017-01-31T17:37:10Z] <jynus> stopping mysql, upgrading and restarting db1011- temporary outage of tendril & dbtree T111654

36 pending hosts:

db1030.eqiad.wmnet: NULL
db1045.eqiad.wmnet: NULL
db1020.eqiad.wmnet: NULL
db1001.eqiad.wmnet: NULL
db1039.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
labsdb1001.eqiad.wmnet: NULL
labsdb1003.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
db1009.eqiad.wmnet: NULL
db1016.eqiad.wmnet: NULL
db1069.eqiad.wmnet: NULL
db1067.eqiad.wmnet: cacert
db1022.eqiad.wmnet: cacert
db1021.eqiad.wmnet: cacert
db1015.eqiad.wmnet: cacert
db1036.eqiad.wmnet: cacert
db2044.codfw.wmnet: cacert
db2063.codfw.wmnet: cacert
db2051.codfw.wmnet: cacert
db2046.codfw.wmnet: cacert
db2059.codfw.wmnet: cacert
db2065.codfw.wmnet: cacert
db2053.codfw.wmnet: cacert
db2039.codfw.wmnet: cacert
db2054.codfw.wmnet: cacert
db2061.codfw.wmnet: cacert
db2050.codfw.wmnet: cacert
db2041.codfw.wmnet: cacert
db2036.codfw.wmnet: cacert
db2037.codfw.wmnet: cacert
db2045.codfw.wmnet: cacert
db2052.codfw.wmnet: cacert
db2058.codfw.wmnet: cacert
db2064.codfw.wmnet: cacert
db2043.codfw.wmnet: cacert
jcrespo added a comment.EditedFeb 1 2017, 11:50 PM
sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
130

18 with expired certs:

db1021.eqiad.wmnet: cacert
db1022.eqiad.wmnet: cacert
db1036.eqiad.wmnet: cacert
db1015.eqiad.wmnet: cacert
db2052.codfw.wmnet: cacert
db2059.codfw.wmnet: cacert
db2063.codfw.wmnet: cacert
db2053.codfw.wmnet: cacert
db2054.codfw.wmnet: cacert
db2041.codfw.wmnet: cacert
db2061.codfw.wmnet: cacert
db2039.codfw.wmnet: cacert
db2046.codfw.wmnet: cacert
db2045.codfw.wmnet: cacert
db2036.codfw.wmnet: cacert
db2064.codfw.wmnet: cacert
db2050.codfw.wmnet: cacert
db2043.codfw.wmnet: cacert

Mentioned in SAL (#wikimedia-operations) [2017-02-02T17:43:54Z] <jynus> upgrade & restart of db2052 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-02T17:56:56Z] <jynus> upgrade & restart of db2059 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T09:54:08Z] <jynus> upgrade & restart of db2063 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T10:54:49Z] <jynus> preparing to reimage db2053 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T11:21:13Z] <jynus> preparing to reimage db2054 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T13:58:04Z] <jynus> restarting and upgrading db2041 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T14:30:58Z] <jynus> upgrade and restart db2061 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-03T15:01:47Z] <jynus> preparing to reimage db2039 T111654

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2039.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702031641_jynus_2666.log.

Completed auto-reimage of hosts:

['db2039.codfw.wmnet']

and were ALL successful.

After resolving T152188, pending hosts:

$ sudo salt --output=txt -C 'G@cluster:mysql' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"' | grep NULL
db1020.eqiad.wmnet: NULL
db1001.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
labsdb1003.eqiad.wmnet: NULL
db1009.eqiad.wmnet: NULL
db1016.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
db1030.eqiad.wmnet: NULL
labsdb1001.eqiad.wmnet: NULL
db1045.eqiad.wmnet: NULL
db1069.eqiad.wmnet: NULL

Some of those will be decomissioned very soon and probably never deployed TLS. Counting only core dbs:

$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"' | grep NULL
db1045.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
db1030.eqiad.wmnet: NULL

Change 336601 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1045 for maintenance

https://gerrit.wikimedia.org/r/336601

Change 336601 merged by jenkins-bot:
mariadb: Depool db1045 for maintenance

https://gerrit.wikimedia.org/r/336601

Mentioned in SAL (#wikimedia-operations) [2017-02-08T10:39:54Z] <jynus> upgrading and restarting db1045 T111654

Change 336609 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1037 for maintenance

https://gerrit.wikimedia.org/r/336609

Change 336609 merged by Jcrespo:
mariadb: Depool db1037 for maintenance

https://gerrit.wikimedia.org/r/336609

Mentioned in SAL (#wikimedia-operations) [2017-02-08T12:17:30Z] <jynus> upgrading and restarting db1037 T111654

Change 336620 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1026 for maintenance

https://gerrit.wikimedia.org/r/336620

Change 336620 merged by jenkins-bot:
mariadb: Depool db1026 for maintenance

https://gerrit.wikimedia.org/r/336620

Mentioned in SAL (#wikimedia-operations) [2017-02-08T14:17:32Z] <jynus> upgrading and restarting db1026 T111654

Change 336636 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1023 for maintenance

https://gerrit.wikimedia.org/r/336636

Change 336636 merged by jenkins-bot:
mariadb: Depool db1030 for maintenance

https://gerrit.wikimedia.org/r/336636

Mentioned in SAL (#wikimedia-operations) [2017-02-08T16:19:06Z] <jynus> upgrading and restarting db1030 T111654

TLS is now deployed on all core servers:

root@neodymium:~$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"'
db1071.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1041.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1030.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1045.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1037.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1076.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1022.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1067.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1092.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1074.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1063.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1029.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1084.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1082.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1091.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1018.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1015.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1077.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1080.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1012.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1090.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1061.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1089.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1083.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1052.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1013.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1068.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1060.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1081.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1011.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1094.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1053.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1086.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1087.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1054.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1051.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1070.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1085.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1073.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1078.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1079.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1093.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1075.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1015.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1088.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1016.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1065.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1033.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1028.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1057.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1056.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1062.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1050.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1024.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1035.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1059.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1014.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1026.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1023.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1018.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1039.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1019.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1064.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1040.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1031.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1021.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1036.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1044.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1034.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1066.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2029.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2011.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2018.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2067.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2038.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2019.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2041.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2019.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2013.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2063.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2045.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2017.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2065.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2064.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1049.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2048.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2058.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2042.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2039.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2012.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2033.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2034.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2054.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2043.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2061.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2014.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2057.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2070.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2049.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2046.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2040.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2060.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2037.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2069.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2066.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2047.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2044.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2053.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2036.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2056.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2068.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2050.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2055.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2062.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2059.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2035.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1017.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2016.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2018.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2028.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1072.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2051.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2052.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1055.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1038.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2016.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2023.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2017.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2015.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem

Enabling it on all pending hosts:

db1028.eqiad.wmnet:            Master_SSL_Allowed: No
db1077.eqiad.wmnet:            Master_SSL_Allowed: No
db1034.eqiad.wmnet:            Master_SSL_Allowed: No
db1067.eqiad.wmnet:            Master_SSL_Allowed: No
db1044.eqiad.wmnet:            Master_SSL_Allowed: No
es1013.eqiad.wmnet:            Master_SSL_Allowed: No
db1065.eqiad.wmnet:            Master_SSL_Allowed: No
es1017.eqiad.wmnet:            Master_SSL_Allowed: No
db1078.eqiad.wmnet:            Master_SSL_Allowed: No
db1055.eqiad.wmnet:            Master_SSL_Allowed: No
db2067.codfw.wmnet:            Master_SSL_Allowed: No
db2065.codfw.wmnet:            Master_SSL_Allowed: No
db2044.codfw.wmnet:            Master_SSL_Allowed: No
db2057.codfw.wmnet:            Master_SSL_Allowed: No
db2038.codfw.wmnet:            Master_SSL_Allowed: No
db2052.codfw.wmnet:            Master_SSL_Allowed: No
db2035.codfw.wmnet:            Master_SSL_Allowed: No
db2059.codfw.wmnet:            Master_SSL_Allowed: No
db2048.codfw.wmnet:            Master_SSL_Allowed: No
db2069.codfw.wmnet:            Master_SSL_Allowed: No
db2051.codfw.wmnet:            Master_SSL_Allowed: No
db2037.codfw.wmnet:            Master_SSL_Allowed: No
db2070.codfw.wmnet:            Master_SSL_Allowed: No
db2058.codfw.wmnet:            Master_SSL_Allowed: No
db2062.codfw.wmnet:            Master_SSL_Allowed: No
db2068.codfw.wmnet:            Master_SSL_Allowed: No
db2066.codfw.wmnet:            Master_SSL_Allowed: No
db2056.codfw.wmnet:            Master_SSL_Allowed: No
db2055.codfw.wmnet:            Master_SSL_Allowed: No

Mentioned in SAL (#wikimedia-operations) [2017-02-08T17:04:01Z] <jynus> rolling restart of replication thread of 29 mysql hosts T111654

Enabled everywhere except on db1034 and db2057, which probably require a package upgrade.

$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql --skip-ssl -e "SHOW SLAVE STATUS\G" | grep Master_SSL_Allowed'
db1040.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1037.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1077.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1084.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1045.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1078.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2018.codfw.wmnet:            Master_SSL_Allowed: Yes
db1094.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2037.codfw.wmnet:            Master_SSL_Allowed: Yes
db1036.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1083.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2048.codfw.wmnet:            Master_SSL_Allowed: Yes
db2046.codfw.wmnet:            Master_SSL_Allowed: Yes
db2069.codfw.wmnet:            Master_SSL_Allowed: Yes
db1072.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1051.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2019.codfw.wmnet:            Master_SSL_Allowed: Yes
db1056.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1071.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1091.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1029.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1052.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1031.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1068.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1050.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1082.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1055.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2065.codfw.wmnet:            Master_SSL_Allowed: Yes
db2044.codfw.wmnet:            Master_SSL_Allowed: Yes
db1076.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1092.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1061.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1059.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2060.codfw.wmnet:            Master_SSL_Allowed: Yes
db2053.codfw.wmnet:            Master_SSL_Allowed: Yes
db1086.eqiad.wmnet:            Master_SSL_Allowed: Yes
es1017.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2070.codfw.wmnet:            Master_SSL_Allowed: Yes
db2040.codfw.wmnet:            Master_SSL_Allowed: Yes
db2055.codfw.wmnet:            Master_SSL_Allowed: Yes
db2063.codfw.wmnet:            Master_SSL_Allowed: Yes
db2023.codfw.wmnet:            Master_SSL_Allowed: Yes
db2059.codfw.wmnet:            Master_SSL_Allowed: Yes
db1018.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1074.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2061.codfw.wmnet:            Master_SSL_Allowed: Yes
db1060.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2014.codfw.wmnet:            Master_SSL_Allowed: Yes
db1067.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2051.codfw.wmnet:            Master_SSL_Allowed: Yes
db1026.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2047.codfw.wmnet:            Master_SSL_Allowed: Yes
db1028.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1034.eqiad.wmnet:            Master_SSL_Allowed: No
db1015.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2064.codfw.wmnet:            Master_SSL_Allowed: Yes
db2054.codfw.wmnet:            Master_SSL_Allowed: Yes
db2019.codfw.wmnet:            Master_SSL_Allowed: Yes
db1030.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1088.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1038.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1063.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1080.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1049.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1053.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1079.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1089.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2067.codfw.wmnet:            Master_SSL_Allowed: Yes
db1064.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1073.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1057.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2045.codfw.wmnet:            Master_SSL_Allowed: Yes
es2016.codfw.wmnet:            Master_SSL_Allowed: Yes
db1024.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2015.codfw.wmnet:            Master_SSL_Allowed: Yes
db1093.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1085.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2029.codfw.wmnet:            Master_SSL_Allowed: Yes
db2038.codfw.wmnet:            Master_SSL_Allowed: Yes
db2058.codfw.wmnet:            Master_SSL_Allowed: Yes
db2049.codfw.wmnet:            Master_SSL_Allowed: Yes
db2052.codfw.wmnet:            Master_SSL_Allowed: Yes
db1065.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1033.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1070.eqiad.wmnet:            Master_SSL_Allowed: Yes
es1013.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1023.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1081.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1054.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1066.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1022.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2033.codfw.wmnet:            Master_SSL_Allowed: Yes
db2057.codfw.wmnet:            Master_SSL_Allowed: No
db2034.codfw.wmnet:            Master_SSL_Allowed: Yes
es1019.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1062.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1087.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2050.codfw.wmnet:            Master_SSL_Allowed: Yes
db2066.codfw.wmnet:            Master_SSL_Allowed: Yes
db2035.codfw.wmnet:            Master_SSL_Allowed: Yes
db2028.codfw.wmnet:            Master_SSL_Allowed: Yes
db2017.codfw.wmnet:            Master_SSL_Allowed: Yes
db2016.codfw.wmnet:            Master_SSL_Allowed: Yes
db2036.codfw.wmnet:            Master_SSL_Allowed: Yes
db2018.codfw.wmnet:            Master_SSL_Allowed: Yes
db1041.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1021.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2043.codfw.wmnet:            Master_SSL_Allowed: Yes
es2017.codfw.wmnet:            Master_SSL_Allowed: Yes
db2056.codfw.wmnet:            Master_SSL_Allowed: Yes
db2062.codfw.wmnet:            Master_SSL_Allowed: Yes
db2039.codfw.wmnet:            Master_SSL_Allowed: Yes
db1090.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1044.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1035.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2041.codfw.wmnet:            Master_SSL_Allowed: Yes
es1015.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2068.codfw.wmnet:            Master_SSL_Allowed: Yes
db1075.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1039.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2042.codfw.wmnet:            Master_SSL_Allowed: Yes

Change 336644 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db2057 for mariadb upgrade

https://gerrit.wikimedia.org/r/336644

TLS is now deployed on all core servers:

Congratulations, that was a massive and tedious effort.

Change 336644 merged by jenkins-bot:
mariadb: Depool db2057 for mariadb upgrade

https://gerrit.wikimedia.org/r/336644

Mentioned in SAL (#wikimedia-operations) [2017-02-08T18:02:21Z] <jynus> upgrading and restarting db2057 T111654

Change 336661 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1034 for maintenance

https://gerrit.wikimedia.org/r/336661

jcrespo added a comment.EditedFeb 8 2017, 7:07 PM

db1034 is left, pending of the reimage marked above^.

Of the non core hosts, only the following are left (all to be decommed, marking only as such the ones that already have replacements):

db1020.eqiad.wmnet: NULL - m2 master
db1009.eqiad.wmnet: NULL - m5 master
db1001.eqiad.wmnet: NULL - m1 slave
labsdb1001.eqiad.wmnet: NULL - to be decommed
labsdb1003.eqiad.wmnet: NULL - to be decommed
db1016.eqiad.wmnet: NULL - m1 master
db1069.eqiad.wmnet: NULL - to be decommed

Change 336661 merged by jenkins-bot:
mariadb: Depool db1034 for maintenance

https://gerrit.wikimedia.org/r/336661

Mentioned in SAL (#wikimedia-operations) [2017-02-09T09:38:56Z] <jynus> upgrading and restarting db1034 T111654

Mentioned in SAL (#wikimedia-operations) [2017-02-09T10:26:58Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1034 - T111654 (duration: 00m 41s)

Change 336774 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db2040 for maintenance

https://gerrit.wikimedia.org/r/336774

Change 336774 merged by jenkins-bot:
mariadb: Depool db2040 for maintenance

https://gerrit.wikimedia.org/r/336774

Mentioned in SAL (#wikimedia-operations) [2017-02-09T10:42:39Z] <jynus> preparing to reimage db2040 T111654

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2040.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702091223_jynus_7278.log.

Completed auto-reimage of hosts:

['db2040.codfw.wmnet']

and were ALL successful.

All core servers/server with core data now support TLS connections and use it for replication (except labs- the new server suport it, but are not accesible remotely for security, and the old ones, to be decommissioned, do not support it ):

$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -BN -h $host -P $port $db -e "SELECT 1"; done
db2034.codfw.wmnet 3306: 1
db2042.codfw.wmnet 3306: 1
db2048.codfw.wmnet 3306: 1
db2055.codfw.wmnet 3306: 1
db2062.codfw.wmnet 3306: 1
db2069.codfw.wmnet 3306: 1
db2070.codfw.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2016.codfw.wmnet 3306: 1
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1080.eqiad.wmnet 3306: 1
db1083.eqiad.wmnet 3306: 1
db1089.eqiad.wmnet 3306: 1
db1073.eqiad.wmnet 3306: 1
db1072.eqiad.wmnet 3306: 1
db1066.eqiad.wmnet 3306: 1
db1065.eqiad.wmnet 3306: 1
db1055.eqiad.wmnet 3306: 1
db1051.eqiad.wmnet 3306: 1
db1047.eqiad.wmnet 3306: 1
db1057.eqiad.wmnet 3306: 1
db1052.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2035.codfw.wmnet 3306: 1
db2041.codfw.wmnet 3306: 1
db2049.codfw.wmnet 3306: 1
db2056.codfw.wmnet 3306: 1
db2063.codfw.wmnet 3306: 1
db2064.codfw.wmnet 3306: 1
db2017.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306: 1
dbstore1001.eqiad.wmnet 3306: 1
db1021.eqiad.wmnet 3306: 1
db1024.eqiad.wmnet 3306: 1
db1036.eqiad.wmnet 3306: 1
db1047.eqiad.wmnet 3306: 1
db1054.eqiad.wmnet 3306: 1
db1060.eqiad.wmnet 3306: 1
db1063.eqiad.wmnet 3306: 1
db1067.eqiad.wmnet 3306: 1
db1074.eqiad.wmnet 3306: 1
db1076.eqiad.wmnet 3306: 1
db1090.eqiad.wmnet 3306: 1
db1018.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2036.codfw.wmnet 3306: 1
db2043.codfw.wmnet 3306: 1
db2050.codfw.wmnet 3306: 1
db2057.codfw.wmnet 3306: 1
db2018.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306: 1
db1035.eqiad.wmnet 3306: 1
db1038.eqiad.wmnet 3306: 1
db1044.eqiad.wmnet 3306: 1
db1077.eqiad.wmnet 3306: 1
db1078.eqiad.wmnet 3306: 1
db1075.eqiad.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
db2065.codfw.wmnet 3306: 1
db2058.codfw.wmnet 3306: 1
db2051.codfw.wmnet 3306: 1
db2044.codfw.wmnet 3306: 1
db2037.codfw.wmnet 3306: 1
db2019.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306: 1
db1056.eqiad.wmnet 3306: 1
db1059.eqiad.wmnet 3306: 1
db1064.eqiad.wmnet 3306: 1
db1068.eqiad.wmnet 3306: 1
db1081.eqiad.wmnet 3306: 1
db1084.eqiad.wmnet 3306: 1
db1091.eqiad.wmnet 3306: 1
db1040.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2038.codfw.wmnet 3306: 1
db2045.codfw.wmnet 3306: 1
db2052.codfw.wmnet 3306: 1
db2059.codfw.wmnet 3306: 1
db2066.codfw.wmnet 3306: 1
db2023.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1026.eqiad.wmnet 3306: 1
db1045.eqiad.wmnet 3306: 1
db1070.eqiad.wmnet 3306: 1
db1071.eqiad.wmnet 3306: 1
db1082.eqiad.wmnet 3306: 1
db1087.eqiad.wmnet 3306: 1
db1092.eqiad.wmnet 3306: 1
db1049.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2039.codfw.wmnet 3306: 1
db2046.codfw.wmnet 3306: 1
db2053.codfw.wmnet 3306: 1
db2060.codfw.wmnet 3306: 1
db2067.codfw.wmnet 3306: 1
db2028.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306: 1
db1023.eqiad.wmnet 3306: 1
db1030.eqiad.wmnet 3306: 1
db1037.eqiad.wmnet 3306: 1
db1061.eqiad.wmnet 3306: 1
db1085.eqiad.wmnet 3306: 1
db1088.eqiad.wmnet 3306: 1
db1093.eqiad.wmnet 3306: 1
db1050.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2040.codfw.wmnet 3306: 1
db2047.codfw.wmnet 3306: 1
db2054.codfw.wmnet 3306: 1
db2061.codfw.wmnet 3306: 1
db2068.codfw.wmnet 3306: 1
db2029.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1028.eqiad.wmnet 3306: 1
db1033.eqiad.wmnet 3306: 1
db1034.eqiad.wmnet 3306: 1
db1039.eqiad.wmnet 3306: 1
db1062.eqiad.wmnet 3306: 1
db1079.eqiad.wmnet 3306: 1
db1086.eqiad.wmnet 3306: 1
db1094.eqiad.wmnet 3306: 1
db1041.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2033.codfw.wmnet 3306: 1
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1029.eqiad.wmnet 3306: 1
db1031.eqiad.wmnet 3306: 1
root@neodymium:~/software/dbtools$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -BN -h $host -P $port $db -e "SHOW STATUS like 'Ssl_cipher'"; done
db2034.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2042.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2048.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2055.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2062.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2069.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2070.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2016.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1080.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1083.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1089.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1073.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1072.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1066.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1065.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1055.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1051.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1047.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1057.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1052.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2035.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2041.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2049.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2056.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2063.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2064.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2017.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1021.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1024.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1036.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1047.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1054.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1060.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1063.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1067.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1074.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1076.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1090.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1018.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2036.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2043.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2050.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2057.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2018.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1035.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1038.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1044.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1077.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1078.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1075.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2065.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2058.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2051.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2044.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2037.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2019.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1056.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1059.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1064.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1068.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1081.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1084.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1091.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1040.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2038.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2045.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2052.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2059.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2066.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2023.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1026.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1045.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1070.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1071.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1082.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1087.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1092.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1049.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2039.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2046.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2053.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2060.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2067.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2028.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1023.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1030.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1037.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1061.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1085.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1088.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1093.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1050.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2040.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2047.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2054.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2061.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2068.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2029.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1028.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1033.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1034.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1039.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1062.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1079.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1086.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1094.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1041.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2033.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1029.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1031.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384

All single-shard hosts use it for replication, the dbstores and other multi-source replication's host have to restart its io thread to enable TLS:

$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -B -h $host -P $port $db -e "SHOW ALL SLAVES STATUS\G" | grep Master_SSL_Allowed; done
db2034.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2042.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2048.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2055.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2062.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2069.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2070.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2016.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1080.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1083.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1089.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1073.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1072.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1066.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1065.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1055.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1051.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1047.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
db1057.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1052.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2035.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2041.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2049.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2056.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2063.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2064.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2017.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
db1021.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1024.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1036.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1047.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
db1054.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1060.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1063.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1067.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1074.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1076.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1090.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1018.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2036.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2043.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2050.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2057.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2018.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1035.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1038.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1044.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1077.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1078.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1075.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2065.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2058.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2051.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2044.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2037.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2019.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1056.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1059.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1064.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1068.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1081.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1084.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1091.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1040.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2038.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2045.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2052.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2059.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2066.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2023.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1026.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1045.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1070.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1071.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1082.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1087.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1092.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1049.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2039.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2046.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2053.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2060.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2067.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2028.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1023.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1030.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1037.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1061.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1085.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1088.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1093.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1050.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2040.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2047.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2054.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2061.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2068.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2029.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1028.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1033.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1034.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1039.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1062.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1079.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1086.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1094.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1041.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2033.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1029.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1031.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes

Mentioned in SAL (#wikimedia-operations) [2017-02-09T16:06:23Z] <jynus> rolling restart of replication threads for dbstore1002/2001/2002 T111654

jcrespo closed this task as Resolved.Feb 9 2017, 5:08 PM
jcrespo claimed this task.

I have restarted all replication channels of dbstore1002/2001/2002 and db1047.

I consider this task resolved, with some follow-ups, less critical that I will handle on a separate ticket, with a different priority.