Page MenuHomePhabricator
Create Task
Maniphest T111654

Set up TLS for MariaDB replication
Closed, ResolvedPublic
Actions

Description

MariaDB replication flows are currently not encrypted. They should be, as sometimes they cross datacenter boundaries. I think TLS is supported by MySQL/MariaDB and it would probably be the easiest way forward for this.

Details

SubjectRepoBranchLines +/-
mariadb: Depool db2040 for maintenanceoperations/mediawiki-configmaster+6 -6
mariadb: Depool db1034 for maintenanceoperations/mediawiki-configmaster+6 -6
mariadb: Depool db2057 for mariadb upgradeoperations/mediawiki-configmaster+1 -1
mariadb: Depool db1030 for maintenanceoperations/mediawiki-configmaster+4 -4
mariadb: Depool db1026 for maintenanceoperations/mediawiki-configmaster+7 -7
mariadb: Depool db1037 for maintenanceoperations/mediawiki-configmaster+7 -7
mariadb: Depool db1045 for maintenanceoperations/mediawiki-configmaster+4 -4
mariadb: Add TLS support for tendriloperations/puppetproduction+34 -0
sanitarium2: Enable TLS, disable Toku-specific configoperations/puppetproduction+32 -3
osc_host.sh: Add skip-ssl for the connectionoperations/softwaremaster+1 -1
Enable ssl (TLS) on misc database serversoperations/puppetproduction+1 -0
Allow SSL (TLS) and performance_schema on misc serversoperations/puppetproduction+52 -0
Update phabricator my.cnf config template to include TLS configoperations/puppetproduction+12 -0
Change phabricator misc dbs to use puppet TLS certificatesoperations/puppetproduction+1 -1
MariaDB: remove special SSL option multiple-caoperations/puppetproduction+0 -29
Remove temporary certificate with both CAsoperations/puppet/mariadbmaster+1 -20
MariaDB: Change mariadb::core parametersoperations/puppetproduction+12 -73
[WIP] Script to generate openssh TLS keys for mysql replicationoperations/softwaremaster+54 -0
[WIP] DB: Use generated CA for the TLS transitionoperations/puppetproduction+11 -11
Skip SSL options when using socket connectionoperations/puppet/mariadbmaster+5 -1
MariaDB: complete TLS and master configurationoperations/puppetproduction+58 -48
Repool with low weight new s3 DB after TLS upgradeoperations/mediawiki-configmaster+3 -3
MariaDB: use Puppet cert for s3 cross-dc replicaoperations/puppetproduction+11 -1
Depool new s3 slaves for TLS upgradeoperations/mediawiki-configmaster+3 -3
Repool db1042 after TLS upgradeoperations/mediawiki-configmaster+4 -4
MariaDB: use Puppet cert for s4 cross-dc replicaoperations/puppetproduction+2 -0
Depool db1042 for TLS upgradeoperations/mediawiki-configmaster+4 -4
Repool db1057 after TLS upgrade on s1operations/mediawiki-configmaster+1 -1
MariaDB: use Puppet cert for s1 cross-dc replicaoperations/puppetproduction+11 -1
Depool db1057 for TLS upgrade for s1operations/mediawiki-configmaster+1 -1
MariaDB: Update submodule referenceoperations/puppetproduction+1 -1
Remove require on resource not managed by Puppetoperations/puppet/mariadbmaster+1 -4
MariaDB: Use Puppet certs for s2operations/puppetproduction+11 -1
Reduce db1050 weightoperations/mediawiki-configmaster+1 -1
Repool db1050 and db1041 for TLS upgradeoperations/mediawiki-configmaster+2 -2
MariaDB: use Puppet certs for TLSoperations/puppetproduction+13 -1
MW: remove mw1080, has RO filesystemoperations/puppetproduction+4 -6
Depool db1050 and db1041 for TLS upgradeoperations/mediawiki-configmaster+2 -2
Repool db1049 for vslow and dumpoperations/mediawiki-configmaster+4 -4
MariaDB: use Puppet certs for s5 cross-dc replicaoperations/puppetproduction+2 -0
Depool db1049 to deploy Pupper certs for TLSoperations/mediawiki-configmaster+4 -4
MariaDB: updated submodule referenceoperations/puppetproduction+1 -1
Removed require that is not directly managed by Puppetoperations/puppet/mariadbmaster+0 -1
MariaDB: allow multiple MySQL TLS configurationsoperations/puppetproduction+27 -1
Allow a third value to use the Puppet certsoperations/puppet/mariadbmaster+2 -1
DB: Expose Puppet SSL certs and generate CA certoperations/puppetproduction+8 -1
Resolve conflict on /etc/mysql/ssl directoryoperations/puppet/mariadbmaster+7 -17
Refactor of the CAs certificate genearationoperations/puppet/mariadbmaster+13 -17
DB: Expose Puppet SSL certs and generate CA certoperations/puppet/mariadbmaster+34 -1
Enable TLS by default on all core mysql hostsoperations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Volans mentioned this in rOPUP6fa89090fa32: MariaDB: remove special SSL option multiple-ca.Jun 17 2016, 6:09 PM
Volans mentioned this in rOPUP86f5e8be0498: MariaDB: remove special SSL option multiple-ca.
Volans mentioned this in rOPUP3f0b430080e4: MariaDB: Change mariadb::core parameters.
Volans mentioned this in rOPUP10210e7acb73: MariaDB: Change default ssl mode to puppet-cert.
Volans mentioned this in rOPUPa23a3b172704: MariaDB: Change mariadb::core parameters.
gerritbot added a comment.Oct 10 2016, 8:39 AM

Change 315049 had a related patch set uploaded (by Jcrespo):
Change phabricator misc dbs to use puppet TLS certificates

https://gerrit.wikimedia.org/r/315049

jcrespo added a subscriber: Marostegui.Oct 10 2016, 8:42 AM

Above commands as of now:

$ sudo salt -C 'G@cluster:mysql and G@site:eqiad' cmd.run 'grep -l 'server\.key' /etc/my.cnf' | grep -c '/etc/my\.cnf'
102
$ sudo salt -C 'G@cluster:mysql and G@site:eqiad' cmd.run 'pt-config-diff --defaults-file=/root/.my.cnf --report-width=200 h=localhost /etc/my.cnf | grep "server\.key"' | grep -c 'server\.key'
20
gerritbot added a comment.Oct 10 2016, 8:53 AM

Change 315049 merged by Jcrespo:
Change phabricator misc dbs to use puppet TLS certificates

https://gerrit.wikimedia.org/r/315049

gerritbot added a comment.Oct 10 2016, 9:01 AM

Change 315051 had a related patch set uploaded (by Jcrespo):
Update phabricator my.cnf config template to include TLS config

https://gerrit.wikimedia.org/r/315051

gerritbot added a comment.Oct 10 2016, 9:25 AM

Change 315051 merged by Jcrespo:
Update phabricator my.cnf config template to include TLS config

https://gerrit.wikimedia.org/r/315051

gerritbot added a comment.Nov 4 2016, 9:28 AM

Change 319806 had a related patch set uploaded (by Jcrespo):
Allow SSL (TLS) and performance_schema on misc servers

https://gerrit.wikimedia.org/r/319806

gerritbot added a comment.Nov 4 2016, 10:48 AM

Change 319806 merged by Jcrespo:
Allow SSL (TLS) and performance_schema on misc servers

https://gerrit.wikimedia.org/r/319806

gerritbot added a comment.Nov 4 2016, 12:01 PM

Change 319831 had a related patch set uploaded (by Jcrespo):
Enable ssl (TLS) on misc database servers

https://gerrit.wikimedia.org/r/319831

gerritbot added a comment.Nov 4 2016, 12:03 PM

Change 319831 merged by Jcrespo:
Enable ssl (TLS) on misc database servers

https://gerrit.wikimedia.org/r/319831

jcrespo moved this task from Backlog to Meta/Epic on the DBA board.Nov 10 2016, 12:26 PM
fgiunchedi mentioned this in T150822: Internal PKI for secure communication - Barcelona Ops offsite 2016.Nov 16 2016, 1:23 AM
jcrespo lowered the priority of this task from High to Medium.Nov 24 2016, 11:43 AM
jcrespo created subtask T151583: Use tls for dump backup generation.Nov 24 2016, 6:50 PM
Volans mentioned this in T150823: Puppet CA rollover.Nov 25 2016, 9:46 AM
jcrespo created subtask T151995: Rolling restart of external storage servers for TLS certificate update.Nov 30 2016, 11:34 AM
jcrespo created subtask T152029: Rolling restart of parsercache servers for TLS certificate update.Nov 30 2016, 7:28 PM
jcrespo closed subtask T151995: Rolling restart of external storage servers for TLS certificate update as Resolved.Dec 1 2016, 7:04 PM
jcrespo closed subtask T152029: Rolling restart of parsercache servers for TLS certificate update as Resolved.Dec 1 2016, 7:53 PM
jcrespo added a comment.Dec 1 2016, 8:05 PM

Out of 157 active hosts responding to salt, 15 host with no TLS deployed, 42 with the old certificate, 100 with the puppet one:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
15
$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
42
$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
100
jcrespo added a comment.Dec 1 2016, 8:15 PM

List of eqiad hosts with the old cert:

db1015.eqiad.wmnet
db1021.eqiad.wmnet
db1022.eqiad.wmnet
db1036.eqiad.wmnet
db1054.eqiad.wmnet
db1060.eqiad.wmnet
db1063.eqiad.wmnet
db1067.eqiad.wmnet
db1074.eqiad.wmnet
db1076.eqiad.wmnet

db1046.eqiad.wmnet
db1047.eqiad.wmnet
dbstore1002.eqiad.wmnet
dbstore1001.eqiad.wmnet
labsdb1009.eqiad.wmnet
labsdb1010.eqiad.wmnet
labsdb1011.eqiad.wmnet
jcrespo created subtask T152188: Restart pending mysql hosts with old TLS cert.Dec 2 2016, 8:56 AM
jcrespo added a subtask: T152595: Implement TLS expiration/validation checking for MariaDB certificates.Dec 7 2016, 9:19 AM
jcrespo added a subtask: T152427: Create a check/calendar alert for MariaDB TLS certs.Dec 7 2016, 9:25 AM
jcrespo added a comment.Dec 14 2016, 6:17 PM

MySQLs wit no SSL

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
14

MySQL with expired TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
30

MySQL with latest TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
114
jcrespo removed a subtask: T151583: Use tls for dump backup generation.Dec 14 2016, 6:17 PM
jcrespo added a parent task: T151583: Use tls for dump backup generation.
jcrespo added a comment.Dec 15 2016, 6:09 PM

I have enabled TLS on neodymium and sarin, but because the mysql clients there are not using OpenSSL, clients will fail with:

ERROR 2026 (HY000): SSL connection error: unknown error number

I said we shouldn't, but we may have to create client packages after all, to allow for TLS 1.2 clients beyond the mysql servers.

gerritbot added a comment.Dec 16 2016, 7:40 AM

Change 327703 had a related patch set uploaded (by Marostegui):
osc_host.sh: Added skip-ssl for the connection

https://gerrit.wikimedia.org/r/327703

gerritbot added a comment.Dec 16 2016, 9:04 AM

Change 327703 merged by jenkins-bot:
osc_host.sh: Add skip-ssl for the connection

https://gerrit.wikimedia.org/r/327703

jcrespo added a subtask: T156005: Reimage db1065 and db1066.Jan 23 2017, 4:22 PM
jcrespo added a comment.Jan 25 2017, 2:29 AM

MySQLs with no SSL

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'NULL'
13

MySQL with expired TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'cacert.pem'
26

MySQL with latest TLS cert:

$ sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
120
Marostegui closed subtask T156005: Reimage db1065 and db1066 as Resolved.Jan 25 2017, 6:55 AM
gerritbot added a comment.Jan 31 2017, 1:48 PM

Change 335227 had a related patch set uploaded (by Jcrespo):
sanitarium2: Enable TLS, disable Toku-specific config

https://gerrit.wikimedia.org/r/335227

gerritbot added a comment.Jan 31 2017, 2:31 PM

Change 335227 merged by Jcrespo:
sanitarium2: Enable TLS, disable Toku-specific config

https://gerrit.wikimedia.org/r/335227

gerritbot added a comment.Jan 31 2017, 3:09 PM

Change 335233 had a related patch set uploaded (by Jcrespo):
mariadb: Add TLS support for tendril

https://gerrit.wikimedia.org/r/335233

gerritbot added a comment.Jan 31 2017, 5:26 PM

Change 335233 merged by Jcrespo:
mariadb: Add TLS support for tendril

https://gerrit.wikimedia.org/r/335233

Stashbot added a comment.Jan 31 2017, 5:37 PM

Mentioned in SAL (#wikimedia-operations) [2017-01-31T17:37:10Z] <jynus> stopping mysql, upgrading and restarting db1011- temporary outage of tendril & dbtree T111654

jcrespo added a comment.Feb 1 2017, 12:47 AM

36 pending hosts:

db1030.eqiad.wmnet: NULL
db1045.eqiad.wmnet: NULL
db1020.eqiad.wmnet: NULL
db1001.eqiad.wmnet: NULL
db1039.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
labsdb1001.eqiad.wmnet: NULL
labsdb1003.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
db1009.eqiad.wmnet: NULL
db1016.eqiad.wmnet: NULL
db1069.eqiad.wmnet: NULL
db1067.eqiad.wmnet: cacert
db1022.eqiad.wmnet: cacert
db1021.eqiad.wmnet: cacert
db1015.eqiad.wmnet: cacert
db1036.eqiad.wmnet: cacert
db2044.codfw.wmnet: cacert
db2063.codfw.wmnet: cacert
db2051.codfw.wmnet: cacert
db2046.codfw.wmnet: cacert
db2059.codfw.wmnet: cacert
db2065.codfw.wmnet: cacert
db2053.codfw.wmnet: cacert
db2039.codfw.wmnet: cacert
db2054.codfw.wmnet: cacert
db2061.codfw.wmnet: cacert
db2050.codfw.wmnet: cacert
db2041.codfw.wmnet: cacert
db2036.codfw.wmnet: cacert
db2037.codfw.wmnet: cacert
db2045.codfw.wmnet: cacert
db2052.codfw.wmnet: cacert
db2058.codfw.wmnet: cacert
db2064.codfw.wmnet: cacert
db2043.codfw.wmnet: cacert
jcrespo added a comment.EditedFeb 1 2017, 11:50 PM
sudo salt -C 'G@cluster:mysql' cmd.run 'mysql --skip-ssl -e "SELECT @@ssl_ca"' | grep -c 'Puppet'
130

18 with expired certs:

db1021.eqiad.wmnet: cacert
db1022.eqiad.wmnet: cacert
db1036.eqiad.wmnet: cacert
db1015.eqiad.wmnet: cacert
db2052.codfw.wmnet: cacert
db2059.codfw.wmnet: cacert
db2063.codfw.wmnet: cacert
db2053.codfw.wmnet: cacert
db2054.codfw.wmnet: cacert
db2041.codfw.wmnet: cacert
db2061.codfw.wmnet: cacert
db2039.codfw.wmnet: cacert
db2046.codfw.wmnet: cacert
db2045.codfw.wmnet: cacert
db2036.codfw.wmnet: cacert
db2064.codfw.wmnet: cacert
db2050.codfw.wmnet: cacert
db2043.codfw.wmnet: cacert
Stashbot added a comment.Feb 2 2017, 5:43 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-02T17:43:54Z] <jynus> upgrade & restart of db2052 T111654

Stashbot added a comment.Feb 2 2017, 5:56 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-02T17:56:56Z] <jynus> upgrade & restart of db2059 T111654

Stashbot added a comment.Feb 3 2017, 9:54 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-03T09:54:08Z] <jynus> upgrade & restart of db2063 T111654

Stashbot added a comment.Feb 3 2017, 10:54 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-03T10:54:49Z] <jynus> preparing to reimage db2053 T111654

Stashbot added a comment.Feb 3 2017, 11:21 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-03T11:21:13Z] <jynus> preparing to reimage db2054 T111654

Stashbot added a comment.Feb 3 2017, 1:58 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-03T13:58:04Z] <jynus> restarting and upgrading db2041 T111654

Stashbot added a comment.Feb 3 2017, 2:30 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-03T14:30:58Z] <jynus> upgrade and restart db2061 T111654

Stashbot added a comment.Feb 3 2017, 3:01 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-03T15:01:47Z] <jynus> preparing to reimage db2039 T111654

ops-monitoring-bot subscribed.Feb 3 2017, 4:42 PM

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2039.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702031641_jynus_2666.log.

ops-monitoring-bot added a comment.Feb 3 2017, 5:09 PM

Completed auto-reimage of hosts:

['db2039.codfw.wmnet']

and were ALL successful.

jcrespo mentioned this in T152188: Restart pending mysql hosts with old TLS cert.Feb 8 2017, 12:24 AM
jcrespo closed subtask T152188: Restart pending mysql hosts with old TLS cert as Resolved.
jcrespo added a comment.Feb 8 2017, 12:28 AM

After resolving T152188, pending hosts:

$ sudo salt --output=txt -C 'G@cluster:mysql' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"' | grep NULL
db1020.eqiad.wmnet: NULL
db1001.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
labsdb1003.eqiad.wmnet: NULL
db1009.eqiad.wmnet: NULL
db1016.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
db1030.eqiad.wmnet: NULL
labsdb1001.eqiad.wmnet: NULL
db1045.eqiad.wmnet: NULL
db1069.eqiad.wmnet: NULL

Some of those will be decomissioned very soon and probably never deployed TLS. Counting only core dbs:

$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"' | grep NULL
db1045.eqiad.wmnet: NULL
db1037.eqiad.wmnet: NULL
db1026.eqiad.wmnet: NULL
db1030.eqiad.wmnet: NULL
gerritbot added a comment.Feb 8 2017, 10:20 AM

Change 336601 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1045 for maintenance

https://gerrit.wikimedia.org/r/336601

gerritbot added a comment.Feb 8 2017, 10:34 AM

Change 336601 merged by jenkins-bot:
mariadb: Depool db1045 for maintenance

https://gerrit.wikimedia.org/r/336601

Stashbot added a comment.Feb 8 2017, 10:39 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-08T10:39:54Z] <jynus> upgrading and restarting db1045 T111654

gerritbot added a comment.Feb 8 2017, 12:09 PM

Change 336609 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1037 for maintenance

https://gerrit.wikimedia.org/r/336609

gerritbot added a comment.Feb 8 2017, 12:12 PM

Change 336609 merged by Jcrespo:
mariadb: Depool db1037 for maintenance

https://gerrit.wikimedia.org/r/336609

Stashbot added a comment.Feb 8 2017, 12:17 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-08T12:17:30Z] <jynus> upgrading and restarting db1037 T111654

gerritbot added a comment.Feb 8 2017, 12:51 PM

Change 336620 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1026 for maintenance

https://gerrit.wikimedia.org/r/336620

gerritbot added a comment.Feb 8 2017, 1:37 PM

Change 336620 merged by jenkins-bot:
mariadb: Depool db1026 for maintenance

https://gerrit.wikimedia.org/r/336620

Stashbot added a comment.Feb 8 2017, 2:17 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-08T14:17:32Z] <jynus> upgrading and restarting db1026 T111654

gerritbot added a comment.Feb 8 2017, 4:02 PM

Change 336636 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1023 for maintenance

https://gerrit.wikimedia.org/r/336636

gerritbot added a comment.Feb 8 2017, 4:09 PM

Change 336636 merged by jenkins-bot:
mariadb: Depool db1030 for maintenance

https://gerrit.wikimedia.org/r/336636

Stashbot added a comment.Feb 8 2017, 4:19 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-08T16:19:06Z] <jynus> upgrading and restarting db1030 T111654

jcrespo added a comment.Feb 8 2017, 4:51 PM

TLS is now deployed on all core servers:

root@neodymium:~$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql -BN --skip-ssl -e "SELECT @@ssl_ca"'
db1071.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1041.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1030.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1045.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1037.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1076.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1022.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1067.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1092.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1074.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1063.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1029.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1084.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1082.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1091.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1018.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1015.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1077.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1080.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1012.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1090.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1061.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1089.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1083.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1052.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1013.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1068.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1060.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1081.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1011.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1094.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1053.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1086.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1087.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1054.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1051.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1070.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1085.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1073.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1078.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1079.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1093.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1075.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1015.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1088.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1016.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1065.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1033.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1028.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1057.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1056.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1062.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1050.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1024.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1035.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1059.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1014.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1026.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1023.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1018.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1039.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1019.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1064.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1040.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1031.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1021.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1036.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1044.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1034.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1066.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2029.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2011.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2018.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2067.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2038.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2019.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2041.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2019.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2013.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2063.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2045.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2017.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2065.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2064.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1049.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2048.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2058.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2042.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2039.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2012.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2033.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2034.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2054.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2043.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2061.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2014.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2057.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2070.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2049.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2046.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2040.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2060.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2037.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2069.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2066.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2047.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2044.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2053.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2036.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2056.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2068.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2050.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2055.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2062.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2059.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2035.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es1017.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2016.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2018.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2028.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1072.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2051.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2052.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1055.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db1038.eqiad.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2016.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
db2023.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2017.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem
es2015.codfw.wmnet: /etc/ssl/certs/Puppet_Internal_CA.pem

Enabling it on all pending hosts:

db1028.eqiad.wmnet:            Master_SSL_Allowed: No
db1077.eqiad.wmnet:            Master_SSL_Allowed: No
db1034.eqiad.wmnet:            Master_SSL_Allowed: No
db1067.eqiad.wmnet:            Master_SSL_Allowed: No
db1044.eqiad.wmnet:            Master_SSL_Allowed: No
es1013.eqiad.wmnet:            Master_SSL_Allowed: No
db1065.eqiad.wmnet:            Master_SSL_Allowed: No
es1017.eqiad.wmnet:            Master_SSL_Allowed: No
db1078.eqiad.wmnet:            Master_SSL_Allowed: No
db1055.eqiad.wmnet:            Master_SSL_Allowed: No
db2067.codfw.wmnet:            Master_SSL_Allowed: No
db2065.codfw.wmnet:            Master_SSL_Allowed: No
db2044.codfw.wmnet:            Master_SSL_Allowed: No
db2057.codfw.wmnet:            Master_SSL_Allowed: No
db2038.codfw.wmnet:            Master_SSL_Allowed: No
db2052.codfw.wmnet:            Master_SSL_Allowed: No
db2035.codfw.wmnet:            Master_SSL_Allowed: No
db2059.codfw.wmnet:            Master_SSL_Allowed: No
db2048.codfw.wmnet:            Master_SSL_Allowed: No
db2069.codfw.wmnet:            Master_SSL_Allowed: No
db2051.codfw.wmnet:            Master_SSL_Allowed: No
db2037.codfw.wmnet:            Master_SSL_Allowed: No
db2070.codfw.wmnet:            Master_SSL_Allowed: No
db2058.codfw.wmnet:            Master_SSL_Allowed: No
db2062.codfw.wmnet:            Master_SSL_Allowed: No
db2068.codfw.wmnet:            Master_SSL_Allowed: No
db2066.codfw.wmnet:            Master_SSL_Allowed: No
db2056.codfw.wmnet:            Master_SSL_Allowed: No
db2055.codfw.wmnet:            Master_SSL_Allowed: No
Stashbot added a comment.Feb 8 2017, 5:04 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-08T17:04:01Z] <jynus> rolling restart of replication thread of 29 mysql hosts T111654

jcrespo added a comment.Feb 8 2017, 5:17 PM

Enabled everywhere except on db1034 and db2057, which probably require a package upgrade.

$ sudo salt --output=txt -C 'G@cluster:mysql and G@mysql_group:core' cmd.run 'mysql --skip-ssl -e "SHOW SLAVE STATUS\G" | grep Master_SSL_Allowed'
db1040.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1037.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1077.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1084.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1045.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1078.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2018.codfw.wmnet:            Master_SSL_Allowed: Yes
db1094.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2037.codfw.wmnet:            Master_SSL_Allowed: Yes
db1036.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1083.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2048.codfw.wmnet:            Master_SSL_Allowed: Yes
db2046.codfw.wmnet:            Master_SSL_Allowed: Yes
db2069.codfw.wmnet:            Master_SSL_Allowed: Yes
db1072.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1051.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2019.codfw.wmnet:            Master_SSL_Allowed: Yes
db1056.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1071.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1091.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1029.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1052.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1031.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1068.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1050.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1082.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1055.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2065.codfw.wmnet:            Master_SSL_Allowed: Yes
db2044.codfw.wmnet:            Master_SSL_Allowed: Yes
db1076.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1092.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1061.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1059.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2060.codfw.wmnet:            Master_SSL_Allowed: Yes
db2053.codfw.wmnet:            Master_SSL_Allowed: Yes
db1086.eqiad.wmnet:            Master_SSL_Allowed: Yes
es1017.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2070.codfw.wmnet:            Master_SSL_Allowed: Yes
db2040.codfw.wmnet:            Master_SSL_Allowed: Yes
db2055.codfw.wmnet:            Master_SSL_Allowed: Yes
db2063.codfw.wmnet:            Master_SSL_Allowed: Yes
db2023.codfw.wmnet:            Master_SSL_Allowed: Yes
db2059.codfw.wmnet:            Master_SSL_Allowed: Yes
db1018.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1074.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2061.codfw.wmnet:            Master_SSL_Allowed: Yes
db1060.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2014.codfw.wmnet:            Master_SSL_Allowed: Yes
db1067.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2051.codfw.wmnet:            Master_SSL_Allowed: Yes
db1026.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2047.codfw.wmnet:            Master_SSL_Allowed: Yes
db1028.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1034.eqiad.wmnet:            Master_SSL_Allowed: No
db1015.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2064.codfw.wmnet:            Master_SSL_Allowed: Yes
db2054.codfw.wmnet:            Master_SSL_Allowed: Yes
db2019.codfw.wmnet:            Master_SSL_Allowed: Yes
db1030.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1088.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1038.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1063.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1080.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1049.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1053.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1079.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1089.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2067.codfw.wmnet:            Master_SSL_Allowed: Yes
db1064.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1073.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1057.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2045.codfw.wmnet:            Master_SSL_Allowed: Yes
es2016.codfw.wmnet:            Master_SSL_Allowed: Yes
db1024.eqiad.wmnet:            Master_SSL_Allowed: Yes
es2015.codfw.wmnet:            Master_SSL_Allowed: Yes
db1093.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1085.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2029.codfw.wmnet:            Master_SSL_Allowed: Yes
db2038.codfw.wmnet:            Master_SSL_Allowed: Yes
db2058.codfw.wmnet:            Master_SSL_Allowed: Yes
db2049.codfw.wmnet:            Master_SSL_Allowed: Yes
db2052.codfw.wmnet:            Master_SSL_Allowed: Yes
db1065.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1033.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1070.eqiad.wmnet:            Master_SSL_Allowed: Yes
es1013.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1023.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1081.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1054.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1066.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1022.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2033.codfw.wmnet:            Master_SSL_Allowed: Yes
db2057.codfw.wmnet:            Master_SSL_Allowed: No
db2034.codfw.wmnet:            Master_SSL_Allowed: Yes
es1019.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1062.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1087.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2050.codfw.wmnet:            Master_SSL_Allowed: Yes
db2066.codfw.wmnet:            Master_SSL_Allowed: Yes
db2035.codfw.wmnet:            Master_SSL_Allowed: Yes
db2028.codfw.wmnet:            Master_SSL_Allowed: Yes
db2017.codfw.wmnet:            Master_SSL_Allowed: Yes
db2016.codfw.wmnet:            Master_SSL_Allowed: Yes
db2036.codfw.wmnet:            Master_SSL_Allowed: Yes
db2018.codfw.wmnet:            Master_SSL_Allowed: Yes
db1041.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1021.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2043.codfw.wmnet:            Master_SSL_Allowed: Yes
es2017.codfw.wmnet:            Master_SSL_Allowed: Yes
db2056.codfw.wmnet:            Master_SSL_Allowed: Yes
db2062.codfw.wmnet:            Master_SSL_Allowed: Yes
db2039.codfw.wmnet:            Master_SSL_Allowed: Yes
db1090.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1044.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1035.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2041.codfw.wmnet:            Master_SSL_Allowed: Yes
es1015.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2068.codfw.wmnet:            Master_SSL_Allowed: Yes
db1075.eqiad.wmnet:            Master_SSL_Allowed: Yes
db1039.eqiad.wmnet:            Master_SSL_Allowed: Yes
db2042.codfw.wmnet:            Master_SSL_Allowed: Yes
gerritbot added a comment.Feb 8 2017, 5:23 PM

Change 336644 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db2057 for mariadb upgrade

https://gerrit.wikimedia.org/r/336644

Marostegui added a comment.Feb 8 2017, 5:25 PM
In T111654#3009760, @jcrespo wrote:

TLS is now deployed on all core servers:

Congratulations, that was a massive and tedious effort.

gerritbot added a comment.Feb 8 2017, 5:43 PM

Change 336644 merged by jenkins-bot:
mariadb: Depool db2057 for mariadb upgrade

https://gerrit.wikimedia.org/r/336644

Stashbot added a comment.Feb 8 2017, 6:02 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-08T18:02:21Z] <jynus> upgrading and restarting db2057 T111654

gerritbot added a comment.Feb 8 2017, 6:57 PM

Change 336661 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db1034 for maintenance

https://gerrit.wikimedia.org/r/336661

jcrespo added a comment.EditedFeb 8 2017, 7:07 PM

db1034 is left, pending of the reimage marked above^.

Of the non core hosts, only the following are left (all to be decommed, marking only as such the ones that already have replacements):

db1020.eqiad.wmnet: NULL - m2 master
db1009.eqiad.wmnet: NULL - m5 master
db1001.eqiad.wmnet: NULL - m1 slave
labsdb1001.eqiad.wmnet: NULL - to be decommed
labsdb1003.eqiad.wmnet: NULL - to be decommed
db1016.eqiad.wmnet: NULL - m1 master
db1069.eqiad.wmnet: NULL - to be decommed

gerritbot added a comment.Feb 9 2017, 9:24 AM

Change 336661 merged by jenkins-bot:
mariadb: Depool db1034 for maintenance

https://gerrit.wikimedia.org/r/336661

Stashbot added a comment.Feb 9 2017, 9:38 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-09T09:38:56Z] <jynus> upgrading and restarting db1034 T111654

Stashbot added a comment.Feb 9 2017, 10:26 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-09T10:26:58Z] <marostegui@tin> Synchronized wmf-config/db-eqiad.php: Repool db1034 - T111654 (duration: 00m 41s)

gerritbot added a comment.Feb 9 2017, 10:29 AM

Change 336774 had a related patch set uploaded (by Jcrespo):
mariadb: Depool db2040 for maintenance

https://gerrit.wikimedia.org/r/336774

gerritbot added a comment.Feb 9 2017, 10:36 AM

Change 336774 merged by jenkins-bot:
mariadb: Depool db2040 for maintenance

https://gerrit.wikimedia.org/r/336774

Stashbot added a comment.Feb 9 2017, 10:42 AM

Mentioned in SAL (#wikimedia-operations) [2017-02-09T10:42:39Z] <jynus> preparing to reimage db2040 T111654

ops-monitoring-bot added a comment.Feb 9 2017, 12:23 PM

Script wmf_auto_reimage was launched by jynus on neodymium.eqiad.wmnet for hosts:

['db2040.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201702091223_jynus_7278.log.

ops-monitoring-bot added a comment.Feb 9 2017, 1:57 PM

Completed auto-reimage of hosts:

['db2040.codfw.wmnet']

and were ALL successful.

jcrespo added a comment.Feb 9 2017, 3:21 PM

All core servers/server with core data now support TLS connections and use it for replication (except labs- the new server suport it, but are not accesible remotely for security, and the old ones, to be decommissioned, do not support it ):

$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -BN -h $host -P $port $db -e "SELECT 1"; done
db2034.codfw.wmnet 3306: 1
db2042.codfw.wmnet 3306: 1
db2048.codfw.wmnet 3306: 1
db2055.codfw.wmnet 3306: 1
db2062.codfw.wmnet 3306: 1
db2069.codfw.wmnet 3306: 1
db2070.codfw.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2016.codfw.wmnet 3306: 1
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1080.eqiad.wmnet 3306: 1
db1083.eqiad.wmnet 3306: 1
db1089.eqiad.wmnet 3306: 1
db1073.eqiad.wmnet 3306: 1
db1072.eqiad.wmnet 3306: 1
db1066.eqiad.wmnet 3306: 1
db1065.eqiad.wmnet 3306: 1
db1055.eqiad.wmnet 3306: 1
db1051.eqiad.wmnet 3306: 1
db1047.eqiad.wmnet 3306: 1
db1057.eqiad.wmnet 3306: 1
db1052.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2035.codfw.wmnet 3306: 1
db2041.codfw.wmnet 3306: 1
db2049.codfw.wmnet 3306: 1
db2056.codfw.wmnet 3306: 1
db2063.codfw.wmnet 3306: 1
db2064.codfw.wmnet 3306: 1
db2017.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306: 1
dbstore1001.eqiad.wmnet 3306: 1
db1021.eqiad.wmnet 3306: 1
db1024.eqiad.wmnet 3306: 1
db1036.eqiad.wmnet 3306: 1
db1047.eqiad.wmnet 3306: 1
db1054.eqiad.wmnet 3306: 1
db1060.eqiad.wmnet 3306: 1
db1063.eqiad.wmnet 3306: 1
db1067.eqiad.wmnet 3306: 1
db1074.eqiad.wmnet 3306: 1
db1076.eqiad.wmnet 3306: 1
db1090.eqiad.wmnet 3306: 1
db1018.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2036.codfw.wmnet 3306: 1
db2043.codfw.wmnet 3306: 1
db2050.codfw.wmnet 3306: 1
db2057.codfw.wmnet 3306: 1
db2018.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306: 1
db1035.eqiad.wmnet 3306: 1
db1038.eqiad.wmnet 3306: 1
db1044.eqiad.wmnet 3306: 1
db1077.eqiad.wmnet 3306: 1
db1078.eqiad.wmnet 3306: 1
db1075.eqiad.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
db2065.codfw.wmnet 3306: 1
db2058.codfw.wmnet 3306: 1
db2051.codfw.wmnet 3306: 1
db2044.codfw.wmnet 3306: 1
db2037.codfw.wmnet 3306: 1
db2019.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306: 1
db1056.eqiad.wmnet 3306: 1
db1059.eqiad.wmnet 3306: 1
db1064.eqiad.wmnet 3306: 1
db1068.eqiad.wmnet 3306: 1
db1081.eqiad.wmnet 3306: 1
db1084.eqiad.wmnet 3306: 1
db1091.eqiad.wmnet 3306: 1
db1040.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2038.codfw.wmnet 3306: 1
db2045.codfw.wmnet 3306: 1
db2052.codfw.wmnet 3306: 1
db2059.codfw.wmnet 3306: 1
db2066.codfw.wmnet 3306: 1
db2023.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1026.eqiad.wmnet 3306: 1
db1045.eqiad.wmnet 3306: 1
db1070.eqiad.wmnet 3306: 1
db1071.eqiad.wmnet 3306: 1
db1082.eqiad.wmnet 3306: 1
db1087.eqiad.wmnet 3306: 1
db1092.eqiad.wmnet 3306: 1
db1049.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2039.codfw.wmnet 3306: 1
db2046.codfw.wmnet 3306: 1
db2053.codfw.wmnet 3306: 1
db2060.codfw.wmnet 3306: 1
db2067.codfw.wmnet 3306: 1
db2028.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306: 1
db1023.eqiad.wmnet 3306: 1
db1030.eqiad.wmnet 3306: 1
db1037.eqiad.wmnet 3306: 1
db1061.eqiad.wmnet 3306: 1
db1085.eqiad.wmnet 3306: 1
db1088.eqiad.wmnet 3306: 1
db1093.eqiad.wmnet 3306: 1
db1050.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2040.codfw.wmnet 3306: 1
db2047.codfw.wmnet 3306: 1
db2054.codfw.wmnet 3306: 1
db2061.codfw.wmnet 3306: 1
db2068.codfw.wmnet 3306: 1
db2029.codfw.wmnet 3306: 1
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1028.eqiad.wmnet 3306: 1
db1033.eqiad.wmnet 3306: 1
db1034.eqiad.wmnet 3306: 1
db1039.eqiad.wmnet 3306: 1
db1062.eqiad.wmnet 3306: 1
db1079.eqiad.wmnet 3306: 1
db1086.eqiad.wmnet 3306: 1
db1094.eqiad.wmnet 3306: 1
db1041.eqiad.wmnet 3306: 1
dbstore2001.codfw.wmnet 3306: 1
dbstore2002.codfw.wmnet 3306: 1
db2033.codfw.wmnet 3306: 1
dbstore1001.eqiad.wmnet 3306: 1
dbstore1002.eqiad.wmnet 3306: 1
db1029.eqiad.wmnet 3306: 1
db1031.eqiad.wmnet 3306: 1
root@neodymium:~/software/dbtools$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -BN -h $host -P $port $db -e "SHOW STATUS like 'Ssl_cipher'"; done
db2034.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2042.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2048.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2055.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2062.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2069.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2070.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2016.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1080.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1083.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1089.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1073.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1072.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1066.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1065.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1055.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1051.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1047.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1057.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1052.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2035.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2041.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2049.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2056.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2063.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2064.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2017.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1021.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1024.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1036.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1047.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1054.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1060.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1063.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1067.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1074.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1076.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1090.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1018.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2036.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2043.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2050.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2057.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2018.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1035.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1038.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1044.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1077.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1078.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1075.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2065.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2058.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2051.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2044.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2037.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2019.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1056.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1059.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1064.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1068.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1081.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1084.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1091.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1040.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2038.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2045.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2052.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2059.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2066.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2023.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1026.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1045.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1070.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1071.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1082.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1087.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1092.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1049.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2039.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2046.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2053.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2060.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2067.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2028.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1023.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1030.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1037.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1061.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1085.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1088.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1093.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1050.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2040.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2047.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2054.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2061.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2068.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2029.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1028.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1033.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1034.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1039.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1062.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1079.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1086.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1094.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1041.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2001.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore2002.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db2033.codfw.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1001.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
dbstore1002.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1029.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384
db1031.eqiad.wmnet 3306: Ssl_cipher	DHE-RSA-AES256-GCM-SHA384

All single-shard hosts use it for replication, the dbstores and other multi-source replication's host have to restart its io thread to enable TLS:

$ cat [sx]*.hosts | while read host port; do echo -n "$host $port: "; /usr/local/bin/mysql -B -h $host -P $port $db -e "SHOW ALL SLAVES STATUS\G" | grep Master_SSL_Allowed; done
db2034.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2042.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2048.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2055.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2062.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2069.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2070.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2016.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3311: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1080.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1083.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1089.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1073.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1072.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1066.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1065.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1055.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1051.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1047.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
db1057.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1052.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2035.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2041.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2049.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2056.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2063.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2064.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2017.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3312: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
db1021.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1024.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1036.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1047.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
db1054.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1060.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1063.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1067.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1074.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1076.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1090.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1018.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2036.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2043.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2050.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2057.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2018.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3313: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1015.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1035.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1038.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1044.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1077.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1078.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1075.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2065.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2058.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2051.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2044.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2037.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2019.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3314: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1053.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1056.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1059.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1064.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1068.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1081.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1084.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1091.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1040.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2038.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2045.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2052.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2059.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2066.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2023.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3315: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1026.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1045.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1070.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1071.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1082.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1087.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1092.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1049.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2039.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2046.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2053.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2060.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2067.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2028.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3316: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1022.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1023.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1030.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1037.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1061.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1085.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1088.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1093.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1050.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2040.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2047.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2054.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2061.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2068.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
db2029.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
labsdb1001.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1003.eqiad.wmnet 3306: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
db1069.eqiad.wmnet 3317: ERROR 2026 (HY000): SSL connection error: SSL is required, but the server does not support it
labsdb1009.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1010.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
labsdb1011.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
db1095.eqiad.wmnet 3306: ERROR 1045 (28000): Access denied for user 'root'@'10.64.32.20' (using password: YES)
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1028.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1033.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1034.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1039.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1062.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1079.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1086.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1094.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1041.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore2001.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
dbstore2002.codfw.wmnet 3306:            Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db2033.codfw.wmnet 3306:            Master_SSL_Allowed: Yes
dbstore1001.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: Yes
dbstore1002.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: Yes
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
           Master_SSL_Allowed: No
db1029.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
db1031.eqiad.wmnet 3306:            Master_SSL_Allowed: Yes
Stashbot added a comment.Feb 9 2017, 4:06 PM

Mentioned in SAL (#wikimedia-operations) [2017-02-09T16:06:23Z] <jynus> rolling restart of replication threads for dbstore1002/2001/2002 T111654

jcrespo closed this task as Resolved.Feb 9 2017, 5:08 PM
jcrespo claimed this task.

I have restarted all replication channels of dbstore1002/2001/2002 and db1047.

I consider this task resolved, with some follow-ups, less critical that I will handle on a separate ticket, with a different priority.

jcrespo mentioned this in T157702: Followup for TLS MariaDB server roll-out.Feb 9 2017, 5:31 PM
jcrespo created subtask T157702: Followup for TLS MariaDB server roll-out.
jcrespo removed a parent task: T151583: Use tls for dump backup generation.Feb 9 2017, 5:37 PM
jcrespo removed a subtask: T152427: Create a check/calendar alert for MariaDB TLS certs.Feb 21 2017, 4:11 PM
Jgreen mentioned this in T170320: encrypt fundraising mariadb replication.Jul 11 2017, 8:15 PM
Krinkle removed a subtask: T157702: Followup for TLS MariaDB server roll-out.May 31 2022, 11:41 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL