Apache <=> mariadb SSL/TLS for cross-datacenter writes
Open, NormalPublic

Description

Unless we can somehow entirely avoid writes on GET/HEAD (which might be hard for CentralAuth and a few other things), these will still happen on occasion, mostly as post-send writes in DeferredUpdates.

Such updates should use encryption, instead of just sending passwords and data over the wire.

aaron created this task.May 9 2016, 8:21 PM
Restricted Application added subscribers: Zppix, Aklapper. · View Herald TranscriptMay 9 2016, 8:21 PM
aaron removed aaron as the assignee of this task.May 16 2016, 6:54 PM
aaron moved this task from Inbox to Backlog on the Performance-Team board.

I've put T111654 as a blocker, but in reality, cross datacenter writes using SSL should be already possible in all cases, as I can guarantee already that *all masters are already using TLS 1.2*. Only some slaves within the datacenter are still using plain text and only need to be rebooted.

We need to coordinate the certificate handling, the current system may not be the most adequate method.

I've been playing around with some open source SQL proxies lately.
These support persistent connections. I wonder how much of the
overhead of MySQL for a remote server is establishing connection and
TLS negotiation, and how much is real extra latency (e.g
cross-datacenter replication seems to not be that great, but it is a
single connection with low bandwidth requirements).

A test could be done to evaluate the overhead in both cases to help
plan the architecture.

BBlack moved this task from Triage to Watching on the Traffic board.Oct 4 2016, 12:44 PM

re: certificate handling that @jcrespo mentioned, see also T150822: Internal PKI for secure communication - Barcelona Ops offsite 2016 for the related discussion we had at the Ops offsite 2016

Gilles moved this task from Backlog to Blocked on the Performance-Team board.Dec 7 2016, 6:46 PM

TLS is deployed on all core MySQLs (s*, x2, es*, pc* shards)- although for obvious reasons, it is not enforced, I think this was the largest blocker for this task and is no more.

Reading things like https://www.percona.com/blog/2013/10/10/mysql-ssl-performance-overhead/ I think this may only make sense for DB_MASTER (also index 0) connections.

I suppose I can do some testing in script/eval.php to gauge serial connection and query rate differences.

aaron added a comment.Jul 19 2017, 5:04 PM

Roll-out should probably be something like:
a) DB_MASTER connections for testwiki/mediawikiwiki (group 0)
b) DB_MASTER connections for S3 (25%, 50%, then 100% of connections)
c) DB_MASTER connections for external stores (25%, 50%, then 100% of connections)
d) All DB_MASTER connections remaining shards (25%, 50%, then 100% of connections)

That would take care of any cross-DC write scenario (without having to check which DC is active) and fulfill this task.

If it's acceptable, this can be expanded to all connections (e.g. DB_REPLICA), preferably if proxying is setup IMO. That would be a separate task though.

1978Gage2001 moved this task from Triage to In progress on the DBA board.Dec 11 2017, 9:45 AM
Marostegui moved this task from In progress to Triage on the DBA board.Dec 11 2017, 10:59 AM
Imarlier moved this task from Blocked to Backlog on the Performance-Team board.Jan 18 2018, 5:29 PM
Krinkle moved this task from Potential goals to Blocked on the Performance-Team board.
jcrespo moved this task from Triage to Meta/Epic on the DBA board.Sep 14 2018, 1:02 PM