Since version v1.5.13, memcached supports TLS!
Problem
Backstory: Our mcrouter instances have 2 server pools: one that includes all mc* hosts on the local primary DC and another pool that consists of 4 mw* servers which act as a "mcrouter proxy" to the other primary DC, for example in eqiad we have:
"codfw": {
"servers": [
"10.192.0.61:11214:ascii:ssl",
"10.192.16.56:11214:ascii:ssl",
"10.192.32.113:11214:ascii:ssl",
"10.192.48.94:11214:ascii:ssl"
]
},
"eqiad": {
"servers": [
"10.64.0.80:11211:ascii:plain",
"10.64.0.81:11211:ascii:plain",
"10.64.0.82:11211:ascii:plain",
"10.64.0.83:11211:ascii:plain",
"10.64.0.84:11211:ascii:plain",
"10.64.16.107:11211:ascii:plain",
"10.64.16.108:11211:ascii:plain",
"10.64.16.109:11211:ascii:plain",
"10.64.16.110:11211:ascii:plain",
"10.64.32.208:11211:ascii:plain",
"10.64.32.209:11211:ascii:plain",
"10.64.32.210:11211:ascii:plain",
"10.64.32.211:11211:ascii:plain",
"10.64.32.212:11211:ascii:plain",
"10.64.48.155:11211:ascii:plain",
"10.64.48.156:11211:ascii:plain",
"10.64.48.157:11211:ascii:plain",
"10.64.48.158:11211:ascii:plain"
]
}Goal
If we were to enable TLS, will eliminate the need to use those "mcrouter proxies", and secure connectivity between mediawiki and the memcached cluster. This will eliminate 4 snowflake mediawiki servers from production! We can run memcached on two ports, a TLS one, for cross-dc replication, and a nontls one for local datacentre traffic.
Versions:
- v1.6.6: we have this version packaged and ready, but it will need to be deployed with caution since there are some changes which can affect a busy cluster like ours
How? (mediawiki is on eqiad)
We will enable_tls so to have memcached listening on 11214 for TLS connections and on 11211 for notls connections. When both clusters are listening to both ports, we can replace the relevant pools in mcrouter
- Create the relevant puppet changes
- Test on mwdebug2001: we can enable_tlson mc2019, add it on mwdebug2001's pool and run a simple url list against mwdebug2001.
- Enable both tls and notls listening ports on codfw
- Enable both tls and notls listening ports on eqiad (after June 2021 switchover)
- Replace the eqiad pool in the codfw mcrouter configs (after June 2021 switchover)
- Replace the codfw pool in the eqiad mcrouter configs
Notes
We could consider switching all memcached traffic to TLS, but this comes with a major drawback: all tools that can provide real time key traffic (such as memkeys etc all), practically dump the network traffic. If this traffic is encrypted, the tools become useless. We are going to solve this problem at a later time.
