Since version v1.5.13, memcached supports TLS!
**Problem**
Backstory: Our mcrouter instances have 2 server pools: one that includes all mc* hosts on the local primary DC and another pool that consists of 4 mw* servers which act as a "mcrouter proxy" to the other primary DC, for example in eqiad we have:
```
"codfw": {
"servers": [
"10.192.0.61:11214:ascii:ssl",
"10.192.16.56:11214:ascii:ssl",
"10.192.32.113:11214:ascii:ssl",
"10.192.48.94:11214:ascii:ssl"
]
},
"eqiad": {
"servers": [
"10.64.0.80:11211:ascii:plain",
"10.64.0.81:11211:ascii:plain",
"10.64.0.82:11211:ascii:plain",
"10.64.0.83:11211:ascii:plain",
"10.64.0.84:11211:ascii:plain",
"10.64.16.107:11211:ascii:plain",
"10.64.16.108:11211:ascii:plain",
"10.64.16.109:11211:ascii:plain",
"10.64.16.110:11211:ascii:plain",
"10.64.32.208:11211:ascii:plain",
"10.64.32.209:11211:ascii:plain",
"10.64.32.210:11211:ascii:plain",
"10.64.32.211:11211:ascii:plain",
"10.64.32.212:11211:ascii:plain",
"10.64.48.155:11211:ascii:plain",
"10.64.48.156:11211:ascii:plain",
"10.64.48.157:11211:ascii:plain",
"10.64.48.158:11211:ascii:plain"
]
}
```
**Goal**
If we were to enable TLS, will eliminate the need to use those "mcrouter proxies", and secure connectivity between mediawiki and the memcached cluster. This will eliminate 4 snowflake mediawiki servers from production! We can run memcached on two ports, a TLS one, for cross-dc replication, and a nontls one for local datacentre traffic.
**Versions:**
* v1.6.6: we have this version packaged and ready, but it will need to be deployed with caution since there are some changes which can affect a busy cluster like ours
**How? (mediawiki is on eqiad)**
We will `enable_tls` so to have memcached listening on 11214 for TLS connections and on 11211 for `notls` connections. When both clusters are listening to both ports, we can replace the relevant pools in mcrouter
[x] Create the relevant puppet changes
[x] Test on mwdebug2001: we can `enable_tls`on mc2019, add it on mwdebug2001's pool and run a simple url list against mwdebug2001.
[x] Enable both tls and notls listening ports on codfw
[] Enable both tls and notls listening ports on eqiad (after June 2021 switchover)
[] Replace the eqiad pool in the codfw mcrouter configs (after June 2021 switchover)
[x] Replace the codfw pool in the eqiad mcrouter configs
**Notes**
We could consider switching all memcached traffic to TLS, but this comes with a major drawback: all tools that can provide real time key traffic (such as memkeys etc all), practically dump the network traffic. If this traffic is encrypted, the tools become useless. We are going to solve this problem at a later time.