We need the ability for varnish backends to make outbound HTTPS connections in at least two scenarios:
1) For tier-1 cross-DC applayer traffic (e.g. cp2001.codfw.wmnet -> appservers.svc.eqiad.wmnet) - we don't have any known other solution to secure this traffic on inter-DC links at this time.
2) For inter-tier varnish-be->varnish-be traffic: IPSec is currently protecting this, but HTTPS has the potential to be operationally-better and make the current IPSec deployment less-critical, or open us to ditching the current host-based IPSec and waiting on tunnels of some kind.
Known options for making this happen:
1) Deploy an stunnel configuration locally on the varnish machines.
- The idea here would be to deploy stunnel with a separately-configured tunnel instance for each defined varnish backend.
- Instead of simply backending to appservers.svc.eqiad.wmnet:80 today, it would backend to localhost:12345 (unique port assignment), which is an stunnel configured to connect to appservers.svc.eqiad.wmnet:443.
- There could be other alternatives similar to stunnel, but stunnel looks like a legit/default option here.
- Significant Con: Yet another piece of software in the request flow for reliability/debugging woes.
2) Patch varnish3 ourselves for outbound HTTPS
- Possibly using Amazon's s2n library, as it could be much simpler than using OpenSSL directly.
- Would deepen the amount of source-level customization we're doing with Varnish3 today, which is already a long-term problem for maintainability and tech debt.
- Patch would probably be significantly difficult. This is not lightweight patchwork. There are risks we could make varnish less stable, make a security-affecting mistake in the code, and/or make it much more difficult to continue merging in upstream 3.0.x fixes.
3) Upgrade to Varnish4, and then create our own custom director module for outbound HTTPS
- Varnish4 does this already in the commercial Plus variant, but not open source. We could do the same, as open source.
- It's sad to redundantly re-do the upstream closed-source work here, but as a module in varnish4 it would be significantly cleaner than hacking it into varnish3.
- Depends on Varnish4 upgrade, which is significantly difficult and off in the Future for now, and may not ever happen if we find an alternative first.
4) Upgrade to something non-Varnish that supports outbound TLS out of the box:
- e.g. ATS: T96853
- As with the above, this is neither near-term nor easy