We need the ability for varnish backends to make outbound HTTPS connections in at least two scenarios:
- For tier-1 cross-DC applayer traffic (e.g. cp2001.codfw.wmnet -> appservers.svc.eqiad.wmnet) - we don't have any known other solution to secure this traffic on inter-DC links at this time.
- For inter-tier varnish-be->varnish-be traffic: IPSec is currently protecting this, but HTTPS has the potential to be operationally-better and make the current IPSec deployment less-critical, or open us to ditching the current host-based IPSec and waiting on tunnels of some kind.
Known options for making this happen:
- Deploy an stunnel configuration locally on the varnish machines.
- The idea here would be to deploy stunnel with a separately-configured tunnel instance for each defined varnish backend.
- Instead of simply backending to appservers.svc.eqiad.wmnet:80 today, it would backend to localhost:12345 (unique port assignment), which is an stunnel configured to connect to appservers.svc.eqiad.wmnet:443.
- There could be other alternatives similar to stunnel, but stunnel looks like a legit/default option here.
- Significant Con: Yet another piece of software in the request flow for reliability/debugging woes.
- Patch varnish3 ourselves for outbound HTTPS
- Possibly using Amazon's s2n library, as it could be much simpler than using OpenSSL directly.
- Would deepen the amount of source-level customization we're doing with Varnish3 today, which is already a long-term problem for maintainability and tech debt.
- Patch would probably be significantly difficult. This is not lightweight patchwork. There are risks we could make varnish less stable, make a security-affecting mistake in the code, and/or make it much more difficult to continue merging in upstream 3.0.x fixes.
- Upgrade to Varnish4, and then create our own custom director module for outbound HTTPS
- Varnish4 does this already in the commercial Plus variant, but not open source. We could do the same, as open source.
- It's sad to redundantly re-do the upstream closed-source work here, but as a module in varnish4 it would be significantly cleaner than hacking it into varnish3.
- Depends on Varnish4 upgrade, which is significantly difficult and off in the Future for now, and may not ever happen if we find an alternative first.
- Upgrade to something non-Varnish that supports outbound TLS out of the box:
- e.g. ATS: T96853
- As with the above, this is neither near-term nor easy