We'll eventually want to switch the cache clusters' port 80 to being served directly by nginx (ideally, with no proxy backend - just 301/403/etc logic to force other traffic off to HTTPS). It's more efficient and it simplifies request-routing as there's no conditional on where the traffic first enters at - all initial termination of HTTP(S) happens in one piece of software.
We're not really ready to make this transition yet, as we're still allowing HTTP traffic to flow through the varnish instances in various corner cases. In the meantime, we can take a few preparatory steps to ease the transition in the future.
[ ] - Configure varnish frontend listeners to listen on an alternate port, in addition to port 80. The backends use 3128. Port 3127 is available and makes sense.
[ ] - Restart varnishes to get port 3127 working
[ ] - Configure nginx HTTPS proxies to proxy traffic into port 3127 instead of port 80. This should have the side-benefit of making it easier to analyze remaining non-HTTPS traffic that's hitting varnish's port 80 directly (negative regexes against ReqStart or XFP are annoyingly difficult to mix with other varnishlog filters).
[ ] - Wait until we've resolved all the other issues and effectively killed off all non redirect/forbidden traffic on port 80
[ ] - Set up new 301/403 code in our nginx config that can terminate all HTTP traffic directly, but on a temporary alternate port such as 8080, and test/vet it.
[ ] - Go through a complicated two-stage process of depooling nodes one by one and switching off the varnish port 80 listener then turning on the nginx port 80 listener.
[ ] - Cleanup - varnish code can now assume all requests are HTTPS, so we can kill some related logic here