So, I can confirm the libraries exist upstream at what I think are the expected URLs. I can consistently reproduce the issue with curl against the inactive proxy server.

Confirmed the URLs are formed right. The same thing happens on both the active and inactive server, but both do not use public ips, so they would appear to have the same origin. Curl against the upstream URL does not produce the problem, though.

IP-based block?

Considering a full checkout of the cdnjs is not possible given our disk space (T182604, unless we use bigdisk...), I see a few possibilities forward:

• Negotiate with Cloudflare -- human factors could potentially take forever
• Find another mirror -- we will probably be much more likely to be 'seen'
• Do a checkout like the old way, but on a bigdisk -- we are going to run out. Every library version added to cdnjs will add s lot of space usage; this won't last.
• Do a checkout like the old way, but with transparent compression -- btrfs can do this... though the disk requirement is still probably gonna be larger than what m1.medium can hold, so probably need new instance. the space usage will still increase, but at a much slower rate (my laptop has 28% from zlib on 3.1G of C/C++ source code, into 915M, however, minified JS / CSS code are likely less compressible than C/C++ sources), so the above still applies
• Dynamically generate the contents from a bare git clone -- we know git compresses & deduplicates the data really well (T182604#3932696), but we currently do not have the code to support this. I think of two ways this could be done
  • Some nginx plugin to read from git
  • Some FUSE module to mount the git <= I prefer this one out of the two given that it is easier to test, and the FUSE filesystem API is less of an attack surface than HTTP (I would think).

Trying different proxy settings...

@zhuyifei1999 It is almost certainly not an ip block. I can curl the site from the proxy servers just fine.

Oh oops... I curled tools-static rather than upstream.

Change 586464 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] tools-static: CDNJS suddenly requires SNI name to be past along

https://gerrit.wikimedia.org/r/586464

Ok, summarizing what happened during IRC debugging:

So, on the spare server, tools-static-12, @Bstorm began turning on very verbose logging in nginx to /var/log/nginx/error.log (no idea how this is done, elaborate needed?), so we could see exactly what is sent to the upstream cdnjs server, she did curl -v -H "X-Forwarded-Proto: https" localhost/cdnjs/ajax/libs/react-dom/16.13.1/cjs/react-dom-server.browser.development.js, and we have:

2020/04/06 22:45:30 [debug] 22505#22505: *1 http proxy header:
"GET /ajax/libs/react-dom/16.13.1/cjs/react-dom-server.browser.development.js HTTP/1.0
Referer: https://tools.wmflabs.org/cdnjs/
Host: cdnjs.cloudflare.com
Connection: close
User-Agent: curl/7.52.1
Accept: */*
X-Forwarded-Proto: https

"

Logs of 403 comes after that.

Ok, we now know exactly what is being sent to the server, so I tried reproducing it in curl:

Nope a success. What I see as the only difference between nginx headers and curl's is the ordering of the headers, so why not recreate it with ssl?

I tried openssl client and with curl headers:

Nope. a fail. My reaction: are you kidding me?

Then @Krenair pointed there's a servername in ssl negotiation, and (cat broke; sleep 1) | openssl s_client -connect 104.16.133.229:443 -servername cdnjs.cloudflare.com works, but not (cat broke; sleep 1) | openssl s_client -connect cdnjs.cloudflare.com:443`, on his laptop. I can reproduce on both the spare static server and my laptop.

Then @Bstorm found https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_server_name

Thanks for figuring out the root cause together :)

Change 586464 merged by Bstorm:
[operations/puppet@production] tools-static: CDNJS suddenly requires SNI name to be past along

https://gerrit.wikimedia.org/r/586464

Change 586475 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] tools-static: apply SNI name setting to fontcdn as well

https://gerrit.wikimedia.org/r/586475

Thanks a lot!

Change 586475 merged by Bstorm:
[operations/puppet@production] tools-static: apply SNI name setting to fontcdn as well

https://gerrit.wikimedia.org/r/586475