Page MenuHomePhabricator

tools-static.wmflabs.org/cdnjs URLs getting 403
Closed, ResolvedPublic

Description

All URLs from the cdnjs tool started getting 403 Forbidden from the upstream provider as of 06/Apr/2020 17:46:04 UTC

Looking for a fix

Event Timeline

Bstorm triaged this task as Unbreak Now! priority.Apr 6 2020, 9:51 PM
Bstorm created this task.
Restricted Application added subscribers: Liuxinyu970226, Aklapper. · View Herald TranscriptApr 6 2020, 9:51 PM
Bstorm added a comment.Apr 6 2020, 9:52 PM

So, I can confirm the libraries exist upstream at what I think are the expected URLs. I can consistently reproduce the issue with curl against the inactive proxy server.

jrbs added a subscriber: jrbs.Apr 6 2020, 9:52 PM
Erutuon added a subscriber: Erutuon.Apr 6 2020, 9:53 PM
Bstorm added a comment.Apr 6 2020, 9:55 PM

Confirmed the URLs are formed right. The same thing happens on both the active and inactive server, but both do not use public ips, so they would appear to have the same origin. Curl against the upstream URL does not produce the problem, though.

zhuyifei1999 added a subscriber: zhuyifei1999.EditedApr 6 2020, 10:10 PM

IP-based block?

Considering a full checkout of the cdnjs is not possible given our disk space (T182604, unless we use bigdisk...), I see a few possibilities forward:

  • Negotiate with Cloudflare -- human factors could potentially take forever
  • Find another mirror -- we will probably be much more likely to be 'seen'
  • Do a checkout like the old way, but on a bigdisk -- we are going to run out. Every library version added to cdnjs will add s lot of space usage; this won't last.
  • Do a checkout like the old way, but with transparent compression -- btrfs can do this... though the disk requirement is still probably gonna be larger than what m1.medium can hold, so probably need new instance. the space usage will still increase, but at a much slower rate (my laptop has 28% from zlib on 3.1G of C/C++ source code, into 915M, however, minified JS / CSS code are likely less compressible than C/C++ sources), so the above still applies
  • Dynamically generate the contents from a bare git clone -- we know git compresses & deduplicates the data really well (T182604#3932696), but we currently do not have the code to support this. I think of two ways this could be done
    • Some nginx plugin to read from git
    • Some FUSE module to mount the git <= I prefer this one out of the two given that it is easier to test, and the FUSE filesystem API is less of an attack surface than HTTP (I would think).

Trying different proxy settings...

@zhuyifei1999 It is almost certainly not an ip block. I can curl the site from the proxy servers just fine.

zhuyifei1999 added a comment.EditedApr 6 2020, 10:21 PM

Oh oops... I curled tools-static rather than upstream.

Change 586464 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] tools-static: CDNJS suddenly requires SNI name to be past along

https://gerrit.wikimedia.org/r/586464

Ok, summarizing what happened during IRC debugging:

So, on the spare server, tools-static-12, @Bstorm began turning on very verbose logging in nginx to /var/log/nginx/error.log (no idea how this is done, elaborate needed?), so we could see exactly what is sent to the upstream cdnjs server, she did curl -v -H "X-Forwarded-Proto: https" localhost/cdnjs/ajax/libs/react-dom/16.13.1/cjs/react-dom-server.browser.development.js, and we have:

2020/04/06 22:45:30 [debug] 22505#22505: *1 http proxy header:
"GET /ajax/libs/react-dom/16.13.1/cjs/react-dom-server.browser.development.js HTTP/1.0
Referer: https://tools.wmflabs.org/cdnjs/
Host: cdnjs.cloudflare.com
Connection: close
User-Agent: curl/7.52.1
Accept: */*
X-Forwarded-Proto: https

"

Logs of 403 comes after that.

Ok, we now know exactly what is being sent to the server, so I tried reproducing it in curl:

110:53:00 0 ✓ zhuyifei1999@tools-static-12: ~/T249558$ curl -v -H 'Host: cdnjs.cloudflare.com' -H 'Referer: https://tools.wmflabs.org/cdnjs/' -H 'Connection: close' -H 'User-Agent: curl/7.52.1' -H 'Accept: */*' -H 'X-Forwarded-Proto: https' --http1.0 https://cdnjs.cloudflare.com/ajax/libs/react-dom/16.13.1/cjs/react-dom-server.browser.development.js > /dev/null
2 % Total % Received % Xferd Average Speed Time Time Time Current
3 Dload Upload Total Spent Left Speed
4 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 104.16.132.229...
5* TCP_NODELAY set
6* Connected to cdnjs.cloudflare.com (104.16.132.229) port 443 (#0)
7* ALPN, offering http/1.1
8* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
9* successfully set certificate verify locations:
10* CAfile: /etc/ssl/certs/ca-certificates.crt
11 CApath: /etc/ssl/certs
12* TLSv1.2 (OUT), TLS header, Certificate Status (22):
13} [5 bytes data]
14* TLSv1.2 (OUT), TLS handshake, Client hello (1):
15} [512 bytes data]
16* TLSv1.2 (IN), TLS handshake, Server hello (2):
17{ [106 bytes data]
18* TLSv1.2 (IN), TLS handshake, Certificate (11):
19{ [2250 bytes data]
20* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
21{ [148 bytes data]
22* TLSv1.2 (IN), TLS handshake, Server finished (14):
23{ [4 bytes data]
24* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
25} [70 bytes data]
26* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
27} [1 bytes data]
28* TLSv1.2 (OUT), TLS handshake, Finished (20):
29} [16 bytes data]
30* TLSv1.2 (IN), TLS change cipher, Client hello (1):
31{ [1 bytes data]
32* TLSv1.2 (IN), TLS handshake, Finished (20):
33{ [16 bytes data]
34* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
35* ALPN, server accepted to use http/1.1
36* Server certificate:
37* subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare.com
38* start date: Jan 7 00:00:00 2020 GMT
39* expire date: Oct 9 12:00:00 2020 GMT
40* subjectAltName: host "cdnjs.cloudflare.com" matched cert's "*.cloudflare.com"
41* issuer: C=US; ST=CA; L=San Francisco; O=CloudFlare, Inc.; CN=CloudFlare Inc ECC CA-2
42* SSL certificate verify ok.
43} [5 bytes data]
44> GET /ajax/libs/react-dom/16.13.1/cjs/react-dom-server.browser.development.js HTTP/1.0
45> Host: cdnjs.cloudflare.com
46> Referer: https://tools.wmflabs.org/cdnjs/
47> Connection: close
48> User-Agent: curl/7.52.1
49> Accept: */*
50> X-Forwarded-Proto: https
51>
52{ [5 bytes data]
53< HTTP/1.1 200 OK
54< Date: Mon, 06 Apr 2020 22:53:08 GMT
55< Content-Type: application/javascript; charset=utf-8
56< Connection: close
57< Last-Modified: Thu, 19 Mar 2020 20:16:11 GMT
58< ETag: W/"5e73d30b-1f7ab"
59< Expires: Sat, 27 Mar 2021 22:53:08 GMT
60< Cache-Control: public, max-age=30672000
61< Vary: Accept-Encoding
62< Timing-Allow-Origin: *
63< Access-Control-Allow-Origin: *
64< Served-In-Seconds: 0.004
65< CF-Cache-Status: HIT
66< Age: 276403
67< Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
68< Strict-Transport-Security: max-age=15780000; includeSubDomains
69< Server: cloudflare
70< CF-RAY: 57ff1552183b7451-IAD
71< alt-svc: h3-27=":443"; ma=86400, h3-25=":443"; ma=86400, h3-24=":443"; ma=86400, h3-23=":443"; ma=86400
72<
73{ [629 bytes data]
74100 629 0 629 0 0 10974 0 --:--:-- --:--:-- --:--:-- 10844* Curl_http_done: called premature == 0
75100 125k 0 125k 0 0 2139k 0 --:--:-- --:--:-- --:--:-- 2134k
76* Closing connection 0
77} [5 bytes data]
78* TLSv1.2 (OUT), TLS alert, Client hello (1):
79} [2 bytes data]

Nope a success. What I see as the only difference between nginx headers and curl's is the ordering of the headers, so why not recreate it with ssl?

I tried openssl client and with curl headers:

111:00:24 0 ✓ zhuyifei1999@tools-static-12: ~/T249558$ cat broke
2GET /ajax/libs/react-dom/16.13.1/cjs/react-dom-server.browser.development.js HTTP/1.0
3Host: cdnjs.cloudflare.com
4Referer: https://tools.wmflabs.org/cdnjs/
5Connection: close
6User-Agent: curl/7.52.1
7Accept: */*
8X-Forwarded-Proto: https
9
1011:00:49 0 ✓ zhuyifei1999@tools-static-12: ~/T249558$ (cat broke; sleep 1) | openssl s_client -connect cdnjs.cloudflare.com:443
11CONNECTED(00000003)
12depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
13verify return:1
14depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert ECC Extended Validation Server CA
15verify return:1
16depth=0 businessCategory = Private Organization, jurisdictionC = US, jurisdictionST = Delaware, serialNumber = 4710875, C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = cloudflare.com
17verify return:1
18---
19Certificate chain
20 0 s:/businessCategory=Private Organization/jurisdictionC=US/jurisdictionST=Delaware/serialNumber=4710875/C=US/ST=California/L=San Francisco/O=Cloudflare, Inc./CN=cloudflare.com
21 i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert ECC Extended Validation Server CA
22 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert ECC Extended Validation Server CA
23 i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
24---
25Server certificate
26-----BEGIN CERTIFICATE-----
27MIIGDzCCBZWgAwIBAgIQCmi7mEpQc5n0cW6AmkSnsDAKBggqhkjOPQQDAjB0MQsw
28CQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3d3cu
29ZGlnaWNlcnQuY29tMTMwMQYDVQQDEypEaWdpQ2VydCBFQ0MgRXh0ZW5kZWQgVmFs
30aWRhdGlvbiBTZXJ2ZXIgQ0EwHhcNMTgxMDMwMDAwMDAwWhcNMjAxMTAzMTIwMDAw
31WjCBzzEdMBsGA1UEDwwUUHJpdmF0ZSBPcmdhbml6YXRpb24xEzARBgsrBgEEAYI3
32PAIBAxMCVVMxGTAXBgsrBgEEAYI3PAIBAhMIRGVsYXdhcmUxEDAOBgNVBAUTBzQ3
33MTA4NzUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
34Ew1TYW4gRnJhbmNpc2NvMRkwFwYDVQQKExBDbG91ZGZsYXJlLCBJbmMuMRcwFQYD
35VQQDEw5jbG91ZGZsYXJlLmNvbTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABM7X
36YUlJ/Us1ixuGvKPFvNggbjEXLZKKtzT02xFwTkkWYfyu+n+6bwwFU3TGeX+BEor3
374l5s9foQaWtn2dWWUbCjggOrMIIDpzAfBgNVHSMEGDAWgBT4JdmmOcfDgYclPjBU
38kRghQJsXnTAdBgNVHQ4EFgQU3n9/5nzt7WFDYEdnXYYvhP2meK0wLQYDVR0RBCYw
39JIIOY2xvdWRmbGFyZS5jb22CEnd3dy5jbG91ZGZsYXJlLmNvbTAOBgNVHQ8BAf8E
40BAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMIGfBgNVHR8EgZcw
41gZQwSKBGoESGQmh0dHA6Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEVDQ0V4
42dGVuZGVkVmFsaWRhdGlvblNlcnZlckNBLmNybDBIoEagRIZCaHR0cDovL2NybDQu
43ZGlnaWNlcnQuY29tL0RpZ2lDZXJ0RUNDRXh0ZW5kZWRWYWxpZGF0aW9uU2VydmVy
44Q0EuY3JsMEsGA1UdIAREMEIwNwYJYIZIAYb9bAIBMCowKAYIKwYBBQUHAgEWHGh0
45dHBzOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwBwYFZ4EMAQEwgYcGCCsGAQUFBwEB
46BHsweTAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQuY29tMFEGCCsG
47AQUFBzAChkVodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGlnaUNlcnRFQ0NF
48eHRlbmRlZFZhbGlkYXRpb25TZXJ2ZXJDQS5jcnQwDAYDVR0TAQH/BAIwADCCAX4G
49CisGAQQB1nkCBAIEggFuBIIBagFoAHYApLkJkLQYWBSHuxOizGdwCjw1mAT5G9+4
5043fNDsgN3BAAAAFmxoh1DAAABAMARzBFAiEAzenMFsA9n1D/J+H/XmaL3hGzK3e8
51QFKtAsiP0Fg6j4QCIHG5UotqeYii1bpVaDbbLFarWXeJZthUroAdhco/GzBRAHYA
52h3W/51l8+IxDmV+9827/Vo1HVjb/SrVgwbTq/16ggw8AAAFmxoh2ywAABAMARzBF
53AiAwbN3c/gPILHLRYuUb3XyTO6sBCeSnpXO8V2l1HDqBBQIhAPbYKCsHQtCFsCVa
548YGOlfc6dmf3WTJKnncxIlNE1UHbAHYAu9nfvB+KcbWTlCOXqpJ7RzhXlQqrUuga
55kJZkNo4e0YUAAAFmxoh1EQAABAMARzBFAiB54/V5wvYQapweKriha1ch1AiwQSPL
56Y5nwVBUcyjgItgIhAMVQjCVeMn9YBrSdXwsNe/7oMvhNZQK+VsAyfo+r1S7aMAoG
57CCqGSM49BAMCA2gAMGUCMB4bPRCaUCMu5oYRE0aoHehj+C9glkNJCkkwc1X4JWMd
58RlnaqUuYaJk9UKjE/FIP4wIxANJkzK34krZr/repTowGO/vTCJ/ZBBCAuVKXChQk
59pFqK1yc8HobLt6i+w8CY+kqRrg==
60-----END CERTIFICATE-----
61subject=/businessCategory=Private Organization/jurisdictionC=US/jurisdictionST=Delaware/serialNumber=4710875/C=US/ST=California/L=San Francisco/O=Cloudflare, Inc./CN=cloudflare.com
62issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert ECC Extended Validation Server CA
63---
64No client certificate CA names sent
65Peer signing digest: SHA256
66Server Temp Key: X25519, 253 bits
67---
68SSL handshake has read 3024 bytes and written 261 bytes
69Verification: OK
70---
71New, TLSv1.2, Cipher is ECDHE-ECDSA-CHACHA20-POLY1305
72Server public key is 256 bit
73Secure Renegotiation IS supported
74Compression: NONE
75Expansion: NONE
76No ALPN negotiated
77SSL-Session:
78 Protocol : TLSv1.2
79 Cipher : ECDHE-ECDSA-CHACHA20-POLY1305
80 Session-ID: 25E8BA00EE5C49D346491360738C6ECFC75AB55B229C67EC84055A43BDEFD743
81 Session-ID-ctx:
82 Master-Key: A1605DC754601A4651EE75652C34A3277A89B530318106A3C62FF7E01721130CD0301927194976E864C80A8FCF5C40E6
83 PSK identity: None
84 PSK identity hint: None
85 SRP username: None
86 TLS session ticket lifetime hint: 64800 (seconds)
87 TLS session ticket:
88 0000 - 27 08 9a f1 79 e8 33 a8-a2 e0 62 89 c3 56 6b 51 '...y.3...b..VkQ
89 0010 - 1a 12 d3 ff 13 2e 6b 60-7b 0e a2 7f cd 53 f0 ed ......k`{....S..
90 0020 - fc 42 6d 1f f5 f7 b4 9f-49 50 22 33 78 bc 8d 16 .Bm.....IP"3x...
91 0030 - 4b d9 a0 60 8b 2b 9b 9d-ae cb 45 24 b8 37 12 5e K..`.+....E$.7.^
92 0040 - a3 ea 3c 0b 27 24 26 52-13 3f 96 c5 10 cc 4e 4c ..<.'$&R.?....NL
93 0050 - 1f c0 4b cc 7f 94 26 97-aa a8 77 b7 3c 4e f0 da ..K...&...w.<N..
94 0060 - f7 7f e1 07 ad 46 55 09-6b 68 3c b5 e7 75 58 28 .....FU.kh<..uX(
95 0070 - 52 07 75 ac aa 94 73 fa-6e 40 17 d6 81 40 3c 2a R.u...s.n@...@<*
96 0080 - b7 82 eb aa ad 57 b9 93-01 ed 56 40 fd c7 54 3d .....W....V@..T=
97 0090 - 6a 2f ce 15 6e 07 80 43-bd 4c 51 f6 43 95 a1 b3 j/..n..C.LQ.C...
98 00a0 - 04 e7 4d 05 7b d8 a7 74-3c 4f d2 6e 65 a6 e7 e2 ..M.{..t<O.ne...
99 00b0 - 43 31 28 34 95 cb a5 3b-a0 06 95 b0 47 ea 7d df C1(4...;....G.}.
100
101 Start Time: 1586214052
102 Timeout : 7200 (sec)
103 Verify return code: 0 (ok)
104 Extended master secret: yes
105---
106HTTP/1.1 403 Forbidden
107Server: cloudflare
108Date: Mon, 06 Apr 2020 23:00:52 GMT
109Content-Type: text/html
110Content-Length: 151
111Connection: close
112CF-RAY: 57ff20a24eb6cef4-IAD
113
114<html>
115<head><title>403 Forbidden</title></head>
116<body>
117<center><h1>403 Forbidden</h1></center>
118<hr><center>cloudflare</center>
119</body>
120</html>
121read:errno=0

Nope. a fail. My reaction: are you kidding me?

Then @Krenair pointed there's a servername in ssl negotiation, and (cat broke; sleep 1) | openssl s_client -connect 104.16.133.229:443 -servername cdnjs.cloudflare.com works, but not (cat broke; sleep 1) | openssl s_client -connect cdnjs.cloudflare.com:443`, on his laptop. I can reproduce on both the spare static server and my laptop.

Then @Bstorm found https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_ssl_server_name

Thanks for figuring out the root cause together :)

Change 586464 merged by Bstorm:
[operations/puppet@production] tools-static: CDNJS suddenly requires SNI name to be past along

https://gerrit.wikimedia.org/r/586464

Bstorm closed this task as Resolved.Apr 6 2020, 11:48 PM

Ok, the patch is applied and it looks like things are fixed. For instance, https://tools-static.wmflabs.org/cdnjs/ajax/libs/moment.js/2.20.1/locale/en-ca.js works again.

Change 586475 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] tools-static: apply SNI name setting to fontcdn as well

https://gerrit.wikimedia.org/r/586475

Thanks a lot!

Change 586475 merged by Bstorm:
[operations/puppet@production] tools-static: apply SNI name setting to fontcdn as well

https://gerrit.wikimedia.org/r/586475