Page MenuHomePhabricator

When downloading from git using HTTPS: HTTP 500 / GnuTLS recv error (-110)
Closed, ResolvedPublic

Description

I was about to download an extension from git from my server, but I receive this error:

GnuTLS recv error (-110): The TLS connection was non-properly terminated.

This is the command: git clone -b REL1_32 https://gerrit.wikimedia.org/r/p/mediawiki/extensions/AJAXPoll.git

Surprisingly, this works well from a different machine (a local virtual machine).

I've enabling trace to diagnose the problem, and those are the differencies:

GIT_TRACE_CURL=1 git clone -b REL1_32 https://gerrit.wikimedia.org/r/p/mediawiki/extensions/AJAXPoll.git

Production server (fails):

113:03:57.223222 http.c:586 == Info: Couldn't find host gerrit.wikimedia.org in the .netrc file; using defaults
213:03:57.226687 http.c:586 == Info: Trying 208.80.154.85...
313:03:57.226718 http.c:586 == Info: TCP_NODELAY set
413:03:57.305911 http.c:586 == Info: Connected to gerrit.wikimedia.org (208.80.154.85) port 443 (#0)
513:03:57.323077 http.c:586 == Info: found 166 certificates in /etc/ssl/certs/ca-certificates.crt
613:03:57.377945 http.c:586 == Info: found 664 certificates in /etc/ssl/certs
713:03:57.378013 http.c:586 == Info: ALPN, offering http/1.1
813:03:57.542131 http.c:586 == Info: SSL connection using TLS1.2 / ECDHE_ECDSA_AES_256_GCM_SHA384
913:03:57.543324 http.c:586 == Info: server certificate verification OK
1013:03:57.543353 http.c:586 == Info: server certificate status verification SKIPPED
1113:03:57.543585 http.c:586 == Info: common name: gerrit.wikimedia.org (matched)
1213:03:57.543601 http.c:586 == Info: server certificate expiration date OK
1313:03:57.543613 http.c:586 == Info: server certificate activation date OK
1413:03:57.543635 http.c:586 == Info: certificate public key: EC/ECDSA
1513:03:57.543645 http.c:586 == Info: certificate version: #3
1613:03:57.543680 http.c:586 == Info: subject: CN=gerrit.wikimedia.org
1713:03:57.543707 http.c:586 == Info: start date: Mon, 27 May 2019 13:00:16 GMT
1813:03:57.543730 http.c:586 == Info: expire date: Sun, 25 Aug 2019 13:00:16 GMT
1913:03:57.543774 http.c:586 == Info: issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
2013:03:57.543801 http.c:586 == Info: compression: NULL
2113:03:57.543811 http.c:586 == Info: ALPN, server did not agree to a protocol
2213:03:57.543938 http.c:533 => Send header, 0000000228 bytes (0x000000e4)
2313:03:57.543961 http.c:545 => Send header: GET /r/p/mediawiki/extensions/AJAXPoll.git/info/refs?service=git-upload-pack HTTP/1.1
2413:03:57.543970 http.c:545 => Send header: Host: gerrit.wikimedia.org
2513:03:57.543974 http.c:545 => Send header: User-Agent: git/2.11.0
2613:03:57.543979 http.c:545 => Send header: Accept: */*
2713:03:57.543983 http.c:545 => Send header: Accept-Encoding: gzip
2813:03:57.543986 http.c:545 => Send header: Accept-Language: en-US, *;q=0.9
2913:03:57.544007 http.c:545 => Send header: Pragma: no-cache
3013:03:57.544016 http.c:545 => Send header:
3113:03:57.623588 http.c:533 <= Recv header, 0000000036 bytes (0x00000024)
3213:03:57.623646 http.c:545 <= Recv header: HTTP/1.1 500 Internal Server Error
3313:03:57.623680 http.c:533 <= Recv header, 0000000037 bytes (0x00000025)
3413:03:57.623692 http.c:545 <= Recv header: Date: Sat, 08 Jun 2019 11:03:57 GMT
3513:03:57.623704 http.c:533 <= Recv header, 0000000016 bytes (0x00000010)
3613:03:57.623714 http.c:545 <= Recv header: Server: Apache
3713:03:57.623724 http.c:533 <= Recv header, 0000000074 bytes (0x0000004a)
3813:03:57.623734 http.c:545 <= Recv header: Strict-Transport-Security: max-age=106384710; includeSubDomains; preload
3913:03:57.623746 http.c:533 <= Recv header, 0000000074 bytes (0x0000004a)
4013:03:57.623755 http.c:545 <= Recv header: Strict-Transport-Security: max-age=106384710; includeSubDomains; preload
4113:03:57.623766 http.c:533 <= Recv header, 0000000019 bytes (0x00000013)
4213:03:57.623775 http.c:545 <= Recv header: Connection: close
4313:03:57.623786 http.c:533 <= Recv header, 0000000045 bytes (0x0000002d)
4413:03:57.623794 http.c:545 <= Recv header: Content-Type: text/html; charset=iso-8859-1
4513:03:57.623805 http.c:533 <= Recv header, 0000000002 bytes (0x00000002)
4613:03:57.623813 http.c:545 <= Recv header:
4713:03:57.623848 http.c:559 <= Recv data, 0000000661 bytes (0x00000295)
4813:03:57.623860 http.c:574 <= Recv data: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">.<html><he
4913:03:57.623868 http.c:574 <= Recv data: ad>.<title>500 Internal Server Error</title>.</head><body>.<
5013:03:57.623877 http.c:574 <= Recv data: h1>Internal Server Error</h1>.<p>The server encountered an i
5113:03:57.623886 http.c:574 <= Recv data: nternal error or.misconfiguration and was unable to complete
5213:03:57.623894 http.c:574 <= Recv data: .your request.</p>.<p>Please contact the server administrato
5313:03:57.623903 http.c:574 <= Recv data: r at . noc@wikimedia.org to inform them of the time this err
5413:03:57.623911 http.c:574 <= Recv data: or occurred,. and the actions you performed just before this
5513:03:57.623920 http.c:574 <= Recv data: error.</p>.<p>More information about this error may be avai
5613:03:57.623928 http.c:574 <= Recv data: lable.in the server error log.</p>.<p>Additionally, a 500 In
5713:03:57.623937 http.c:574 <= Recv data: ternal Server Error.error was encountered while trying to us
5813:03:57.623946 http.c:574 <= Recv data: e an ErrorDocument to handle the request.</p>.</body></html>
5913:03:57.623954 http.c:574 <= Recv data: .
6013:03:57.624087 http.c:586 == Info: GnuTLS recv error (-110): The TLS connection was non-properly terminated.
6113:03:57.624110 http.c:586 == Info: Curl_http_done: called premature == 1
6213:03:57.624122 http.c:586 == Info: stopped the pause stream!
6313:03:57.624150 http.c:586 == Info: Closing connection 0
64fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/extensions/AJAXPoll.git/': GnuTLS recv error (-110): The TLS connection was non-properly terminated.

Local virtual machine (OK):

113:05:05.095388 http.c:586 == Info: Couldn't find host gerrit.wikimedia.org in the .netrc file; using defaults
213:05:05.217609 http.c:586 == Info: Trying 208.80.154.85...
313:05:05.217642 http.c:586 == Info: TCP_NODELAY set
413:05:05.351779 http.c:586 == Info: Connected to gerrit.wikimedia.org (208.80.154.85) port 443 (#0)
513:05:05.359219 http.c:586 == Info: found 166 certificates in /etc/ssl/certs/ca-certificates.crt
613:05:05.394330 http.c:586 == Info: found 666 certificates in /etc/ssl/certs
713:05:05.394383 http.c:586 == Info: ALPN, offering http/1.1
813:05:05.671257 http.c:586 == Info: SSL connection using TLS1.2 / ECDHE_ECDSA_AES_256_GCM_SHA384
913:05:05.671715 http.c:586 == Info: server certificate verification OK
1013:05:05.671724 http.c:586 == Info: server certificate status verification SKIPPED
1113:05:05.671809 http.c:586 == Info: common name: gerrit.wikimedia.org (matched)
1213:05:05.671815 http.c:586 == Info: server certificate expiration date OK
1313:05:05.671820 http.c:586 == Info: server certificate activation date OK
1413:05:05.671829 http.c:586 == Info: certificate public key: EC/ECDSA
1513:05:05.671834 http.c:586 == Info: certificate version: #3
1613:05:05.671845 http.c:586 == Info: subject: CN=gerrit.wikimedia.org
1713:05:05.671853 http.c:586 == Info: start date: Mon, 27 May 2019 13:00:16 GMT
1813:05:05.671858 http.c:586 == Info: expire date: Sun, 25 Aug 2019 13:00:16 GMT
1913:05:05.671874 http.c:586 == Info: issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
2013:05:05.671884 http.c:586 == Info: compression: NULL
2113:05:05.671888 http.c:586 == Info: ALPN, server did not agree to a protocol
2213:05:05.672003 http.c:533 => Send header, 0000000228 bytes (0x000000e4)
2313:05:05.672016 http.c:545 => Send header: GET /r/p/mediawiki/extensions/AJAXPoll.git/info/refs?service=git-upload-pack HTTP/1.1
2413:05:05.672019 http.c:545 => Send header: Host: gerrit.wikimedia.org
2513:05:05.672022 http.c:545 => Send header: User-Agent: git/2.11.0
2613:05:05.672024 http.c:545 => Send header: Accept: */*
2713:05:05.672027 http.c:545 => Send header: Accept-Encoding: gzip
2813:05:05.672029 http.c:545 => Send header: Accept-Language: en-US, *;q=0.9
2913:05:05.672032 http.c:545 => Send header: Pragma: no-cache
3013:05:05.672035 http.c:545 => Send header:
3113:05:05.808784 http.c:533 <= Recv header, 0000000032 bytes (0x00000020)
3213:05:05.808834 http.c:545 <= Recv header: HTTP/1.1 301 Moved Permanently
3313:05:05.808840 http.c:533 <= Recv header, 0000000037 bytes (0x00000025)
3413:05:05.808844 http.c:545 <= Recv header: Date: Sat, 08 Jun 2019 11:05:07 GMT
3513:05:05.808849 http.c:533 <= Recv header, 0000000016 bytes (0x00000010)
3613:05:05.808852 http.c:545 <= Recv header: Server: Apache
3713:05:05.808856 http.c:533 <= Recv header, 0000000074 bytes (0x0000004a)
3813:05:05.808859 http.c:545 <= Recv header: Strict-Transport-Security: max-age=106384710; includeSubDomains; preload
3913:05:05.808865 http.c:533 <= Recv header, 0000000110 bytes (0x0000006e)
4013:05:05.808868 http.c:545 <= Recv header: Location: https://gerrit.wikimedia.org/r/mediawiki/extensions/AJAXPoll.git/info/refs?service=git-upload-pack
4113:05:05.808873 http.c:533 <= Recv header, 0000000021 bytes (0x00000015)
4213:05:05.808876 http.c:545 <= Recv header: Content-Length: 306
4313:05:05.808881 http.c:533 <= Recv header, 0000000045 bytes (0x0000002d)
4413:05:05.808884 http.c:545 <= Recv header: Content-Type: text/html; charset=iso-8859-1
4513:05:05.808888 http.c:533 <= Recv header, 0000000002 bytes (0x00000002)
4613:05:05.808909 http.c:545 <= Recv header:
4713:05:05.808928 http.c:586 == Info: Ignoring the response-body
4813:05:05.808934 http.c:559 <= Recv data, 0000000306 bytes (0x00000132)
4913:05:05.808938 http.c:574 <= Recv data: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">.<html><he
5013:05:05.808941 http.c:574 <= Recv data: ad>.<title>301 Moved Permanently</title>.</head><body>.<h1>M
5113:05:05.808943 http.c:574 <= Recv data: oved Permanently</h1>.<p>The document has moved <a href="htt
5213:05:05.808946 http.c:574 <= Recv data: ps://gerrit.wikimedia.org/r/mediawiki/extensions/AJAXPoll.gi
5313:05:05.808949 http.c:574 <= Recv data: t/info/refs?service=git-upload-pack">here</a>.</p>.</body></
5413:05:05.808952 http.c:574 <= Recv data: html>.
5513:05:05.808959 http.c:586 == Info: Curl_http_done: called premature == 0
5613:05:05.808966 http.c:586 == Info: Connection #0 to host gerrit.wikimedia.org left intact
5713:05:05.808974 http.c:586 == Info: Issue another request to this URL: 'https://gerrit.wikimedia.org/r/mediawiki/extensions/AJAXPoll.git/info/refs?service=git-upload-pack'
5813:05:05.809007 http.c:586 == Info: Couldn't find host gerrit.wikimedia.org in the .netrc file; using defaults
5913:05:05.809017 http.c:586 == Info: Found bundle for host gerrit.wikimedia.org: 0x55fe0c5a1d70 [can pipeline]
6013:05:05.809026 http.c:586 == Info: Re-using existing connection! (#0) with host gerrit.wikimedia.org
6113:05:05.809033 http.c:586 == Info: Connected to gerrit.wikimedia.org (208.80.154.85) port 443 (#0)
6213:05:05.809153 http.c:533 => Send header, 0000000226 bytes (0x000000e2)
6313:05:05.809163 http.c:545 => Send header: GET /r/mediawiki/extensions/AJAXPoll.git/info/refs?service=git-upload-pack HTTP/1.1
6413:05:05.809166 http.c:545 => Send header: Host: gerrit.wikimedia.org

Note that the server that fails, the remote server sends a 500 error (?) for the same request, which I do not understand why.

Since it causes a server error for the same request, I guess this is not my server's fault... isn't it?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 8 2019, 11:17 AM

More information: This happens with all the extensions I've tried. Also, I used the same method to download extensions 1 month ago without any issues

BBlack added a subscriber: BBlack.Jun 8 2019, 12:31 PM

The TLS-level error is just complaining that, at the end of the transaction, the connection was aborted abruptly instead of torn down cleanly. It would probably be more-ideal if gerrit's TLS stack would cleanly close on 500s when it can, but the real issue here is probably the 500 error, not the TLS error. At a glance, the GET request headers look identical in the two cases, so I'm at a loss as to what's happening on gerrit's side here. Is there perhaps a request difference in some HTTP-level authentication or cookie stuff that's not shown in the trace?

Ciencia_Al_Poder added a comment.EditedJun 8 2019, 12:47 PM

I'm just downloading a release branch as anonymous (hence https and not ssh), so it shouldn't be a problem with authentication. The only relevant differences may be:

  • Different TLS stack (the 2 machines were installed at about the same time and are running the same Debian Stretch version, maybe there's a difference in some package version)
  • Different request IP address. My server is on OVH. Wondering if it's some sort of server blacklist.
Paladox added a subscriber: Paladox.Jun 8 2019, 2:29 PM

When i looked earlier at https://gerrit.wikimedia.org/r/monitoring earlier, i saw nothing that would have explained this.

So i think it may have been apache rather than gerrit.

ArielGlenn triaged this task as Normal priority.Jun 11 2019, 8:03 AM
hashar added a subscriber: hashar.Jun 11 2019, 8:20 AM

The TLS stack is just fine and the query does reach the Apache in front of Gerrit,. The reason is the OVH one is being rejected by our configuration. Or in short it is not a configuration / software stack issue on @Ciencia_Al_Poder machine.

Since I am not familiar with that specific configuration and there are private data involved (IP address of the machine), I have filled a private task to get more informations form people that knows better than me: T225480

ema moved this task from Triage to Watching on the Traffic board.Jun 17 2019, 9:16 AM
sbassett added a subscriber: sbassett.EditedJun 21 2019, 4:18 PM

Hello @Ciencia_Al_Poder -

Apologies for the delay on a response to this issue. Due to an ongoing security incident [0], certain IP ranges continue to be restricted from accessing various Wikimedia development tools. We realize the incredible inconvenience this places upon legitimate Wikimedia developers affected by these restrictions, but we cannot provide a date by which these restrictions will be removed at this time. In the interim, we can offer a couple of workarounds:

  1. For read-only access to various Wikimedia git repositories, github.com/wikimedia should serve as an effective mirror which, at worst, should only ever be marginally out-of-date with our canonical git repositories at gerrit.wikimedia.org.
  1. For confirmed, trusted developers (a determination to be made by the Wikimedia Security-Team and Trust-and-Safety), we can potentially offer access to certain Wikimedia developer tools via static IP addresses. Please contact security@wikimedia.org for further information and to initiate this process.

[0] https://lists.wikimedia.org/pipermail/wikitech-l/2019-March/091834.html

Ok, thanks for the update. I only need a read-only access to the repos, so I'd have to live with the github mirror...

sbassett closed this task as Resolved.Jun 21 2019, 8:35 PM