Page MenuHomePhabricator
Create Task
Maniphest T182892

I can't connect to DB replica on Toolforge due to TLS-related failure
Closed, ResolvedPublic
Actions

Description

My bot on Toolforge (tool name is "tools.mbh") connects to ruwiki's DB replica. Bot written on C#, running through mono, uses MySql.Data library. Several days earlier it worked fine, now it can't connect to database due to TLS-related failure. Error report:

Unhandled Exception:
 System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
  at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (System.IAsyncResult asyncResult) [0x00040] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (System.IAsyncResult ar, System.Boolean ignoreEmpty) [0x00000] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (System.IAsyncResult result) [0x00071] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslClientStream.EndNegotiateHandshake (System.IAsyncResult result) [0x0003b] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (System.IAsyncResult asyncResult) [0x0000c] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslStreamBase.EndRead (System.IAsyncResult asyncResult) [0x00057] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Net.Security.Private.LegacySslStream.EndAuthenticateAsClient (System.IAsyncResult asyncResult) [0x00011] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at Mono.Net.Security.Private.LegacySslStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x0000e] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at Mono.Net.Security.Private.MonoSslStreamImpl.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x00006] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at System.Net.Security.SslStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x00006] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at MySql.Data.MySqlClient.NativeDriver.StartSSL () [0x0005d] in <0a135c8e4d604d948724bf6960583b7f>:0
  at MySql.Data.MySqlClient.NativeDriver.Open () [0x002ce] in <0a135c8e4d604d948724bf6960583b7f>:0
  at MySql.Data.MySqlClient.Driver.Open () [0x0000b] in <0a135c8e4d604d948724bf6960583b7f>:0
  at MySql.Data.MySqlClient.Driver.Create (MySql.Data.MySqlClient.MySqlConnectionStringBuilder settings) [0x0003f] in <0a135c8e4d604d948724bf6960583b7f>:0
 [ERROR] FATAL UNHANDLED EXCEPTION: System.IO.IOException: The authentication or decryption has failed. ---> System.IO.IOException: The authentication or decryption has failed. ---> Mono.Security.Protocol.Tls.TlsException: The authentication or decryption has failed.
  at Mono.Security.Protocol.Tls.RecordProtocol.EndReceiveRecord (System.IAsyncResult asyncResult) [0x00040] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Security.Protocol.Tls.SslClientStream.SafeEndReceiveRecord (System.IAsyncResult ar, System.Boolean ignoreEmpty) [0x00000] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Security.Protocol.Tls.SslClientStream.NegotiateAsyncWorker (System.IAsyncResult result) [0x00071] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslClientStream.EndNegotiateHandshake (System.IAsyncResult result) [0x0003b] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback (System.IAsyncResult asyncResult) [0x0000c] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
   --- End of inner exception stack trace ---
  at Mono.Security.Protocol.Tls.SslStreamBase.EndRead (System.IAsyncResult asyncResult) [0x00057] in <4d95459e5c814a5dad6816d7b3a5a54b>:0
  at Mono.Net.Security.Private.LegacySslStream.EndAuthenticateAsClient (System.IAsyncResult asyncResult) [0x00011] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at Mono.Net.Security.Private.LegacySslStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x0000e] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at Mono.Net.Security.Private.MonoSslStreamImpl.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x00006] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at System.Net.Security.SslStream.AuthenticateAsClient (System.String targetHost, System.Security.Cryptography.X509Certificates.X509CertificateCollection clientCertificates, System.Security.Authentication.SslProtocols enabledSslProtocols, System.Boolean checkCertificateRevocation) [0x00006] in <55b24190fabd4b3fae27ef5b276a5ac0>:0
  at MySql.Data.MySqlClient.NativeDriver.StartSSL () [0x0005d] in <0a135c8e4d604d948724bf6960583b7f>:0
  at MySql.Data.MySqlClient.NativeDriver.Open () [0x002ce] in <0a135c8e4d604d948724bf6960583b7f>:0
  at MySql.Data.MySqlClient.Driver.Open () [0x0000b] in <0a135c8e4d604d948724bf6960583b7f>:0
  at MySql.Data.MySqlClient.Driver.Create (MySql.Data.MySqlClient.MySqlConnectionStringBuilder settings) [0x0003f] in <0a135c8e4d604d948724bf6960583b7f>:0

Event Timeline

MBH created this task.Dec 14 2017, 3:48 PM
Restricted Application added subscribers: Base, Aklapper. · View Herald TranscriptDec 14 2017, 3:48 PM
Framawiki updated the task description. (Show Details)Dec 14 2017, 5:27 PM
Framawiki added a subscriber: Framawiki.
zhuyifei1999 added a project: Data-Services.Dec 14 2017, 5:56 PM
zhuyifei1999 added a subscriber: zhuyifei1999.
bd808 added a project: DBA.Dec 14 2017, 6:00 PM
bd808 added a subscriber: bd808.Dec 14 2017, 6:13 PM

I'm not certain how TLS connections to the servers would have ever worked. I wonder if there is some signal that is coming from the new db cluster that is triggering your client to attempt TLS protection on the connection?

A search result suggests that you could try adding ;SslMode=none to your connection string to disable TLS negotiation by your client.

MBH closed this task as Resolved.Dec 14 2017, 11:06 PM
MBH claimed this task.

It works, thanks.

jcrespo added a subscriber: jcrespo.Dec 15 2017, 9:04 AM

I wonder if there is some signal that is coming from the new db cluster that is triggering your client to attempt TLS protection on the connection

TLS is enabled on the new servers, but only used for internal administration with a self-managed certificate ; as there is no way currently to propagate TLS client keys, it will not work for users, so effectively tls connections are currently not offered (that could change in the future, but right now it is complex- for example, the tls is domain-based, but people connect to a CNAME, so that is not handled now). Because some clients do the right thing and try to use TLS first, they can fail- however, TLS was never offered on the old servers, so no downgrade has happened- TLS was never working before; it should be explicitly disabled for now.

We will see if there is interest on TLS- but it it will not be a high priority for now for cloud databases, as data from wikireplicas and toolsdb should not contain sensitive information or private data (except the password exchange process, which is already pre-hashed without sending clear-text passwords.

bd808 added a comment.Dec 15 2017, 5:35 PM

Thanks for the clarification @jcrespo. I'll try to find a place on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database to highlight the potential for a client to attempt TLS connections and fail. That won't keep people from bumping into the problem in the first place, but hopefully it will make it a bit easier for them to find out that answer is probably hinting to their client not to try TLS at all.

bd808 added a comment.Dec 15 2017, 5:44 PM

https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Database&diff=1778547&oldid=1777977

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL