Page MenuHomePhabricator

Toolforge mono version on stretch grid doesn't trust latest LE certs
Open, MediumPublic

Description

I confirmed this by doing:

aborrero@tools-sgebastion-08:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://jobs.toolforge.org")'
System.Net.WebException: Error: TrustFailure (A call to SSPI failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
[...]

For comparision, that commands outputs the whole HTML page of a HTTPS site with a cert it trust:

aborrero@tools-sgebastion-08:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://www.nuget.org")' | grep body
<body>\r
</body>\r

By the time of this writting, the mono version being used in Toolforge stretch grid is:

aborrero@tools-sgebastion-08:~$ apt-cache policy mono-runtime
mono-runtime:
  Installed: 5.12.0.226-0xamarin3+debian9b1
  Candidate: 5.12.0.226-0xamarin3+debian9b1
  Version table:
 *** 5.12.0.226-0xamarin3+debian9b1 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/thirdparty/mono-project-stretch amd64 Packages
        100 /var/lib/dpkg/status

Event Timeline

aborrero changed the task status from Open to In Progress.Fri, Oct 1, 11:43 AM
aborrero triaged this task as High priority.
aborrero created this task.
aborrero moved this task from Triage to In Progress on the Toolforge board.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.

The upstream repo contains version 6.12.0 in the stable-stretch repo which is the one we're copying from for apt.wikimedia.org.

I will test that the version can cleanly install on the stretch grid.

Question: what if people have running tools that require the old mono version?

For the record, if that mono version installs cleanly on stretch (and we don't break our users) then to finally pull the packages we need this reprepro command:

root@apt1001:~# reprepro --noskipold --component thirdparty/mono-project-stretch update stretch-wikimedia
[..]

Since we have ensure => latest for the mono runtime in the Toolforge grid exec environ, puppet will magically update the packages everywhere in the grid.

But I would like to have confirmation that we won't be breaking arbitrary tools, cc @bd808 for a +1 or further comments.

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T12:12:02Z] <arturo> experimenting with newer mono runtime on toolsbeta-sgebastion-04 (T292289)

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T12:34:11Z] <arturo> rebooting toolsbeta-sgebastion-04 (T292289)

I updated the whole mono framework on toolsbeta-sgebastion-04 but I still see the exception when connecting to toolforge HTTPS endpoints.

The upstream documentation reads:

(Recommended) Starting with Mono 3.12.0 a new tool called cert-sync is included which syncs Mono’s certificate store with the system certificate store. It should run automatically when you install the official Mono packages.
Make sure the ca-certificates-mono package is installed.

Which is apparently a lie, because the package is installed just fine

[...]
Setting up ca-certificates-mono (6.12.0.122-0xamarin1+debian9b1) ...
Processing triggers for sgml-base (1.29) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for ca-certificates (20200601~deb9u2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
Updating Mono key store
Mono Certificate Store Sync - version 6.12.0.122
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 142, your new list has 136
Import process completed.

Importing into BTLS system store:
I already trust 141, your new list has 136
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
Import process completed.
Done
done.

and I can use every single TLS client I could find (curl, wget, s_client, python, etc) they all accept the toolforge certificate, except mono.

I just checked in a buster Toolforge node, with similar results:

aborrero@tools-sgebastion-11:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://jobs.toolforge.org")'
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
aborrero@tools-sgebastion-11:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://www.nuget.org")' | grep body
<body>\r
</body>\r
aborrero@tools-sgebastion-11:~$ apt-cache policy mono-runtime
mono-runtime:
  Installed: 5.18.0.240+dfsg-3
  Candidate: 5.18.0.240+dfsg-3
  Version table:
 *** 5.18.0.240+dfsg-3 500
        500 http://deb.debian.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status

So the exception may not be related to the mono version or the LE certificate thing after all?

@Majavah found https://github.com/mono/mono/issues/21233#issuecomment-932211479. Looks like even the trunk mono uses a version of boringssl that has problems with the expired DST Root CA X3 and fails if it is in the trust store.

Related is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995432 with a comment by our own @BBlack. From chat seen on IRC I think @BBlack is working on some Puppet code to remove the DST Root CA X3 cert from the system trust store for WMF servers.

Related is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995432 with a comment by our own @BBlack. From chat seen on IRC I think @BBlack is working on some Puppet code to remove the DST Root CA X3 cert from the system trust store for WMF servers.

FYI, it's this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/725331/

I manually tested removing the DST Root CA X3 cert on tools-sgebastion-08 (dev.toolforge.org) and it seems to be working:

$ ssh dev.toolhub.org
$ grep DST_Root /etc/ca-certificates.conf
mozilla/DST_Root_CA_X3.crt
$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://en.wikipedia.org")'                                                                    System.Net.WebException: Error: TrustFailure (A call to SSPI failed, see inner e
xception.) ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
$ sudo sed -i 's#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#' /etc/ca-certificates.conf && sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 142, your new list has 135
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 141, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://en.wikipedia.org")'
"<!DOCTYPE html>
<html class="client-nojs" lang="en" dir="ltr">
<head>
...

Tests to *.toolforge.org URLs are also no longer failing with a TLS error, but seem to consistently return 403 Forbidden responses. I'm not sure why yet; the same urls work from curl from the same host. This appears to be caused by mono not sending a user-agent header by default. Something like csharp -e 'System.Net.WebClient c = new System.Net.WebClient (); c.Headers.Add ("user-agent", "bd808/1.0 (bot; bd808@wikimedia.org)"); c.DownloadString ("https://toolhub.toolforge.org/toolinfo.json")' works.

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T21:59:37Z] <bd808> clush -w @all -b 'sudo sed -i "s#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#" /etc/ca-certificates.conf && sudo update-ca-certificates' for T292289

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T21:59:37Z] <bd808> clush -w @all -b 'sudo sed -i "s#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#" /etc/ca-certificates.conf && sudo update-ca-certificates' for T292289

I hope this fixes things on the grid and bastions in the near term. Longer term we should follow https://gerrit.wikimedia.org/r/c/operations/puppet/+/725331/ and consider adding its sslcert::ca_deselect_dstx3 module across Cloud VPS.

Output from clush command:

tools-package-builder-04.tools.eqiad1.wikimedia.cloud: bash: warning: setlocale:
 LC_ALL: cannot change locale (en_US.UTF-8)
tools-package-builder-03.tools.eqiad1.wikimedia.cloud: ssh: connect to host tool
s-package-builder-03.tools.eqiad1.wikimedia.cloud port 22: No route to host
---------------
tools-acme-chief-[01-02].tools.eqiad1.wikimedia.cloud,tools-checker-04.tools.eqiad1.wikimedia.cloud,tools-clushmaster-02.tools.eqiad1.wikimedia.cloud,tools-docker-imagebuilder-01.tools.eqiad1.wikimedia.cloud,tools-docker-registry-[05-06].to
ols.eqiad1.wikimedia.cloud,tools-filesystemtest-1.tools.eqiad1.wikimedia.cloud,tools-k8s-control[1-3].tools.eqiad1.wikimedia.cloud,tools-k8s-etcd-[13-15].tools.eqiad1.wikimedia.cloud,tools-k8s-haproxy-[3-4].tools.eqiad1.wikimedia.cloud,tools-k8s-ingress-[4-6].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-[30-62,64-79].tools.eqiad1.wikimedia.cloud,tools-legacy-redirector.tools.eqiad1.wikimedia.cloud,tools-mail-03.tools.eqiad1.wikimedia.cloud,tools-nfs-test-client-01.tools.eqiad1.wikimedia.cloud,tools-prometheus-[03,05].tools.eqiad1.wikimedia.cloud,tools-proxy-[05-06].tools.eqiad1.wikimedia.cloud,tools-redis-[1003-1004].tools.eqiad1.wikimedia.cloud,tools-services-05.tools.eqiad1.wikimedia.cloud,tools-sgegrid-mas
ter.tools.eqiad1.wikimedia.cloud,tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud,tools-static-14.tools.eqiad1.wikimedia.cloud (81)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
---------------
tools-sgebastion-07.tools.eqiad1.wikimedia.cloud,tools-sgecron-01.tools.eqiad1.w
ikimedia.cloud,tools-sgeexec-[0901,0904-0920,0932-0942].tools.eqiad1.wikimedia.c
loud,tools-sgewebgrid-generic-[0901-0902].tools.eqiad1.wikimedia.cloud,tools-sge
webgrid-lighttpd-[0911-0920].tools.eqiad1.wikimedia.cloud (43)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 142, your new list has 135
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 141, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
---------------
tools-elastic-[1-3].tools.eqiad1.wikimedia.cloud,tools-package-builder-04.tools.
eqiad1.wikimedia.cloud (4)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
done.
---------------
tools-sgeexec-[0947,0950-0952].tools.eqiad1.wikimedia.cloud (4)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 135, your new list has 135
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 134, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
---------------
tools-sgebastion-[10-11].tools.eqiad1.wikimedia.cloud (2)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.18.0.240
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 146, your new list has 146
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 145, your new list has 146
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
clush: tools-package-builder-03.tools.eqiad1.wikimedia.cloud: exited with exit code 255
---------------
tools-sgebastion-08.tools.eqiad1.wikimedia.cloud
---------------
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 141, your new list has 135
Import process completed.

Importing into BTLS system store:
I already trust 140, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
Import process completed.
Done
done.

Mentioned in SAL (#countervandalism) [2021-10-02T21:29:06Z] <Krinkle> krinkle@cvn-app9 sudo sed -i 's#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#' /etc/ca-certificates.conf && sudo update-ca-certificates ref T292289, ref https://github.com/mono/mono/issues/21233

Chicocvenancio added a subscriber: Chicocvenancio.

I think this is not relevant to PAWS as we don't have mono installed. Please correct me if this is wrong.

aborrero changed the task status from In Progress to Open.Mon, Oct 4, 10:15 AM
aborrero lowered the priority of this task from High to Medium.

Thanks you all for working on this. I'm reducing priority and will follow up in the next few days if sslcert::ca_deselect_dstx3 gets finally merged.