Page MenuHomePhabricator
Create Task
Maniphest T292289

Toolforge mono version on stretch grid doesn't trust latest LE certs
Closed, ResolvedPublic
Actions

Description

I confirmed this by doing:

aborrero@tools-sgebastion-08:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://jobs.toolforge.org")'
System.Net.WebException: Error: TrustFailure (A call to SSPI failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
[...]

For comparision, that commands outputs the whole HTML page of a HTTPS site with a cert it trust:

aborrero@tools-sgebastion-08:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://www.nuget.org")' | grep body
<body>\r
</body>\r

By the time of this writting, the mono version being used in Toolforge stretch grid is:

aborrero@tools-sgebastion-08:~$ apt-cache policy mono-runtime
mono-runtime:
  Installed: 5.12.0.226-0xamarin3+debian9b1
  Candidate: 5.12.0.226-0xamarin3+debian9b1
  Version table:
 *** 5.12.0.226-0xamarin3+debian9b1 1001
       1001 http://apt.wikimedia.org/wikimedia stretch-wikimedia/thirdparty/mono-project-stretch amd64 Packages
        100 /var/lib/dpkg/status

Details

SubjectRepoBranchLines +/-
toolforge: exclude DST_Root_CA_X3operations/puppetproduction+3 -0
sslcert: introduce ca_deselect_dstx3operations/puppetproduction+17 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

aborrero changed the task status from Open to In Progress.Oct 1 2021, 11:43 AM
aborrero triaged this task as High priority.
aborrero created this task.
aborrero moved this task from Backlog to In Progress on the Toolforge board.
aborrero moved this task from Inbox to Doing on the cloud-services-team (Kanban) board.
aborrero added a comment.Oct 1 2021, 11:50 AM

The upstream repo contains version 6.12.0 in the stable-stretch repo which is the one we're copying from for apt.wikimedia.org.

I will test that the version can cleanly install on the stretch grid.

Question: what if people have running tools that require the old mono version?

aborrero added a comment.Oct 1 2021, 11:55 AM

For the record, if that mono version installs cleanly on stretch (and we don't break our users) then to finally pull the packages we need this reprepro command:

root@apt1001:~# reprepro --noskipold --component thirdparty/mono-project-stretch update stretch-wikimedia
[..]

Since we have ensure => latest for the mono runtime in the Toolforge grid exec environ, puppet will magically update the packages everywhere in the grid.

But I would like to have confirmation that we won't be breaking arbitrary tools, cc @bd808 for a +1 or further comments.

Stashbot added a comment.Oct 1 2021, 12:12 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T12:12:02Z] <arturo> experimenting with newer mono runtime on toolsbeta-sgebastion-04 (T292289)

Stashbot added a comment.Oct 1 2021, 12:34 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T12:34:11Z] <arturo> rebooting toolsbeta-sgebastion-04 (T292289)

aborrero added a comment.Oct 1 2021, 12:57 PM

I updated the whole mono framework on toolsbeta-sgebastion-04 but I still see the exception when connecting to toolforge HTTPS endpoints.

The upstream documentation reads:

(Recommended) Starting with Mono 3.12.0 a new tool called cert-sync is included which syncs Mono’s certificate store with the system certificate store. It should run automatically when you install the official Mono packages.
Make sure the ca-certificates-mono package is installed.

Which is apparently a lie, because the package is installed just fine

[...]
Setting up ca-certificates-mono (6.12.0.122-0xamarin1+debian9b1) ...
Processing triggers for sgml-base (1.29) ...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for ca-certificates (20200601~deb9u2) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
Updating Mono key store
Mono Certificate Store Sync - version 6.12.0.122
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 142, your new list has 136
Import process completed.

Importing into BTLS system store:
I already trust 141, your new list has 136
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
Import process completed.
Done
done.

and I can use every single TLS client I could find (curl, wget, s_client, python, etc) they all accept the toolforge certificate, except mono.

I just checked in a buster Toolforge node, with similar results:

aborrero@tools-sgebastion-11:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://jobs.toolforge.org")'
System.Net.WebException: Error: TrustFailure (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
aborrero@tools-sgebastion-11:~$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://www.nuget.org")' | grep body
<body>\r
</body>\r
aborrero@tools-sgebastion-11:~$ apt-cache policy mono-runtime
mono-runtime:
  Installed: 5.18.0.240+dfsg-3
  Candidate: 5.18.0.240+dfsg-3
  Version table:
 *** 5.18.0.240+dfsg-3 500
        500 http://deb.debian.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status

So the exception may not be related to the mono version or the LE certificate thing after all?

bd808 added a subscriber: Reedy.EditedOct 1 2021, 2:51 PM

@Majavah found https://github.com/mono/mono/issues/21233#issuecomment-932211479. Looks like even the trunk mono uses a version of boringssl that has problems with the expired DST Root CA X3 and fails if it is in the trust store.

bd808 added a comment.Oct 1 2021, 2:59 PM

Related is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995432 with a comment by our own @BBlack. From chat seen on IRC I think @BBlack is working on some Puppet code to remove the DST Root CA X3 cert from the system trust store for WMF servers.

• MoritzMuehlenhoff added a comment.Oct 1 2021, 3:14 PM
In T292289#7394992, @bd808 wrote:

Related is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=995432 with a comment by our own @BBlack. From chat seen on IRC I think @BBlack is working on some Puppet code to remove the DST Root CA X3 cert from the system trust store for WMF servers.

FYI, it's this patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/725331/

bd808 added a comment.Oct 1 2021, 4:27 PM

I manually tested removing the DST Root CA X3 cert on tools-sgebastion-08 (dev.toolforge.org) and it seems to be working:

$ ssh dev.toolhub.org
$ grep DST_Root /etc/ca-certificates.conf
mozilla/DST_Root_CA_X3.crt
$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://en.wikipedia.org")'                                                                    System.Net.WebException: Error: TrustFailure (A call to SSPI failed, see inner e
xception.) ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
$ sudo sed -i 's#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#' /etc/ca-certificates.conf && sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 142, your new list has 135
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 141, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://en.wikipedia.org")'
"<!DOCTYPE html>
<html class="client-nojs" lang="en" dir="ltr">
<head>
...

Tests to *.toolforge.org URLs are also no longer failing with a TLS error, but seem to consistently return 403 Forbidden responses. I'm not sure why yet; the same urls work from curl from the same host. This appears to be caused by mono not sending a user-agent header by default. Something like csharp -e 'System.Net.WebClient c = new System.Net.WebClient (); c.Headers.Add ("user-agent", "bd808/1.0 (bot; bd808@wikimedia.org)"); c.DownloadString ("https://toolhub.toolforge.org/toolinfo.json")' works.

Stashbot added a comment.Oct 1 2021, 9:59 PM

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T21:59:37Z] <bd808> clush -w @all -b 'sudo sed -i "s#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#" /etc/ca-certificates.conf && sudo update-ca-certificates' for T292289

bd808 added a comment.Oct 1 2021, 10:06 PM
In T292289#7395911, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2021-10-01T21:59:37Z] <bd808> clush -w @all -b 'sudo sed -i "s#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#" /etc/ca-certificates.conf && sudo update-ca-certificates' for T292289

I hope this fixes things on the grid and bastions in the near term. Longer term we should follow https://gerrit.wikimedia.org/r/c/operations/puppet/+/725331/ and consider adding its sslcert::ca_deselect_dstx3 module across Cloud VPS.

Output from clush command:

tools-package-builder-04.tools.eqiad1.wikimedia.cloud: bash: warning: setlocale:
 LC_ALL: cannot change locale (en_US.UTF-8)
tools-package-builder-03.tools.eqiad1.wikimedia.cloud: ssh: connect to host tool
s-package-builder-03.tools.eqiad1.wikimedia.cloud port 22: No route to host
---------------
tools-acme-chief-[01-02].tools.eqiad1.wikimedia.cloud,tools-checker-04.tools.eqiad1.wikimedia.cloud,tools-clushmaster-02.tools.eqiad1.wikimedia.cloud,tools-docker-imagebuilder-01.tools.eqiad1.wikimedia.cloud,tools-docker-registry-[05-06].to
ols.eqiad1.wikimedia.cloud,tools-filesystemtest-1.tools.eqiad1.wikimedia.cloud,tools-k8s-control[1-3].tools.eqiad1.wikimedia.cloud,tools-k8s-etcd-[13-15].tools.eqiad1.wikimedia.cloud,tools-k8s-haproxy-[3-4].tools.eqiad1.wikimedia.cloud,tools-k8s-ingress-[4-6].tools.eqiad1.wikimedia.cloud,tools-k8s-worker-[30-62,64-79].tools.eqiad1.wikimedia.cloud,tools-legacy-redirector.tools.eqiad1.wikimedia.cloud,tools-mail-03.tools.eqiad1.wikimedia.cloud,tools-nfs-test-client-01.tools.eqiad1.wikimedia.cloud,tools-prometheus-[03,05].tools.eqiad1.wikimedia.cloud,tools-proxy-[05-06].tools.eqiad1.wikimedia.cloud,tools-redis-[1003-1004].tools.eqiad1.wikimedia.cloud,tools-services-05.tools.eqiad1.wikimedia.cloud,tools-sgegrid-mas
ter.tools.eqiad1.wikimedia.cloud,tools-sgegrid-shadow.tools.eqiad1.wikimedia.cloud,tools-static-14.tools.eqiad1.wikimedia.cloud (81)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
---------------
tools-sgebastion-07.tools.eqiad1.wikimedia.cloud,tools-sgecron-01.tools.eqiad1.w
ikimedia.cloud,tools-sgeexec-[0901,0904-0920,0932-0942].tools.eqiad1.wikimedia.c
loud,tools-sgewebgrid-generic-[0901-0902].tools.eqiad1.wikimedia.cloud,tools-sge
webgrid-lighttpd-[0911-0920].tools.eqiad1.wikimedia.cloud (43)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 142, your new list has 135
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 141, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
---------------
tools-elastic-[1-3].tools.eqiad1.wikimedia.cloud,tools-package-builder-04.tools.
eqiad1.wikimedia.cloud (4)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
done.
---------------
tools-sgeexec-[0947,0950-0952].tools.eqiad1.wikimedia.cloud (4)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 135, your new list has 135
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 134, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
---------------
tools-sgebastion-[10-11].tools.eqiad1.wikimedia.cloud (2)
---------------
Updating certificates in /etc/ssl/certs...
0 added, 1 removed; done.
Running hooks in /etc/ca-certificates/update.d...

Removing debian:DST_Root_CA_X3.pem
done.
Updating Mono key store
Mono Certificate Store Sync - version 5.18.0.240
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 146, your new list has 146
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 145, your new list has 146
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.
clush: tools-package-builder-03.tools.eqiad1.wikimedia.cloud: exited with exit code 255
---------------
tools-sgebastion-08.tools.eqiad1.wikimedia.cloud
---------------
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

done.
Updating Mono key store
Mono Certificate Store Sync - version 5.12.0.226
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 141, your new list has 135
Import process completed.

Importing into BTLS system store:
I already trust 140, your new list has 135
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
Import process completed.
Done
done.
Stashbot added a comment.Oct 2 2021, 9:29 PM

Mentioned in SAL (#countervandalism) [2021-10-02T21:29:06Z] <Krinkle> krinkle@cvn-app9 sudo sed -i 's#mozilla/DST_Root_CA_X3.crt#!mozilla/DST_Root_CA_X3.crt#' /etc/ca-certificates.conf && sudo update-ca-certificates ref T292289, ref https://github.com/mono/mono/issues/21233

Chicocvenancio removed a project: PAWS.Oct 3 2021, 7:49 PM
Chicocvenancio subscribed.

I think this is not relevant to PAWS as we don't have mono installed. Please correct me if this is wrong.

aborrero changed the task status from In Progress to Open.Oct 4 2021, 10:15 AM
aborrero lowered the priority of this task from High to Medium.

Thanks you all for working on this. I'm reducing priority and will follow up in the next few days if sslcert::ca_deselect_dstx3 gets finally merged.

Vort subscribed.Oct 6 2021, 5:08 AM
• MoritzMuehlenhoff added a comment.Oct 11 2021, 8:17 AM

In the latest upload of ca-certificates to Debian unstable, the old X3 cert has now been removed:
https://tracker.debian.org/news/1265630/accepted-ca-certificates-20211004-source-into-unstable/

aborrero closed this task as Resolved.Oct 18 2021, 3:29 PM
Vort added a comment.Oct 18 2021, 3:34 PM

@aborrero I think that you are wrong about Resolved status.
I still see Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED on my bots.

aborrero reopened this task as Open.Oct 18 2021, 4:38 PM
In T292289#7436978, @Vort wrote:

@aborrero I think that you are wrong about Resolved status.
I still see Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED on my bots.

Sorry about that. I'm reopening and will take a closer look in the next few days.

aborrero added a comment.Oct 22 2021, 11:30 AM

hey @Vort can you please share the name of the tools that are experiencing issues related to this?

Vort added a comment.Oct 22 2021, 11:59 AM

@aborrero my bots are located at wikitasks tool.
You can test ./run.sh wp_cyrlat and look at cat wp_cyrlat.out | tail -n 30.

gerritbot added a comment.Oct 29 2021, 10:59 AM

Change 735599 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] sslcert: introduce ca_deselect_dstx3

https://gerrit.wikimedia.org/r/735599

gerritbot added a project: Patch-For-Review.Oct 29 2021, 10:59 AM

Change 735600 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):

[operations/puppet@production] toolforge: exclude DST_Root_CA_X3

https://gerrit.wikimedia.org/r/735600

gerritbot added a comment.Oct 29 2021, 11:02 AM

Change 735599 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] sslcert: introduce ca_deselect_dstx3

https://gerrit.wikimedia.org/r/735599

gerritbot added a comment.Oct 29 2021, 11:02 AM

Change 735600 merged by Arturo Borrero Gonzalez:

[operations/puppet@production] toolforge: exclude DST_Root_CA_X3

https://gerrit.wikimedia.org/r/735600

Maintenance_bot removed a project: Patch-For-Review.Oct 29 2021, 11:10 AM
aborrero closed this task as Resolved.Oct 29 2021, 11:55 AM

hey @Vort I've reviewed the situation with your tool and here are my findings:

  • your tool embeds a copy of the mono framework, see:
tools.wikitasks@tools-sgebastion-07:~$ ls -la | grep mono
drwxr-sr-x    7 tools.wikitasks tools.wikitasks       4096 Nov  7  2020 mono
drwxr-sr-x    3 tools.wikitasks tools.wikitasks       4096 Feb  8  2019 .mono
drwxr-sr-x   20 tools.wikitasks tools.wikitasks       4096 Nov  7  2020 mono-6.12.0.90
-rw-r--r--    1 tools.wikitasks tools.wikitasks  293275732 Sep  3  2020 mono-6.12.0.90.tar.xz
  • the system-installed mono can run your task:
root@tools-sgebastion-07:/data/project/wikitasks/wp_cyrlat# mono WikiTasks.exe 
Authenticating... Done
Scanning page titles...................^C
  • your tool user has some kind of environment override that prevents it from using the system-installed mono framework:
tools.wikitasks@tools-sgebastion-07:~/wp_cyrlat$ mono WikiTasks.exe 
Authenticating...
Unhandled Exception:
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: Error: TrustFailure (A call to SSPI failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-5.12.0.226/external/boringssl/ssl/handshake_client.c:1132
[..]
  • I though that perhaps the issue was your embedded mono copy wasn't aware of the certificate changes, so I tried a manual sync:
tools.wikitasks@tools-sgebastion-07:~$ mono/bin/cert-sync /etc/ssl/certs/ca-certificates.crt
Mono Certificate Store Sync - version 6.12.0.90
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 145, your new list has 139
Import process completed.

Importing into BTLS system store:
I already trust 144, your new list has 139
Certificate added: OU=GlobalSign ECC Root CA - R5, O=GlobalSign, CN=GlobalSign
1 new root certificates were added to your trust store.
Import process completed.
  • but unfortunately the problem still persists:
tools.wikitasks@tools-sgebastion-07:~/wp_cyrlat$ mono WikiTasks.exe 
Authenticating...
Unhandled Exception:
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request ---> System.Net.WebException: Error: TrustFailure (A call to SSPI failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
  at /build/mono-5.12.0.226/external/boringssl/ssl/handshake_client.c:1132

I suggest you try reinstalling your embedded copy of the mono framework, and see if that solves the problem?

I take you are using the embedded copy because you need a newer version, no?

  • we are close to upgrading the grid to Debian Buster, it will include a newer version of the mono framework
  • if not, we're ready to install a backported mono framework 6.12.0.122 in the current Debian Stretch-based grid, but that would need coordination among all mono users in Toolforge.

I believe the toolforge-wide issue with the bad certificate is solved so I'm closing this task. Please open another ticket if you need specific assistance with your tool.

Vort added a comment.Oct 29 2021, 11:59 AM

Fresh version of mono was needed to debug other issue.
I thought that it was not used in latest runs.
Will recheck and try to disable it.

Vort added a comment.Oct 29 2021, 12:10 PM

@aborrero look at lines in your quoted logs:
at /build/mono-5.12.0.226/external/boringssl/ssl/handshake_client.c:1132
5.12.0.226 is the system version, right?

I even moved mono directory to _mono to exclude possible interference.
Nothing changed.

Vort added a comment.Oct 29 2021, 12:45 PM

@aborrero I made a simple test program, which should print MediaWiki version.
Please try to run it from some other tool account: mono WikiTLSTest.exe.
Until success I think that it is too early to set Resolved status.
Source code:

using System;
using System.Net;
using System.Xml;

namespace WikiTLSTest
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine($"WikiTLSTest check is started.");
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;

            var wc = new WebClient();
            wc.Headers.Add("User-Agent", "WikiTLSTest");

            var xml = wc.DownloadString(
                "https://ru.wikipedia.org/w/api.php" +
                "?action=query&format=xml&meta=siteinfo");

            XmlDocument doc = new XmlDocument();
            doc.LoadXml(xml);

            var generalNode = doc.SelectSingleNode("/api/query/general");

            Console.WriteLine($"WikiTLSTest result: {generalNode.Attributes["generator"].Value}");
        }
    }
}
Vort mentioned this in T295220: CERTIFICATE_VERIFY_FAILED error when trying to access Wikipedia API with Mono.Nov 6 2021, 9:27 AM
Vort mentioned this in T320168: Migrate wikitasks from Toolforge GridEngine to Toolforge Kubernetes.Oct 7 2022, 6:31 AM
bd808 mentioned this in T311466: Create a kubernetes container with mono and dotnet.Oct 19 2022, 5:57 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits