Page MenuHomePhabricator

Provide an up-to-date mono environment on toolforge
Closed, ResolvedPublic

Description

On T194380 we discovered several bots/tools running on toolforge using .NET / mono that aren't able to talk modern TLS and that would be unable to connect to WMF sites after the deprecation of AES128-SHA is done as part of T192555.

I've tested the TLS capabilities of the mono environment using a pretty simple bot that just logins into wikipedia and exits:

hello.cs
using System;
using DotNetWikiBot;

class MyHelloBot : Bot
{
    public static void Main()
    {
        Site site = new Site("https://en.wikipedia.org", "VGutiérrez (WMF)@vgutierrez-test-bot", "password");
    }
}

This code run on a trusty container with mono-project latest stable mono version (5.12) and a stock DotNetWikiBot 3.15 it's able to connect to en.wikipedia.org using TLS 1.2 and ECDHE-ECDSA-CHACHA20-POLY1305 as the ciphersuite.

Dockerfile
FROM ubuntu:trusty
RUN apt-get update && apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF && \
    apt-get install -y apt-transport-https && \
    echo "deb https://download.mono-project.com/repo/ubuntu stable-trusty main" | tee /etc/apt/sources.list.d/mono-official-stable.list && \
    apt-get update && apt-get install -y mono-devel

WORKDIR /app
COPY . .
RUN mono-csc -debug+ -optimize- -reference:DotNetWikiBot.Build.for.Mono.dll hello.cs
CMD mono --debug hello.exe

So it would be great if we could provide an up-to-date mono environment on toolforge that's able to speak modern TLS and comply with our own TLS requirements :)

Event Timeline

Vgutierrez triaged this task as Medium priority.May 14 2018, 3:56 PM
Vgutierrez created this task.

Latest mono (5.10.x) is not even in Debian yet: https://tracker.debian.org/pkg/mono

As of this message, the version table in Debian is:

o-o-stable: 2.10.8.1-8+deb7u1
o-o-sec: 2.10.8.1-8+deb7u1
oldstable: 3.2.8+dfsg-10
stable: 4.6.2.7+dfsg-1
testing: 4.6.2.7+dfsg-1
unstable: 4.6.2.7+dfsg-1

In Ubuntu, latest available version is 4.6.2.7+dfsg-1ubuntu1, and this is only in the last Ubuntu releases.

I mention this information here since @chasemp suggested we could backport the packages to put into our trusty-tool apt repo.

From TLS point of view we should at least aim to mono 4.8.0 because TLS 1.2 support was introduced in that version: http://www.mono-project.com/docs/about-mono/releases/4.8.0/#tls-12-support

Latest mono (5.10.x) is not even in Debian yet: https://tracker.debian.org/pkg/mono

As of this message, the version table in Debian is:

o-o-stable: 2.10.8.1-8+deb7u1
o-o-sec: 2.10.8.1-8+deb7u1
oldstable: 3.2.8+dfsg-10
stable: 4.6.2.7+dfsg-1
testing: 4.6.2.7+dfsg-1
unstable: 4.6.2.7+dfsg-1

In Ubuntu, latest available version is 4.6.2.7+dfsg-1ubuntu1, and this is only in the last Ubuntu releases.

I mention this information here since @chasemp suggested we could backport the packages to put into our trusty-tool apt repo.

http://www.mono-project.com/download/stable/#download-lin-ubuntu
http://www.mono-project.com/download/stable/#download-lin-debian

Just to be sure, I ran the tests using a docker container with buster:

Dockerfile
FROM debian:buster
RUN apt-get update && apt-get install -y mono-devel

WORKDIR /app
COPY . .
RUN mono-csc -debug+ -optimize- -reference:DotNetWikiBot.Build.for.Mono.dll hello.cs
CMD mono --debug hello.exe

and the resulting binary uses TLSv1.0 and AES128-SHA as a ciphersuite (note that we offer several decent (with FS support) ciphersuites for TLSv1.0 (ECDHE-ECDSA-AES128-SHA, ECDHE-RSA-AES128-SHA and DHE-RSA-AES128-SHA).

mono version
root@74c9115bfa43:/app# mono --version
Mono JIT compiler version 4.6.2 (Debian 4.6.2.7+dfsg-1)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen

great, this could be an easy solution.

I have some concerns on using external repos. It could be catastrophic with regards to the integration with the rest of the system. I'm not sure if it worth the risk, considering the scope of the problem.
I always prefer using official debian/ubuntu package as long as possible.

For the sake of completeness, have we considered not being strict regarding the TLS version (given neither Debian or Ubuntu fully support it).

Change 433142 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toollabs: add mono_external class

https://gerrit.wikimedia.org/r/433142

We will also explore the possibility of importing the whole framework in the main reprepro apt repo. Either all the packages or a 'pointer' to the external repo. @MoritzMuehlenhoff what do you think?

We will also explore the possibility of importing the whole framework in the main reprepro apt repo. Either all the packages or a 'pointer' to the external repo. @MoritzMuehlenhoff what do you think?

What you can do it to sync the Mono debs (defined via modules/aptrepo/files/updates) into a custom reprepro component (defined via modules/aptrepo/files/distributions-wikimedia). But if that's limited to a few Toolforge instances (I can't imagine that we use those Mono debs in production), you could just as well simply use the external repository directly.

I have some concerns on using external repos. It could be catastrophic with regards to the integration with the rest of the system. I'm not sure if it worth the risk, considering the scope of the problem.
I always prefer using official debian/ubuntu package as long as possible.

Exactly. https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Local_package_policy

I have some concerns on using external repos. It could be catastrophic with regards to the integration with the rest of the system. I'm not sure if it worth the risk, considering the scope of the problem.
I always prefer using official debian/ubuntu package as long as possible.

Exactly. https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Local_package_policy

Well, I ignored we had such a policy.

I therefore summon @bd808 for additional advice. Summary:

  • importing the whole mono-project repo into prod reprepro seems overkill, given the only user would be toolforge
  • adding the external repo to each toolforge node seems to go against our own policy

Both things above could be done anyway. We just need to decide what is better for us.

Others option could be to recommend Mono/.NET users to migrate the affected tools to the kubernetes backend, and create a docker image with a modern mono framework.

I have some concerns on using external repos. It could be catastrophic with regards to the integration with the rest of the system. I'm not sure if it worth the risk, considering the scope of the problem.
I always prefer using official debian/ubuntu package as long as possible.

Exactly. https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Local_package_policy

Well, I ignored we had such a policy.

I therefore summon @bd808 for additional advice. Summary:

  • importing the whole mono-project repo into prod reprepro seems overkill, given the only user would be toolforge

I was unaware of the policy wrt external packages, but if that's the policy we can just as well do that; the effort isn't _that_ big:

  • Create a component in apt.wikimedia.org for trusty and jessie (if both are still needed, maybe trusty can be skipped)
  • Create a reprepro config to sync the packages from the Mono repo to apt.wikimedia.org
  • Add puppetisation to include the mono48 component to Toolforge

I was unaware of the policy wrt external packages, but if that's the policy we can just as well do that; the effort isn't _that_ big:

We can revisit the policy as well. As far as I know we have not thought deeply about this issue for a while. The main concern we would need to address in an updated policy which allows foreign repos is some criteria for evaluating their trustworthiness. Do you have opinions on this @MoritzMuehlenhoff?

Change 433996 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] reprepro: fetch mono suite from upstream apt repo

https://gerrit.wikimedia.org/r/433996

Ok, so the patch above does configures reprepro to fetch the mono-project repo into our own apt repo. Please @MoritzMuehlenhoff review :-P

I imported the external repo key with:

root@install1002:~# gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
gpg: requesting key 0xA6A19B38D3D831EF from hkp server keyserver.ubuntu.com
gpg: key 0xA6A19B38D3D831EF: public key "Xamarin Public Jenkins (auto-signing) <releng@xamarin.com>" imported
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Change 433996 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] reprepro: fetch mono suite from upstream apt repo

https://gerrit.wikimedia.org/r/433996

Change 434507 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] reprepro: add missing UDebComponents: stanza for mono-project repos

https://gerrit.wikimedia.org/r/434507

Change 434507 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] reprepro: add missing UDebComponents: stanza for mono-project repos

https://gerrit.wikimedia.org/r/434507

Change 434536 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] reprepro: use right syntax for grep-dctrl

https://gerrit.wikimedia.org/r/434536

Change 434536 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] reprepro: use right syntax for grep-dctrl

https://gerrit.wikimedia.org/r/434536

How to use this version of mono, instead of standard Toolforge version 3.2.8?

In T194665#4224612, @MaxBioHazard wrote:

How to use this version of mono, instead of standard Toolforge version 3.2.8?

We are working on it. I think we will make an announcement when the new version becomes available.

If you are currently using mono on Toolforge, do you see any inconvenience in migrating to the newer version?
Will your code break if that happens? Are you using the mono framework from the installed packages, right?

I use mono on Toolforge (see T194380) and doesn't see any inconvenience, conversely I'm waiting for better stability and less number of bugs. My code will not break, I'm not satisfied with current mono version on Toolforge, I compile mono 4.8 in my personal directory and use it instead of Toolforge mono version.

Are you using the mono framework from the installed packages, right?

I don't understand this question.

Mentioned in SAL (#wikimedia-cloud) [2018-05-28T12:06:19Z] <arturo> T194665 adding mono packages to apt.wikimedia.org for trusty-wikimedia

Mentioned in SAL (#wikimedia-operations) [2018-05-28T12:08:22Z] <arturo> T194665 aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-jessie' update jessie-wikimedia

Mentioned in SAL (#wikimedia-cloud) [2018-05-28T12:09:02Z] <arturo> T194665 adding mono packages to apt.wikimedia.org for jessie-wikimedia and stretch-wikimedia

Mentioned in SAL (#wikimedia-operations) [2018-05-28T12:09:57Z] <arturo> T194665 aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-stretch' update stretch-wikimedia

aborrero claimed this task.

The mono framework is now upgraded.

In T194665#4224612, @MaxBioHazard wrote:

How to use this version of mono, instead of standard Toolforge version 3.2.8?

mono has been updated to version 5.12.0.226 on toolforge. If you want to use it, it should be as easy as using the mono installed in the system rather the one you have in your home directory.

@Dmitry89 we've seen some of your tool instances using successfully mono 5.12, please could you restart the long running ones like inc_remindbot (https://tools.wmflabs.org/admin/oge/status#host-tools-exec-1402) to take benefit from the update? Thanks!

mono has been updated to version 5.12.0.226 on toolforge. If you want to use it, it should be as easy as using the mono installed in the system rather the one you have in your home directory.

Is it needed to use flag MONO_TLS_PROVIDER=btls ?

Is it needed to use flag MONO_TLS_PROVIDER=btls ?

No, you can drop it if you use the new mono version provided by toolforge

Change 433142 abandoned by Vgutierrez:
toollabs: add mono_external class

Reason:
T194665 has been resolved

https://gerrit.wikimedia.org/r/433142

Vvjjkkii renamed this task from Provide an up-to-date mono environment on toolforge to yzcaaaaaaa.Jul 1 2018, 1:10 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii removed aborrero as the assignee of this task.
Vvjjkkii raised the priority of this task from Medium to High.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed subscribers: gerritbot, Aklapper.
Vgutierrez renamed this task from yzcaaaaaaa to Provide an up-to-date mono environment on toolforge.Jul 1 2018, 8:05 AM
Vgutierrez closed this task as Resolved.
Vgutierrez assigned this task to aborrero.
Vgutierrez lowered the priority of this task from High to Medium.
Vgutierrez updated the task description. (Show Details)
Vgutierrez added a subscriber: Aklapper.