Page MenuHomePhabricator

Create a kubernetes container with mono and dotnet
Open, Needs TriagePublic

Description

My MilHist bots use C# via mono. Create a kubernetes container with mono and msbuild. Preferably the latest stable version of mono. (6.12.0)

Event Timeline

taavi added a subscriber: taavi.

As someone not very familiar with the C#/.net/Mono ecosystem, I have a few questions which would be really helpful for making this happen.

  • What does package management look like? Is there some kind of a package manager (like pip, composer, npm, ...) that people use to install packages, or is relying on the os package manager (apt) prefered instead?
    • I see that on Debian, the mono package itself includes quite a few libraries and a meta-package to install them all.. are there third-party libraries not included in that bundle?
  • What kind of configuration or setup do web services require? Does the mono app run a web server by itself, or is an external web server such as apache2, nginx or lighttpd required?

@taavi I'm not sure I can help you with these questions. I just run mono myApp.exe and everything runs. I guess any dependencies I have are covered in that default mono package. There a few DLLs which I need and are not delivered by the system; I just reference them when building the exe (compiling).

Maybe @Hawkeye7 can help more.

Running a web service would require a an external web server. It is not part of mono.

However, my bots do not run as a web service, but as cron jobs.

I recommend just installing mono on Debian per https://www.mono-project.com/download/stable/#download-lin-debian

Change 840327 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] mono68-sssd: New image

https://gerrit.wikimedia.org/r/840327

How do you create an interactive shell on this container?

  • What kind of configuration or setup do web services require? Does the mono app run a web server by itself, or is an external web server such as apache2, nginx or lighttpd required?

I'm using a lighttpd web server (default web server on Toolforge). A file .lighttpd.conf contains this strings:

static-file.exclude-extensions += ( ".cgi" )
cgi.assign += ( ".cgi" => "/usr/bin/mono" )

This is result: https://mbh.toolforge.org/pages-wo-iwiki.cgi

mono68-sssd looks like a good starting point, please let me know when I can use toolforge-jobs run ... --image tf-mono68 thanks!

tools.botorder@tools-sgebastion-10:~/staging$ toolforge-jobs run test --command $PWD/test.sh  --image tf-mono68
ERROR: unable to create job: "HTTP 400: invalid container image"

@Ghuron

Change 840327 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] mono68-sssd: New image

https://gerrit.wikimedia.org/r/840327

Change 842340 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/jobs-framework-api@main] Add tf-mono68 image

https://gerrit.wikimedia.org/r/842340

Change 842340 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] Add tf-mono68 image

https://gerrit.wikimedia.org/r/842340

The new container is now available on the jobs framework. I'm not yet sure about the web variant, will get back to that soon. Same thing applies to an interactive shell, since that's currently linked to the webservice tooling.

When trying to run mono executable inside this image, I'm getting exception:

[ERROR] FATAL UNHANDLED EXCEPTION: System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.) ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

at ./external/boringssl/ssl/handshake_client.c:1132

Fast googling reveals that this might be about about OS/mono certificate store. Both require sudo, so I guess I cannot fix it on my side.
Obviously without the ability to establish https connections, I cannot use the mediawiki API, so none of my bots can run as of now.

No good. Does not have msbuild.

Having the same problem as Ghuron

10:37 14 October 2022 Error: TrustFailure (Authentication failed, see inner exception.)

No good. Does not have msbuild.

Does the grid engine environment have msbuild? I'm not able to find it using which msbuild. I'm not seeing any evidence in Puppet that we install any mono packages other than mono-complete and mono-fastcgi-server or that either of those packages have historically included the msbuild tool.

It seems what is being asked for is a dotnet6 sdk, which as far as I can tell, isn't packaged in Debian. I do see it packaged in Fedora, and it seems like it has msbuild you are looking for? Have a look: https://packages.fedoraproject.org/pkgs/dotnet6.0/dotnet-sdk-6.0/fedora-rawhide.html#files.

If so, creating an image would be easiest by having a debian package to install. It seems Microsoft provides a debian repo with packages, but it's unclear what licensing they are under.: https://packages.microsoft.com/debian/11/prod/.

I am curious what is different about this image versus the grid environment? The existing mono package on the grid is running tools successfully now yes? Do we know why an image with similar mono support doesn't work?

We use Rocky 9 here. All software here is patched an kept up to date due to the threat from foreign-government-sponsored hackers and ransomware attacks.

I strongly urge that the latest version of mono be installed. A package is available. You can find the instructions here:

https://www.mono-project.com/download/stable/#download-lin-debian

I have tested this install on Rocky 9.

We used to have msbuild on the grid, but lost it when we were forced onto a server with an older version of mono installed. Since then I haven't been able to rebuild my C# bots.

They do still run though, but I get occasional errors.

msbuild is available in the current version of mono.

Installing the latest stable version not only means that I can rebuild them on the k8s images, I can also build and test them here where I have a debugging environment and be certain that they will compile and run in the image.

I want to emphasize that right now https-connections cannot be established which makes tf-mono68 practically useless in toolforge environment. Steps to reproduce:

  1. Create a test.cs text file with the following content:
class Test {
  public static void Main(){
    new System.Net.WebClient().DownloadString("https://en.wikipedia.org/");
  }
}
  1. Execute mcs test.cs to compile it to test.exe
  2. Run toolforge-jobs run test --command "mono test.exe" --image tf-mono68 and check test.err

The documentation says: "The package ca-certificates-mono should be installed to get SSL certificates for HTTPS connections. Install this package if you run into trouble making HTTPS connections."

The documentation says: "The package ca-certificates-mono should be installed to get SSL certificates for HTTPS connections. Install this package if you run into trouble making HTTPS connections."

I tried adding this in a local build of the container and then using the information from T311466#8323940 to do a test. I am still seeing a "Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" fatal error. This tickled something in my brain though which led me to rediscover https://github.com/mono/mono/issues/21233 and specifically the workaround from https://github.com/mono/mono/issues/21233#issuecomment-932211479. When I apply that certificate exclusion manually in the container (which is running as root in my local tests) the mono test.exe command runs to completion.

This tickled something in my brain though which led me to rediscover https://github.com/mono/mono/issues/21233 and specifically the workaround from https://github.com/mono/mono/issues/21233#issuecomment-932211479. When I apply that certificate exclusion manually in the container (which is running as root in my local tests) the mono test.exe command runs to completion.

T292289: Toolforge mono version on stretch grid doesn't trust latest LE certs

Change 844512 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/docker-images/toollabs-images@master] mono68: Remove expired DST Root CA X3 cert

https://gerrit.wikimedia.org/r/844512

Well done.

I am still unsure as to how the k8s concept is supposed to work. Previously my workflow was:
(1) Check out the sources from git under my account
(2) Build the exes and dlls on my account using msbuild and the csproj project file
(3) Copy them to the milhistbot account
(4) Run then using cron

Currently step 2 does not work due to the loss of msbuild, so I cannot rebuild the project.

Now none of these steps are available with k8s.

I can try:
(1) Build with mono 6.12 using msbuild and the csproj project file on my Red Hat server
(2) Copy the exes and dlls to my account
(3) Copy them again to the milhistbot account
(4) Run them using tool-forge jobs with on schedule

This is less than ideal; it involves building with a newer version of mono than we are running, and on a different platform, risking compatibility issues. And involves sftp copying binary files around. For some reason I have to become milhistbot before being able to use the container.

Although the idea of building things INSIDE execution container seems awkward to me, upgrading mono to the latest stable (6.12.0.182) sounds like a right thing to do

Change 844512 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] mono68: Remove expired DST Root CA X3 cert

https://gerrit.wikimedia.org/r/844512

Mentioned in SAL (#wikimedia-cloud) [2022-10-20T10:54:34Z] <taavi> rebuild mono68-sssd image with the expired DST Root CA X3 removed T311466

Confirm that code executed in tf-mono68 can access mediawiki API
Will continue testing

Confirmed that my executables run on the docker container.

I did get one transient error:

SecureChannelFailure (Unable to read data from the transport connection: Connection reset by peer.)

Per Ghuron, an alternative workflow would be:
(1) Check out the sources from git under my account
(2) Copy them to the bot account
(3) Build them on the container with a rebuild toolforge-job
(4) Run them using toolforge-jobs with on schedule

This would require msbuild to be available, which in turn would require mono 6.12

Per Ghuron, an alternative workflow would be:

Theoretically one can build a bot executable using C# compiler (mcs is mono 6.08), but I believe we need the latest 6.12 for many other reasons.
I was having problems with 6.08 connecting sites, that require TLS 1.3 (which is rare at the moment, but would not be so rare in the near future)

This comment was removed by Kotz.