@nskaggs

@taavi I'm not sure I can help you with these questions. I just run mono myApp.exe and everything runs. I guess any dependencies I have are covered in that default mono package. There a few DLLs which I need and are not delivered by the system; I just reference them when building the exe (compiling).

Maybe @Hawkeye7 can help more.

Running a web service would require a an external web server. It is not part of mono.

However, my bots do not run as a web service, but as cron jobs.

I recommend just installing mono on Debian per https://www.mono-project.com/download/stable/#download-lin-debian

Change 840327 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/docker-images/toollabs-images@master] mono68-sssd: New image

https://gerrit.wikimedia.org/r/840327

How do you create an interactive shell on this container?

• What kind of configuration or setup do web services require? Does the mono app run a web server by itself, or is an external web server such as apache2, nginx or lighttpd required?

I'm using a lighttpd web server (default web server on Toolforge). A file .lighttpd.conf contains this strings:

static-file.exclude-extensions += ( ".cgi" )
cgi.assign += ( ".cgi" => "/usr/bin/mono" )

This is result: https://mbh.toolforge.org/pages-wo-iwiki.cgi

mono68-sssd looks like a good starting point, please let me know when I can use toolforge-jobs run ... --image tf-mono68 thanks!

tools.botorder@tools-sgebastion-10:~/staging$ toolforge-jobs run test --command $PWD/test.sh  --image tf-mono68
ERROR: unable to create job: "HTTP 400: invalid container image"

@Ghuron

Change 840327 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] mono68-sssd: New image

https://gerrit.wikimedia.org/r/840327

Mentioned in SAL (#wikimedia-cloud) [2022-10-12T23:25:34Z] <bd808> Rebuilding all Toolforge docker images (T278436, T311466, T293552)

Change 842340 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/toolforge/jobs-framework-api@main] Add tf-mono68 image

https://gerrit.wikimedia.org/r/842340

Change 842340 merged by jenkins-bot:

[cloud/toolforge/jobs-framework-api@main] Add tf-mono68 image

https://gerrit.wikimedia.org/r/842340

Mentioned in SAL (#wikimedia-cloud-feed) [2022-10-14T07:54:02Z] <wm-bot2> deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (0cc020e) (T311466) - cookbook ran by taavi@runko

The new container is now available on the jobs framework. I'm not yet sure about the web variant, will get back to that soon. Same thing applies to an interactive shell, since that's currently linked to the webservice tooling.

When trying to run mono executable inside this image, I'm getting exception:

[ERROR] FATAL UNHANDLED EXCEPTION: System.AggregateException: One or more errors occurred. (The SSL connection could not be established, see inner exception.) ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

at ./external/boringssl/ssl/handshake_client.c:1132

Fast googling reveals that this might be about about OS/mono certificate store. Both require sudo, so I guess I cannot fix it on my side.
Obviously without the ability to establish https connections, I cannot use the mediawiki API, so none of my bots can run as of now.

No good. Does not have msbuild.

Having the same problem as Ghuron

10:37 14 October 2022 Error: TrustFailure (Authentication failed, see inner exception.)

In T311466#8316470, @Hawkeye7 wrote:

No good. Does not have msbuild.

Does the grid engine environment have msbuild? I'm not able to find it using which msbuild. I'm not seeing any evidence in Puppet that we install any mono packages other than mono-complete and mono-fastcgi-server or that either of those packages have historically included the msbuild tool.

It seems what is being asked for is a dotnet6 sdk, which as far as I can tell, isn't packaged in Debian. I do see it packaged in Fedora, and it seems like it has msbuild you are looking for? Have a look: https://packages.fedoraproject.org/pkgs/dotnet6.0/dotnet-sdk-6.0/fedora-rawhide.html#files.

If so, creating an image would be easiest by having a debian package to install. It seems Microsoft provides a debian repo with packages, but it's unclear what licensing they are under.: https://packages.microsoft.com/debian/11/prod/.

I am curious what is different about this image versus the grid environment? The existing mono package on the grid is running tools successfully now yes? Do we know why an image with similar mono support doesn't work?

We use Rocky 9 here. All software here is patched an kept up to date due to the threat from foreign-government-sponsored hackers and ransomware attacks.

I strongly urge that the latest version of mono be installed. A package is available. You can find the instructions here:

https://www.mono-project.com/download/stable/#download-lin-debian

I have tested this install on Rocky 9.

We used to have msbuild on the grid, but lost it when we were forced onto a server with an older version of mono installed. Since then I haven't been able to rebuild my C# bots.

They do still run though, but I get occasional errors.

msbuild is available in the current version of mono.

Installing the latest stable version not only means that I can rebuild them on the k8s images, I can also build and test them here where I have a debugging environment and be certain that they will compile and run in the image.

I want to emphasize that right now https-connections cannot be established which makes tf-mono68 practically useless in toolforge environment. Steps to reproduce:

1. Create a test.cs text file with the following content:

class Test {
  public static void Main(){
    new System.Net.WebClient().DownloadString("https://en.wikipedia.org/");
  }
}

2. Execute mcs test.cs to compile it to test.exe
3. Run toolforge-jobs run test --command "mono test.exe" --image tf-mono68 and check test.err

The documentation says: "The package ca-certificates-mono should be installed to get SSL certificates for HTTPS connections. Install this package if you run into trouble making HTTPS connections."

In T311466#8327616, @Hawkeye7 wrote:

The documentation says: "The package ca-certificates-mono should be installed to get SSL certificates for HTTPS connections. Install this package if you run into trouble making HTTPS connections."

I tried adding this in a local build of the container and then using the information from T311466#8323940 to do a test. I am still seeing a "Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" fatal error. This tickled something in my brain though which led me to rediscover https://github.com/mono/mono/issues/21233 and specifically the workaround from https://github.com/mono/mono/issues/21233#issuecomment-932211479. When I apply that certificate exclusion manually in the container (which is running as root in my local tests) the mono test.exe command runs to completion.

In T311466#8330113, @bd808 wrote:

This tickled something in my brain though which led me to rediscover https://github.com/mono/mono/issues/21233 and specifically the workaround from https://github.com/mono/mono/issues/21233#issuecomment-932211479. When I apply that certificate exclusion manually in the container (which is running as root in my local tests) the mono test.exe command runs to completion.

T292289: Toolforge mono version on stretch grid doesn't trust latest LE certs

Change 844512 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):

[operations/docker-images/toollabs-images@master] mono68: Remove expired DST Root CA X3 cert

https://gerrit.wikimedia.org/r/844512

Well done.

I am still unsure as to how the k8s concept is supposed to work. Previously my workflow was:
(1) Check out the sources from git under my account
(2) Build the exes and dlls on my account using msbuild and the csproj project file
(3) Copy them to the milhistbot account
(4) Run then using cron

Currently step 2 does not work due to the loss of msbuild, so I cannot rebuild the project.

Now none of these steps are available with k8s.

I can try:
(1) Build with mono 6.12 using msbuild and the csproj project file on my Red Hat server
(2) Copy the exes and dlls to my account
(3) Copy them again to the milhistbot account
(4) Run them using tool-forge jobs with on schedule

This is less than ideal; it involves building with a newer version of mono than we are running, and on a different platform, risking compatibility issues. And involves sftp copying binary files around. For some reason I have to become milhistbot before being able to use the container.

Although the idea of building things INSIDE execution container seems awkward to me, upgrading mono to the latest stable (6.12.0.182) sounds like a right thing to do

Change 844512 merged by jenkins-bot:

[operations/docker-images/toollabs-images@master] mono68: Remove expired DST Root CA X3 cert

https://gerrit.wikimedia.org/r/844512

Mentioned in SAL (#wikimedia-cloud) [2022-10-20T10:54:34Z] <taavi> rebuild mono68-sssd image with the expired DST Root CA X3 removed T311466

Confirm that code executed in tf-mono68 can access mediawiki API
Will continue testing

Confirmed that my executables run on the docker container.

I did get one transient error:

SecureChannelFailure (Unable to read data from the transport connection: Connection reset by peer.)

Per Ghuron, an alternative workflow would be:
(1) Check out the sources from git under my account
(2) Copy them to the bot account
(3) Build them on the container with a rebuild toolforge-job
(4) Run them using toolforge-jobs with on schedule

This would require msbuild to be available, which in turn would require mono 6.12

Per Ghuron, an alternative workflow would be:

Theoretically one can build a bot executable using C# compiler (mcs is mono 6.08), but I believe we need the latest 6.12 for many other reasons.
I was having problems with 6.08 connecting sites, that require TLS 1.3 (which is rare at the moment, but would not be so rare in the near future)

An alternative would be to install the latest version of dotnet

Best practice for containers is for each app to have its own. It is regarded as an error to use them like virtual machines. See https://cloud.google.com/architecture/best-practices-for-building-containers

@Hawkeye7 we are testing build service beta which uses buildpacks on Toolforge.
Do you think an upstream mono buildpack will work for you?

Take a look at our quickstart guide here.
We will be glad to receive feedback after you give it a try.

We are tracking this project on this board using the tag: "toolforge_build_service_beta_release"

Seeing as heroku doesn't seem to have a dotnet buildpack, I'm guessing it would be https://github.com/paketo-buildpacks/dotnet-core.

Following https://paketo.io/docs/howto/dotnet-core/, I tried with https://gitlab.wikimedia.org/toolforge-repos/milhistbot. After adding a property to specify target framework,

<PropertyGroup>
  <TargetFramework>net6.0</TargetFramework>
</PropertyGroup>

It seemed to start the process and build. I'm afraid my knowledge of how to build .net projects is limited, so I get an error running dotnet publish. (The repo has perl and several mono projects it seems?) Validation that this buildpack will or won't work would be useful. The buildpack supports .net 6 and .net 7, and utilizes msbuild.

@komla Sorry, I was away and did not see your comment until now. A build pack will not work for me they way you are trying. I have multiple C# projects in the one git repository. It was never intended to be built in this manner. It will simply give you an error telling you that it does not know what project to build, and there is no workaround for this.

There is NO WAY that I can do what @nskaggs is suggesting! I could easily get it to work here, but I do not have access to get it to build and run on your server.

Dotnet is shipped with Rocky 9, but is not available on the grid either. I can build here but get errors like:
./Liftwing: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./Liftwing)
./Liftwing: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./Liftwing)
./Liftwing: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./Liftwing)

This is very frustrating. My bots are unmaintainable because I have no build environment!

I can build a container here easily - I have a lot of experience with docker - but have no way to deploy it on your servers.

Using a dotnet buildpack is fairly simple. eg.:
$ pack build liftwing --buildpack paketo-buildpacks/dotnet-core --builder paketobuildpacks/builder:base --env BP_DOTNET_PROJECT_PATH=./Liftwing
$ docker run liftwing "Hanford Engineer Works" 2>/dev/nul
Setting ASPNETCORE_URLS=http://0.0.0.0:8080
FA

However, I have no idea how this could be achieved with toolforge. Pack build relies on me being in the correct directory of the checkout. (In this case mono/Liftwing/)

In T311466#9353995, @Hawkeye7 wrote:

I have multiple C# projects in the one git repository. It was never intended to be built in this manner.

These are reasonable statements of fact.

It will simply give you an error telling you that it does not know what project to build, and there is no workaround for this.

There is a workaround, and that is splitting your codebase into distinct gitlab repos which are designed to work in a buildpack system.

In T311466#9358026, @Hawkeye7 wrote:

Using a dotnet buildpack is fairly simple. eg.:
$ pack build liftwing --buildpack paketo-buildpacks/dotnet-core --builder paketobuildpacks/builder:base --env BP_DOTNET_PROJECT_PATH=./Liftwing
$ docker run liftwing "Hanford Engineer Works" 2>/dev/nul
Setting ASPNETCORE_URLS=http://0.0.0.0:8080
FA

However, I have no idea how this could be achieved with toolforge. Pack build relies on me being in the correct directory of the checkout. (In this case mono/Liftwing/)

This reads to me like splitting your "mono/Liftwing/" directory into a dedicated git repo would make the Toolforge supported system work for this part of your project. You can use https://toolsadmin.wikimedia.org/tools/id/milhistbot/repos/create to create as many distinct git repos for your milhistbot tool as are needed.

In T311466#9364202, @bd808 wrote:

This reads to me like splitting your "mono/Liftwing/" directory into a dedicated git repo would make the Toolforge supported system work for this part of your project. You can use https://toolsadmin.wikimedia.org/tools/id/milhistbot/repos/create to create as many distinct git repos for your milhistbot tool as are needed.

My naive assumption that we have a mono pack in the stack has been corrected by @Slst2020. Sorry for the confusion.

As you say, the biggest obstacle is that there is no dotnet buildpack available.

I don't know if any work has been done on this. Or indeed whether you are proceeding with the buildpack approach.

Liftwing is just a test program I created to test out the use of Liftwing instead of Ores. All it does is predict the rating of the article. But it works the same way as all my other C# tools. There is a shared library that handles dealing with Mediawiki (Wikimedia.dll), and a main program that uses it to perform useful functions related to the administration of featured articles on the English language Wikipedia.

I could, as you suggest, place each one in its own git repository with its own copy of Wikimedia.dll. I would have to upload a copy into each repository that requires it. That would work, so long as it was compatible with your environment (preferably identical). Otherwise I will get errors like:
bin/Debug/net7.0/Liftwing: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_2.33' not found (required by bin/Debug/net7.0/Liftwing)

In T311466#9365479, @Hawkeye7 wrote:

As you say, the biggest obstacle is that there is no dotnet buildpack available.

I don't know if any work has been done on this.

Not really, no. There is a community contributed https://elements.heroku.com/buildpacks/jincod/dotnetcore-buildpack that should work with Heroku stack we are currently using, but nobody has actually tested that as far as I am aware. We currently do not have a mechanism for a tool to add custom buildpacks to the default stack or to replace the stack entirely, so testing and deployment would need to happen from the WMCS side of things before you could do much more.

Or indeed whether you are proceeding with the buildpack approach.

We discussed the option of making a new mono base container using Bookworm and the https://www.mono-project.com/download/stable/#download-lin-debian packages on IRC today. We are not sure that is a generally helpful approach in the longer term as we really would like most projects to be able to convert to custom images via buildpack sooner rather than later. We plan to have a larger discussion tomorrow in the WMCS team meeting. Hopefully we can report back on which options we would like to try next to unblock you soon after that meeting.

When I try to use the admin console (https://toolsadmin.wikimedia.org/tools/id/milhistbot/repos/create) to create a new repository it says:
No GitLab accounts found for tool maintainers.

In T311466#9392107, @Hawkeye7 wrote:

When I try to use the admin console (https://toolsadmin.wikimedia.org/tools/id/milhistbot/repos/create) to create a new repository it says:
No GitLab accounts found for tool maintainers.

This is a bootstrapping problem that I haven't quite fixed -- T323767: Add potential next step for Toolforge error "No GitLab accounts found for tool maintainers.". The workaround is for you to go to https://gitlab.wikimedia.org/ and use the "Sign in" button in the upper right corner to create your initial account. You will also need to visit GitLab (Account Approval) after your initial GitLab login to get your Developer account approved to actually do useful work in GitLab.