My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: [infra] Decommission the Grid Engine infrastructure.

As noted in https://techblog.wikimedia.org/2022/03/16/toolforge-gridengine-debian-10-buster-migration/ some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in https://techblog.wikimedia.org/2022/03/18/toolforge-jobs-framework/. We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

I tried moving to K8's when they were first introduced. It went about as well as a cat in a room full of rocking chairs.

I heavily use python, and would rather not modify every script for a virtual environment. One of the major features that both the toolserver and its newer replacement labs has had is the ability to define and use a stable environment. I do not know much about k8's, but is it possible to configure a pod with all of the needed modules/programs and then just set that as a tool default and have everything use that?

In T319587#8300155, @Betacommand wrote:

I tried moving to K8's when they were first introduced. It went about as well as a cat in a room full of rocking chairs.

I heavily use python, and would rather not modify every script for a virtual environment. One of the major features that both the toolserver and its newer replacement labs has had is the ability to define and use a stable environment. I do not know much about k8's, but is it possible to configure a pod with all of the needed modules/programs and then just set that as a tool default and have everything use that?

If you use python, you can now build your own image by using a requirements.txt, that will generate the venv for you automatically.
You'll have to have your code in a public git repository, and place a requirements.txt at the root of the repository (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service, specifically https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Python).

Then you can use that with toolforge jobs (that run in k9s, https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Job), or as a webservice (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Webservice).

You can also install extra packages using an Aptfile (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Installing_Apt_packages).

Feel free to ask for help if you try and find issues/need some guidance.

1. I dont use Git, I use SVN
2. The code is hosted in a private repo to avoid leaking of credentials/non-public information.
3. None of the listed WSGI/Flask/FastAPI are the current CGI format that is used on labs ATM

In T319587#9464438, @Betacommand wrote:

1. I dont use Git, I use SVN

We don't support SVN for the build service :/
If staying on svn is a necessity for you, we can try working something up, though I recommend moving to git anyhow. If you need hosting for your repository, you can get one on gitlab.wikimedia.org (on https://toolsadmin.wikimedia.org/tools/id/betacommand-dev, under git repositories).

2. The code is hosted in a private repo to avoid leaking of credentials/non-public information.

The code should be publicly accessible and have an osi-approved license, according to the toolforge usage rules (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Right_to_fork_policy), if that's not the case, please remediate as soon as possible.

To avoid leaking credentials, you can use the envvars service to provide them as environment variables at run time (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars_Service)

3. None of the listed WSGI/Flask/FastAPI are the current CGI format that is used on labs ATM

You can use any command to startup you python application, just put that on the Procfile file under the web: entry, for example `web: python -m http.server --cgi "$PORT"' (untested, if you provide your source code I can try playing with it to help)

I do recommend though moving to wsgi or any other interface, but that might require bigger changes.

The code should be publicly accessible

Per RFC 2119

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

I have reviewed publishing my code and there are a number of valid reasons for not making it accessible to the world. I am not going to re-hash them.

I don't use git for a number of reasons, I personally find SVN preferable, again not here to debate programming preferences.

As a collection of cgi scripts, having a single webservice makes more since instead of forking them into unique projects, as it provides both flat file (report output) and basic form based tools. Ive kept the code base fairly simple to avoid having to deal with a lot of overhead. Apparently that is now being forced on labs users. The original benefit of using toolserver and later labs is the simplicity of setup and usage.

From Help:Toolforge/Rules:

2. All code in the Tools project must be published under an OSI approved open source license

Emphasis mine.

Ive reverted the unilateral change in policy about source code publication. Providing a OSI license with the code has met the toolserver and labs/cloud requirement for the last 18 years. There are reasons to not publicly publish code, if you want to change the policy this is not the correct way. And Im not going to debate it in a ticket either.

In T319587#9465459, @Betacommand wrote:

The code should be publicly accessible

Per RFC 2119

3. SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

I have reviewed publishing my code and there are a number of valid reasons for not making it accessible to the world. I am not going to re-hash them.

Can you provide a link to the previous discussion? I'll add a note on the tool info page for the next time.
If there was no discussion we should have one, there might be a way to make your code publishable and leave the private parts to a minimum.

I don't use git for a number of reasons, I personally find SVN preferable, again not here to debate programming preferences.

And that's ok, you are free to use whichever tools you want. The toolforge build service does not support SVN at the moment, so you'll have to figure out another solution for your tool or provide a strong-enough reason for us to spend time adding that support instead of working on other things that have priority. Patches are welcome too.

As a collection of cgi scripts, having a single webservice makes more since instead of forking them into unique projects, as it provides both flat file (report output) and basic form based tools. Ive kept the code base fairly simple to avoid having to deal with a lot of overhead. Apparently that is now being forced on labs users. The original benefit of using toolserver and later labs is the simplicity of setup and usage.

With the above entry-point you should be able to host your cgi scripts, so that should work similar to how it works now.
There's nothing being forced on you except the migration out of the grid (given that it's going to be shut down).
All I did is offer you technically possible solution to host your tool in the current infrastructure setup, if you decide that it does not work for you, feel free to figure out a different way, of course within the technical possibilities of the current platform (ex. requesting to keep the grid engine service running is not an option).

This tool has been disabled from running on the Grid.

If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, please reach out to the cloud admins on the mailing list(cloud-admin@lists.wikimedia.org)

However, please note that according to the timeline shared, on the 14th of March 2024, the grid infrastructure will be shut down.