My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: [infra] Decommission the Grid Engine infrastructure.

As noted in https://techblog.wikimedia.org/2022/03/16/toolforge-gridengine-debian-10-buster-migration/ some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in https://techblog.wikimedia.org/2022/03/18/toolforge-jobs-framework/. We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

Hi, my tools here are awk and tcsh scripts mostly, that further invoke other unix tools like sort, uniq etc.. that also depend on a library I wrote called BotWikiAwk which has environment variable and other awk scripts.

I've actually migrated off Toolforge entirely. I still use rsync to upload output and data to the ~/www/static directory

Is it safe to assume *files* will not be deleted?

If I want to try the jobs framework described here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework

..which image should I use for awk and unix tools? Seems like they are all for Python, PHP etc..

How do I install unix tools that are not available in an image?

How would it work if given:

/data/project/botwikiawk/toolname/tool.awk

With environment variable:

AWKPATH=.:/data/project/botwikiawk/BotWikiAwk/lib

..where the awk libraries are located. It is shared by multiple tools, a common configuration file in the /lib directory.

My tools are migrated off Toolforge, nothing is currently running of consequence.

I might restore some to Toolforge, if I can figure this out.

Thank you

In T319607#9394449, @Green_Cardamom wrote:

Hi, my tools here are awk and tcsh scripts mostly, that further invoke other unix tools like sort, uniq etc.. that also depend on a library I wrote called BotWikiAwk which has environment variable and other awk scripts.

I've actually migrated off Toolforge entirely. I still use rsync to upload output and data to the ~/www/static directory

Is it safe to assume *files* will not be deleted?

They won't no, though we encourage to use them only for things that are not code.
For code, we recommend putting it in a public git repository (so it's ensured you are following the usage conditions: https://wikitech.wikimedia.org/wiki/Help:Toolforge#Constraints_of_Toolforge).

If I want to try the jobs framework described here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework

..which image should I use for awk and unix tools? Seems like they are all for Python, PHP etc..

How do I install unix tools that are not available in an image?

You can do something like https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Buildpack_static_tool (just created the page), without starting the webservice, and then use that image, you might need to install extra packages (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Installing_Apt_packages).

Then for each different script, you might want to add a new entry in the Procfile (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Job).

How would it work if given:

/data/project/botwikiawk/toolname/tool.awk

With environment variable:

AWKPATH=.:/data/project/botwikiawk/BotWikiAwk/lib

..where the awk libraries are located. It is shared by multiple tools, a common configuration file in the /lib directory.

You can add the environment variable using the envvar service: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars_Service

That'd be something like toolforge envvars create AWKPATH=.:$TOOL_DATA_DIR/BotWikiAwk/lib, note that this uses the TOOL_DATA_DIR envvar to avoid hardcodding the path (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Using_NFS_shared_storage).

Once you have a public git repository with the code, I can help you better to get started :)

HI @Green_Cardamom, where you able to try this again? Do you need any assistance?

This tool has been disabled from running on the Grid.

If you are the maintainer and you want this re-enabled so that you can work on migrating it off the grid, please reach out to the cloud admins on the mailing list(cloud-admin@lists.wikimedia.org)

However, please note that according to the timeline shared, on the 14th of March 2024, the grid infrastructure will be shut down.