Page MenuHomePhabricator

Migrate bldrwnsch from Toolforge GridEngine to Toolforge Kubernetes
Closed, ResolvedPublic


Kindly migrate your tool( from Toolforge GridEngine to Toolforge Kubernetes.

Toolforge GridEngine is getting deprecated.

Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.

If you would rather shut down this tool, kindly do so and mark this as resolved.

Useful Resources:
Migrating Jobs from GridEngine to Kubernetes
Migrating Web Services from GridEngine to Kubernetes

Event Timeline

My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: [infra] Decommission the Grid Engine infrastructure.

As noted in some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

Please see T308988 why bldrwnsch cannot be migrated at the moment.

simon04 changed the task status from Open to Stalled.Apr 17 2023, 7:09 PM

Please see T308988 why bldrwnsch cannot be migrated at the moment.

We are testing out buildpacks on Toolforge. Can you take a look to see if it can help you migrate using a custom image?
There's a short quickstart here.
Kindly take a look and let us know.

@simon04 and @Slst2020 and I we have been sitting together in the Athens Wikimedia Hackathon discussing how to potentially unblock this tool.

The base problem is that this tool requires a number of specific libraries, like:

These are C/C++ apps usually installed from debian packages (or compiled from source).

We see a couple of ways forward.

  1. Wait for buildpacks to support installing arbitrary debian packages
  2. Even though there is a python binding for gdal, it requires the base library to be installed (debian package). Another option to enable to python binding is to do a static build/compilation of the gdal suite (unknown how difficult is this) then install the python binding by using pip with the base python container image in toolforge (no buildpacks).

Actually the tippecanoe bit was already statically compiled in the tool home, so it doesn't really count as an external dependency.

gdal can be installed with this Heroku Buildpack, but it requires multiple buildpack support: