Page MenuHomePhabricator

Migrate parliamentdiagram from Toolforge GridEngine to Toolforge Kubernetes
Closed, DeclinedPublic

Description

Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/parliamentdiagram) from Toolforge GridEngine to Toolforge Kubernetes.

Toolforge GridEngine is getting deprecated.
See: https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/

Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.

If you would rather shut down this tool, kindly do so and mark this as resolved.

Useful Resources:
Migrating Jobs from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration
Migrating Web Services from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move_a_grid_engine_webservice
Python
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Rebuild_virtualenv_for_python_users

Event Timeline

My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: [infra] Decommission the Grid Engine infrastructure.

As noted in https://techblog.wikimedia.org/2022/03/16/toolforge-gridengine-debian-10-buster-migration/ some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in https://techblog.wikimedia.org/2022/03/18/toolforge-jobs-framework/. We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

taavi subscribed.

The grid engine has been shut down, so I'm closing any remaining migration tasks as Declined. If you're still planning to migrate this tool, please re-open this task and add one or more active project tags to it. (If you need a project tag for your tool, those can be created via the Toolforge admin console.)

Hi, I'm associated with Slashme for maintaining ParliamentDiagram.
I understand the tool has been disabled, but I don't quite understand the context and reasons about it, or what I need to do in order to migrate (or where to migrate to).
What's most surprising to me, is that a sort of copy of the parliamentdiagram tool, called parliamentarch, is still standing and working quite well, despite the fact that I didn't do anything to maintain it throug that migration thingy.
Edit: actually it doesn't seem to work either, but it's nonetheless considerably less broken.

Could someone explain in basic terms what needs to be done to put our tool back on its feet ?

Hi @Gouvernathor the parliamentdiagram tool was running on the old Grid Engine platform, which was decommissioned last month and replaced by the newer Kubernetes platform. You need to re-deploy your tool to the new platform by following the documentation at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web.

It looks like someone already tried to re-deploy the tool, as there is already a webservice running in the Kubernetes platform, but is not working correctly: https://parliamentdiagram.toolforge.org/ returns "403 Forbidden".

I see that the parliamentdiagram tool also included a scheduled job, you will need to recreate it in the Kubernetes platform, following the docs at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration.

If you need further help, you can reply to this task or use one of the other Communication channels.

Thanks. I restarted the webservice and it seems to work better (no 403, and the subpages have the js working properly, and the php subpages are accessible).
There is an error with a place where we call a python file from the js, and the result doesn't seem to be what it was before - but I'm not sure how it did work, either.
The thing is that we're making a POST ajax request to the python file. Before, it executed the file and returned what the file printed (the name of a generated SVG file). Now, it returns the entire file...
I guess I'll figure it out.

I did figure it out by telling .lighttpd.conf to use python3 to handle the .py requests.

Hi, I'm having an issue porting a cron task we used to the new system.
The task needs to remove old files, but there's a permission error and it doesn't have the right to rm any of them.
I scheduled the tool using the tool account.
What can I do, where should I start ?

@Gouvernathor can you give me an example of a file that the task is failing to remove? I'll have a look at the permissions.

Nvm. They were the files in public_html/svgfiles, but the problem came from the fact that an absolute path to the main tool was used, and called from the beta tool, which resulted in obvious permission errors.
Using relative paths solved the issue.
Thanks for the response.