Page MenuHomePhabricator

Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
Open, Stalled, Needs TriagePublic

Description

Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/dtcheck) from Toolforge GridEngine to Toolforge Kubernetes.

Toolforge GridEngine is getting deprecated.
See: https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/

Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.

If you would rather shut down this tool, kindly do so and mark this as resolved.

Useful Resources:
Migrating Jobs from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration
Migrating Web Services from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move_a_grid_engine_webservice
Python
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Rebuild_virtualenv_for_python_users

Related Objects

StatusSubtypeAssignedTask
Stalledmatmarex
Opendcaro
DeclinedNone
DeclinedNone
DeclinedBstorm
Resolveddcaro
Resolvedbd808
Resolvedbd808
Resolvedmmodell
Resolvedmmodell
Resolvedbd808
Resolved dpatrick
Resolvedbd808
Resolvedmmodell
Resolvedjcrespo
Resolvedbd808
Resolvedbd808
Resolvedbd808
Resolvedbd808
ResolvedBstorm
Resolvedfnegri
ResolvedBUG REPORTdcaro
ResolvedLegoktm
ResolvedLegoktm
Opendcaro
ResolvedLegoktm
OpenNone
OpenNone
OpenNone
Resolveddcaro
In Progressdcaro
Resolveddcaro
Resolveddcaro
OpenNone
Declineddduvall
OpenNone
OpenNone
DuplicateNone
Resolveddcaro
ResolvedRaymond_Ndibe
Resolveddcaro
ResolvedSlst2020
ResolvedSlst2020
DuplicateNone
DuplicateNone
OpenNone
DuplicateNone
Resolveddcaro
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
Resolveddcaro
Resolvedfnegri
ResolvedRaymond_Ndibe
ResolvedSlst2020
ResolvedSlst2020
ResolvedKHernandez-WMF
ResolvedRaymond_Ndibe
Resolvedkomla
ResolvedRaymond_Ndibe
ResolvedSlst2020
Resolveddcaro
Resolveddcaro
Resolveddcaro
ResolvedRaymond_Ndibe
Resolveddcaro
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedRaymond_Ndibe
OpenNone
OpenNone
ResolvedRaymond_Ndibe
OpenNone
OpenRaymond_Ndibe
ResolvedSlst2020
ResolvedSlst2020
ResolvedSlst2020
ResolvedSlst2020
ResolvedRaymond_Ndibe
ResolvedSlst2020
ResolvedSlst2020
ResolvedSlst2020
OpenRaymond_Ndibe
OpenNone
ResolvedRaymond_Ndibe
OpenRaymond_Ndibe
Resolveddcaro
ResolvedRaymond_Ndibe
OpenNone
OpenNone
OpenNone
ResolvedRaymond_Ndibe
Resolveddcaro
ResolvedAndrew
ResolvedSlst2020
Resolveddcaro
ResolvedRaymond_Ndibe
InvalidNone
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
Resolveddcaro
OpenNone
OpenNone
OpenNone
ResolvedRaymond_Ndibe
OpenNone
ResolvedRaymond_Ndibe
OpenNone
OpenNone
OpenNone
Openkomla
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenSlst2020
OpenRaymond_Ndibe
ResolvedSlst2020
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedSlst2020
OpenNone
OpenNone
Opendcaro
In Progressdcaro

Event Timeline

I am unable to migrate the tool, because there is no Kubernetes backend that allows both serving static HTML files using lighttpd and running CGI scripts using Ruby. I have talked about this problem previously with @bd808 on IRC.

For reference, I current use this command: https://github.com/MatmaRex/dtcheck/blob/master/webservice-start.sh

My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: Toolforge: Decomission the Grid Engine infrastructure.

As noted in https://techblog.wikimedia.org/2022/03/16/toolforge-gridengine-debian-10-buster-migration/ some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in https://techblog.wikimedia.org/2022/03/18/toolforge-jobs-framework/. We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

I reviewed this at the time of the Buster migration, and I can't migrate to Kuberenetes (without rewriting parts of the tool), for reasons in my previous comment.

Thanks for clarifying! I'm not sure if there is a ticket for this yet, or if this is a usecase expected to be resolved by buildpacks or something else. @bd808 can you confirm?

EDIT: Sorry, stale page cache, I didn't see the triage work confirming the tickets and need for buildpacks.