Page MenuHomePhabricator

Migrate tools from Grid Engine to Kubernetes
Closed, ResolvedPublic8 Estimated Story Points

Description

Migrate tools from Grid Engine to Kubernetes. Some of these are not exactly ours and others are not in use anymore so this could also be a good time to do some pruning and remove ourselves as maintainers if we want to or delete what's not being used.

Tools

  • COMMTECH-COMMONS – Not moved to Kubernetes due to T254636
  • DEADLINKS Removed as a maintainer
  • DATABASE-REPORTS
  • ERANBOT
  • GRANTMETRICS
  • IA-UPLOAD
  • JARRY-COMMON
  • METAVIEWS
  • MUSIKBOT
  • OCR-TEST
  • PHETOOLS Removed as a maintainer
  • POPULARPAGES
  • SVGTRANSLATE
  • SVGTRANSLATE-TEST
  • WSEXPORT
  • WSEXPORT-TEST
  • XTOOLS

Acceptance Criteria

  • Remove tools that are not in use
  • Remove ourselves as maintainers where needed
  • Move remaining tools from Grid Engine to Kubernetes

More Info
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move_a_grid_engine_webservice
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration
https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/About_Toolforge#Communication_and_support

Event Timeline

Tools we should probably be removed from:

  • deadlinks – This is a visualization of dead links that are fixed with InternetArchiveBot. InternetArchive is actively using this. Community Tech actually did author this tool as part of the #1 wish of the 2015 survey, but InternetArchiveBot itself is not maintained by us. Similarly we recently removed ourselves as the maintainer of the DeadlinkChecker library, so I think we should step away from the visualization tool as well. Pinging @Cyberpower678 – would you mind taking ownership of deadlinks.toolforge.org? If so, I can ask Niharika to add you, and we'll remove us. I don't think there will be much if any maintenance moving forward, at least not until some major PHP upgrade. It's running fine now on PHP 7.4. Note I already migrated the tool to Kubernetes, so no action is needed there.
  • jarry-common – I believe we were added because the SVG Check tool was once used by SVG Translate Tool, or something like that. Today I don't think we have any involvement with this suite of tools. Pinging @Samwilson to verify. Jarry is a bit inactive these days, so if we want tools.svgtranslate to be removed, we might need to ask Cloud Services to do it for us. I did go ahead and migrate the tool to Kubernetes, anyway.
  • Phetools – WS Export used to use this for fetching attribution data, but it doesn't any longer. I believe we don't need to be listed as a maintainer anymore. I did not migrate this tool as there are several webservices and a rather complicated Lighttpd config, so I wasn't sure simply switch to k8s would work or if something would break.

I agree that we should be removed as maintainers from all those. Access to jarry-common was I think because the old svgtranslate code used something from there.

How do we formally remove ourselves as maintainers? Agree with @MusikAnimal and @Samwilson cc @dmaza @KSiebert

Tools we should probably be removed from:

  • deadlinks – This is a visualization of dead links that are fixed with InternetArchiveBot. InternetArchive is actively using this. Community Tech actually did author this tool as part of the #1 wish of the 2015 survey, but InternetArchiveBot itself is not maintained by us. Similarly we recently removed ourselves as the maintainer of the DeadlinkChecker library, so I think we should step away from the visualization tool as well. Pinging @Cyberpower678 – would you mind taking ownership of deadlinks.toolforge.org? If so, I can ask Niharika to add you, and we'll remove us. I don't think there will be much if any maintenance moving forward, at least not until some major PHP upgrade. It's running fine now on PHP 7.4. Note I already migrated the tool to Kubernetes, so no action is needed there.
  • jarry-common – I believe we were added because the SVG Check tool was once used by SVG Translate Tool, or something like that. Today I don't think we have any involvement with this suite of tools. Pinging @Samwilson to verify. Jarry is a bit inactive these days, so if we want tools.svgtranslate to be removed, we might need to ask Cloud Services to do it for us. I did go ahead and migrate the tool to Kubernetes, anyway.
  • Phetools – WS Export used to use this for fetching attribution data, but it doesn't any longer. I believe we don't need to be listed as a maintainer anymore. I did not migrate this tool as there are several webservices and a rather complicated Lighttpd config, so I wasn't sure simply switch to k8s would work or if something would break.

Sorry for the delayed response. I was practically disconnected from the web this past month. I don't mind taking ownership of the tool. We might end up decommissioning it actually since we have much more advanced statistics built up and stored now, but it's still being developed.

How do we formally remove ourselves as maintainers?

We need to reach out to individual maintainers, since we can't remove group accounts through the Toolforge admin console (there's probably a Phab task to fix this somewhere). I've already contacted Niharika about deadlinks, and will get in touch with others about the other two tools.

I don't mind taking ownership of the tool. We might end up decommissioning it actually since we have much more advanced statistics built up and stored now, but it's still being developed.

Thank you! You have now been added as a maintainer, and tools.community-tech-tools has been removed https://toolsadmin.wikimedia.org/tools/id/deadlinks

JMcLeod_WMF changed the point value for this task from 5 to 8.May 4 2022, 2:45 PM

This is finally done! I just double-checked and no remaining tools (except for unrelated jobs for erabot, see T306888#7931423) have webservices or cron jobs on the grid that aren't at least running on Buster. I don't think this needs QA; the fact the tools are up and running is enough. I'll just go ahead and mark as resolved.