Page MenuHomePhabricator

Migrate wikihistory from Toolforge GridEngine to Toolforge Kubernetes
Closed, DeclinedPublic

Description

Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/wikihistory) from Toolforge GridEngine to Toolforge Kubernetes.

Toolforge GridEngine is getting deprecated.
See: https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/

Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.

If you would rather shut down this tool, kindly do so and mark this as resolved.

Useful Resources:
Migrating Jobs from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration
Migrating Web Services from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move_a_grid_engine_webservice
Python
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Rebuild_virtualenv_for_python_users

Event Timeline

My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: [infra] Decommission the Grid Engine infrastructure.

As noted in https://techblog.wikimedia.org/2022/03/16/toolforge-gridengine-debian-10-buster-migration/ some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in https://techblog.wikimedia.org/2022/03/18/toolforge-jobs-framework/. We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

I still need a container with mono and php7.x combined.

This is a reminder that the tool for which this ticket is created is still running on the Grid.
The grid is deprecated and all remaining tools need to migrate to Toolforge Kubernetes.

We've sent several emails to maintainers as we continue to make the move away from the Grid.
Many of the issues that have held users back from moving away from the Grid have been addressed in
the latest updates to Build Service. See: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog

You might find the following resources helpful in migrating your tool:

  1. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Migrating_an_existing_tool
  2. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Tutorials_for_popular_languages

Don't hesitate to reach out to us using this ticket or via any of our support channels

If you have already migrated this tool, kindly mark this ticket as 'resolved'

Thank you!

Wikihistory uses php *and* mono in the same script. Is there a cookbook (for dummies) explaining step by step what to do?

I unfortunately, forgot about this tool entirely. Would somebody be willing to take this tool over?

@Cyberpower678 no problem. No additional maintainer is needed, at least for now.

@Wurgl Can you use the system package for mono? If so, try building now with the php buildpack and add mono using apt. See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Installing_Apt_packages.

If not, I would track T311466 and T352774 for more options.

@nskaggs Okay. I have already read that magic "Aptfile".

Question 1: Which of the packages here do I need? Please note: I have almost zero experiance with mono, my Linux@Home is mono-clean. I use mono, because the original author implemented the tool in mono.
https://packages.ubuntu.com/jammy/cli-mono/

Question 2: In the persondata-Tool I start my jobs with option --image php7.4 or --image php8.2 Do I simple use one of these options too? Or do I need to add php-package to that magic "Aptfile" and start without any --image-Option?

I tried the following:
Create an Aptfile like

php

and tried to start with
toolforge jobs run --command 'php dewiki_update_new.php 1' --image mono6.8 -o ~/logs/upd1.out -e ~/logs/upd1.err --continuous job1

The only lines I saw in ~/logs/upd1.err

/bin/sh: 1: php: not found
/bin/sh: 1: php: not found
/bin/sh: 1: php: not found

I miss some action between? The pure existence of a Aptfile seems to cause nothing? What am I doing wrong? What is missing?

@Wurgl it's hard for me to know exactly what you are seeing, but you have the basic idea correct. Pick a buildpack for one of the languages you use, and use aptfile to add system libraries for the other language. I was suggesting using a php buildpack, and then installing mono via apt. However, given mono likely has some compilation and build steps needed, a mono buildpack + php via apt might make more sense. If you'd like to try using a mono buildpack, and install php via apt, T352774: [builds-builder] Investigate how to enable mono/dotnet/c# and implement the best one to unblock us to migrate tools is being finalized now which will add a mono buildpack.

Can you link to your source code? That will make it much easier to give specific advice.

Question 2: In the persondata-Tool I start my jobs with option --image php7.4 or --image php8.2 Do I simple use one of these options too? Or do I need to add php-package to that magic "Aptfile" and start without any --image-Option?

Note that we are talking about images built with the buildservice (see the toolforge build commands here https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Buildpack_PHP_tool).
And then https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Job to run a job with your image.

Sorry, I am too dumb to understand what I should do. Source for the C#-Program of Wikihistory can be found at https://persondata.toolforge.org/WikiHistory_src.zip the php-part is a wrapper using the database, and redis. I chmod a+rx wikihistory, so it is public readable. The php wrapper is: dewiki_update_new.php

My personal experiance with C# starts and ends with this source, the original author is not active any more and the only thing I have done is massive performance improvement. Me and C# did never get friends (and will never).

Is there a proper source code repository (Git etc), instead of some code-at-some-point-in-time-snapshot zip file?

Hi @Wurgl, I see there's still some processes running on the grid, were you able to have a look into migrating? If not, can you make your code available on any public source code repository (gitlab.wikimedia.org offers free repositories, you can create them from https://toolsadmin.wikimedia.org/tools/id/wikihistory on the left pane)? that will help me greatly on helping you migrate so I can send example merge requests and similar.

Thanks!

The grid engine has been shut down, so I'm closing any remaining migration tasks as Declined. If you're still planning to migrate this tool, please re-open this task and add one or more active project tags to it. (If you need a project tag for your tool, those can be created via the Toolforge admin console.)