Page MenuHomePhabricator

Migrate dibot from Toolforge GridEngine to Toolforge Kubernetes
Closed, ResolvedPublic

Description

Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/dibot) from Toolforge GridEngine to Toolforge Kubernetes.

Toolforge GridEngine is getting deprecated.
See: https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/

Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.

If you would rather shut down this tool, kindly do so and mark this as resolved.

Useful Resources:
Migrating Jobs from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration
Migrating Web Services from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move_a_grid_engine_webservice
Python
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Rebuild_virtualenv_for_python_users

Event Timeline

My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: [infra] Decommission the Grid Engine infrastructure.

As noted in https://techblog.wikimedia.org/2022/03/16/toolforge-gridengine-debian-10-buster-migration/ some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in https://techblog.wikimedia.org/2022/03/18/toolforge-jobs-framework/. We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

This is a reminder that the tool for which this ticket is created is still running on the Grid.
The grid is deprecated and all remaining tools need to migrate to Toolforge Kubernetes.

We've sent several emails to maintainers as we continue to make the move away from the Grid.
Many of the issues that have held users back from moving away from the Grid have been addressed in
the latest updates to Build Service. See: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog

You might find the following resources helpful in migrating your tool:

  1. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Migrating_an_existing_tool
  2. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service#Tutorials_for_popular_languages

Don't hesitate to reach out to us using this ticket or via any of our support channels

If you have already migrated this tool, kindly mark this ticket as 'resolved'
To do this, click on the 'Add Action' dropdown above the comment text box, select 'Change Status', then 'Resolved'.
Click 'Submit'

Thank you!

I'm a ruwiki user MBH, like a Dmitry89, owner of this tool account. Both of us write our bots on C# and run them through mono, he also gave me source code of his bots year ago, so I could maintain them. He leaves Wikipedia in 2021, but I could became maintainer of his bots, I already transferred my bots to Kubernetes. Could you gave me access to this tool account, so I can configure his bots to run through Kubernetes? Ruwiki still needs these bots.
@komla @nskaggs

@MBH request to adopt this tool by filing a task using this link
Replace the TOOLNAME and LINKs as applicable.

MBH claimed this task.

I took control over dibot project, re-runned its webservice under k8s and add his bots into my jobs file. Since dibot uses php web tools (unlike me), there are should be no problems with them.

I was wrong: all of his php web tools doesn't work, looks like some refactoring are needed. @dcaro could you see?

Tool: https://dibot.toolforge.org/ (has links to all tools), tool folder: /project/dibot/public_html

There's some logs in the error logs like:

==> access.log <==
192.168.36.67 dibot.toolforge.org - [14/Mar/2024:18:06:58 +0000] "GET /inc.php HTTP/1.1" 200 5552 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0"

==> error.log <==
2024-03-14 18:06:58: mod_fastcgi.c.487) FastCGI-stderr:PHP Fatal error:  Uncaught Error: Call to undefined function mysql_connect() in /data/project/dibot/public_html/inc.php:11
2024-03-14 18:06:58: mod_fastcgi.c.487) FastCGI-stderr:Stack trace:
2024-03-14 18:06:58: mod_fastcgi.c.487) FastCGI-stderr:#0 {main}
2024-03-14 18:06:58: mod_fastcgi.c.487) FastCGI-stderr:  thrown in /data/project/dibot/public_html/inc.php on line 11

Does it have a public git repository? If not, can it be published to one? that makes it way simpler for me to do changes without breaking anything and makes it possible to use the build service.

That specific error is because it uses a deprecated function I think (https://www.php.net/manual/en/function.mysql-connect.php)

I will publish them after my working day, but you can edit this php files directly in the Toolforge folder, they aren't binaries like my tools.

I added all of them to https://github.com/Saisengen/wikibots/tree/main/php-tools. If it's needed to add html/css/js files from Dmitry's folder - say me or add yourself by pull.

I added all of them to https://github.com/Saisengen/wikibots/tree/main/php-tools. If it's needed to add html/css/js files from Dmitry's folder - say me or add yourself by pull.

I would strongly recommend creating a new repository, otherwise all the build process will have to be shared (essentially, for each lang, a different repo, at least).

Is there a way to move these files to the new repo, or I should re-create all of these files manually again?

Is there a way to move these files to the new repo, or I should re-create all of these files manually again?

You'll need to copy them over, I can do it if you have an empty repo to move them to that I can fork.

Thanks, I merged it. Now could I delete https://github.com/Saisengen/wikibots/tree/main/php-tools ? Will you construct a build image, like on my tools? Should I run some building process now?

Thanks, I merged it. Now could I delete https://github.com/Saisengen/wikibots/tree/main/php-tools ?

Yes, you can use the new repository instead.

Will you construct a build image, like on my tools? Should I run some building process now?

You can run the build process as usual dibot.tools@tools-bastion-12$ toolforge build start https://github.com/Saisengen/dmitry89-tools, then toolforge webservice buildservice restart.

Note that I only fixed one script and the index.php as an example for you to change the rest of the scripts.