Page MenuHomePhabricator
Create Task
Maniphest T319812

Migrate inkobot from Toolforge GridEngine to Toolforge Kubernetes
Closed, ResolvedPublic
Actions

Description

Kindly migrate your tool(https://grid-deprecation.toolforge.org/t/inkobot) from Toolforge GridEngine to Toolforge Kubernetes.

Toolforge GridEngine is getting deprecated.
See: https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/

Please note that a volunteer may perform this migration if this has not been done after some time.
If you have already migrated this tool, kindly mark this as resolved.

If you would rather shut down this tool, kindly do so and mark this as resolved.

Useful Resources:
Migrating Jobs from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Grid_Engine_migration
Migrating Web Services from GridEngine to Kubernetes
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Move_a_grid_engine_webservice
Python
https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Rebuild_virtualenv_for_python_users

Related Objects

Event Timeline

komla created this task.Oct 6 2022, 8:02 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 6 2022, 8:02 PM
• nskaggs subscribed.Oct 7 2022, 12:56 AM

My apologies if this ticket comes as a surprise to you. In order to ensure WMCS can provide a stable, secure and supported platform, it’s important we migrate away from GridEngine. I want to assure you that while it is WMCS’s intention to shutdown GridEngine as outlined in the blog post https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/, a shutdown date for GridEngine has not yet been set. The goal of the migration is to migrate as many tools as possible onto kubernetes and ensure as smooth a transition as possible for everyone. Once the majority of tools have migrated, discussion on a shutdown date is more appropriate. See T314664: [infra] Decommission the Grid Engine infrastructure.

As noted in https://techblog.wikimedia.org/2022/03/16/toolforge-gridengine-debian-10-buster-migration/ some use cases are already supported by kubernetes and should be migrated. If your tool can migrate, please do plan a migration. Reach out if you need help or find you are blocked by missing features. Most of all, WMCS is here to support you.

However, it’s possible your tool needs a mixed runtime environment or some other features that aren't yet present in https://techblog.wikimedia.org/2022/03/18/toolforge-jobs-framework/. We’d love to hear of this or any other blocking issues so we can work with you once a migration path is ready. Thanks for your hard work as volunteers and help in this migration!

Inkowik reassigned this task from Inkowik to hgzh.Mar 20 2023, 9:54 PM
Inkowik subscribed.
Inkowik unsubscribed.
hgzh added a comment.Mar 31 2023, 10:20 AM

jobs are all moved to kubernetes, however it seems there are 16 running jobs on GridEngine left that are stuck, don't know if and how I can abort them.

Stashbot added a comment.Mar 31 2023, 10:36 AM

Mentioned in SAL (#wikimedia-cloud) [2023-03-31T10:36:03Z] <arturo> cleanup grid jobs (T319812)

• aborrero subscribed.Mar 31 2023, 10:38 AM

I did this!

tools.inkobot@tools-sgebastion-11:~$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
1284210 0.25750 fcc        tools.inkobo r     03/26/2023 18:00:14 task@tools-sgeexec-10-21.tools     1        
1285515 0.25743 fcc        tools.inkobo r     03/26/2023 19:00:14 task@tools-sgeexec-10-16.tools     1        
1286814 0.25736 fcc        tools.inkobo r     03/26/2023 20:00:14 task@tools-sgeexec-10-21.tools     1        
1288180 0.25730 fcc        tools.inkobo r     03/26/2023 21:00:14 task@tools-sgeexec-10-16.tools     1        
1289522 0.25723 fcc        tools.inkobo r     03/26/2023 22:00:14 task@tools-sgeexec-10-16.tools     1        
1290814 0.25716 fcc        tools.inkobo r     03/26/2023 23:00:14 task@tools-sgeexec-10-14.tools     1        
1292237 0.25710 fcc        tools.inkobo r     03/27/2023 00:00:14 task@tools-sgeexec-10-20.tools     1        
1295284 0.25696 fcc        tools.inkobo r     03/27/2023 02:00:14 task@tools-sgeexec-10-20.tools     1        
1296666 0.25690 fcc        tools.inkobo r     03/27/2023 03:00:14 task@tools-sgeexec-10-17.tools     1        
1298077 0.25683 fcc        tools.inkobo r     03/27/2023 04:00:14 task@tools-sgeexec-10-19.tools     1        
1299497 0.25676 fcc        tools.inkobo r     03/27/2023 05:00:14 task@tools-sgeexec-10-9.tools.     1        
1300965 0.25670 fcc        tools.inkobo r     03/27/2023 06:00:14 task@tools-sgeexec-10-20.tools     1        
1302336 0.25663 fcc        tools.inkobo r     03/27/2023 07:00:14 task@tools-sgeexec-10-19.tools     1        
1303731 0.25656 fcc        tools.inkobo r     03/27/2023 08:00:14 task@tools-sgeexec-10-16.tools     1        
1305117 0.25650 fcc        tools.inkobo r     03/27/2023 09:00:14 task@tools-sgeexec-10-9.tools.     1        
1306545 0.25643 fcc        tools.inkobo r     03/29/2023 12:32:44 task@tools-sgeexec-10-7.tools.     1        
1307964 0.00000 fcc        tools.inkobo qw    03/27/2023 11:00:09                                    1        
1309346 0.00000 fcc        tools.inkobo qw    03/27/2023 12:00:10                                    1        
1310891 0.00000 fcc        tools.inkobo qw    03/27/2023 13:00:08                                    1        
1312255 0.00000 fcc        tools.inkobo qw    03/27/2023 14:00:08                                    1        
1313662 0.00000 fcc        tools.inkobo qw    03/27/2023 15:00:11                                    1        
1315075 0.00000 fcc        tools.inkobo qw    03/27/2023 16:00:10                                    1        
1316447 0.00000 fcc        tools.inkobo qw    03/27/2023 17:00:09                                    1        
1317819 0.00000 fcc        tools.inkobo qw    03/27/2023 18:00:09                                    1        
1319203 0.00000 fcc        tools.inkobo qw    03/27/2023 19:00:07                                    1        
1320558 0.00000 fcc        tools.inkobo qw    03/27/2023 20:00:08                                    1        
1321996 0.00000 fcc        tools.inkobo qw    03/27/2023 21:00:08                                    1        
1323391 0.00000 fcc        tools.inkobo qw    03/27/2023 22:00:07                                    1        
1324817 0.00000 fcc        tools.inkobo qw    03/27/2023 23:00:11                                    1        
1326297 0.00000 fcc        tools.inkobo qw    03/28/2023 00:00:14                                    1        
1327824 0.00000 fcc        tools.inkobo qw    03/28/2023 01:00:09                                    1        
1329155 0.00000 fcc        tools.inkobo qw    03/28/2023 02:00:08                                    1        
1330620 0.00000 fcc        tools.inkobo qw    03/28/2023 03:00:10                                    1        
1332051 0.00000 fcc        tools.inkobo qw    03/28/2023 04:00:09                                    1        
1333404 0.00000 fcc        tools.inkobo qw    03/28/2023 05:00:09                                    1        
1334083 0.00000 nld        tools.inkobo qw    03/28/2023 05:25:05                                    1        
1334291 0.00000 bvksk      tools.inkobo qw    03/28/2023 05:35:05                                    1        
1334780 0.00000 fcc        tools.inkobo qw    03/28/2023 06:00:09                                    1        
1336172 0.00000 fcc        tools.inkobo qw    03/28/2023 07:00:08                                    1        
1337501 0.00000 fcc        tools.inkobo qw    03/28/2023 08:00:10                                    1        
1338832 0.00000 fcc        tools.inkobo qw    03/28/2023 09:00:09                                    1        
1340195 0.00000 fcc        tools.inkobo qw    03/28/2023 10:00:08                                    1        
1341575 0.00000 fcc        tools.inkobo qw    03/28/2023 11:00:09                                    1        
1342854 0.00000 fcc        tools.inkobo qw    03/28/2023 12:00:09                                    1        
1344331 0.00000 fcc        tools.inkobo qw    03/28/2023 13:00:09                                    1        
1345619 0.00000 fcc        tools.inkobo qw    03/28/2023 14:00:07                                    1        
1346953 0.00000 fcc        tools.inkobo qw    03/28/2023 15:00:07                                    1        
1348354 0.00000 fcc        tools.inkobo qw    03/28/2023 16:00:08                                    1        
1349694 0.00000 fcc        tools.inkobo qw    03/28/2023 17:00:08                                    1        
1377826 0.00000 fcc        tools.inkobo qw    03/29/2023 13:00:08                                    1        
tools.inkobot@tools-sgebastion-11:~$ for i in $(qstat | grep ^1 | awk -F' ' '{print $1}') ; do qdel $i ; done
tools.inkobot has registered the job 1284210 for deletion
tools.inkobot has registered the job 1285515 for deletion
tools.inkobot has registered the job 1286814 for deletion
tools.inkobot has registered the job 1288180 for deletion
tools.inkobot has registered the job 1289522 for deletion
tools.inkobot has registered the job 1290814 for deletion
tools.inkobot has registered the job 1292237 for deletion
tools.inkobot has registered the job 1295284 for deletion
tools.inkobot has registered the job 1296666 for deletion
tools.inkobot has registered the job 1298077 for deletion
tools.inkobot has registered the job 1299497 for deletion
tools.inkobot has registered the job 1300965 for deletion
tools.inkobot has registered the job 1302336 for deletion
tools.inkobot has registered the job 1303731 for deletion
tools.inkobot has registered the job 1305117 for deletion
tools.inkobot has registered the job 1306545 for deletion
tools.inkobot has deleted job 1307964
tools.inkobot has deleted job 1309346
tools.inkobot has deleted job 1310891
tools.inkobot has deleted job 1312255
tools.inkobot has deleted job 1313662
tools.inkobot has deleted job 1315075
tools.inkobot has deleted job 1316447
tools.inkobot has deleted job 1317819
tools.inkobot has deleted job 1319203
tools.inkobot has deleted job 1320558
tools.inkobot has deleted job 1321996
tools.inkobot has deleted job 1323391
tools.inkobot has deleted job 1324817
tools.inkobot has deleted job 1326297
tools.inkobot has deleted job 1327824
tools.inkobot has deleted job 1329155
tools.inkobot has deleted job 1330620
tools.inkobot has deleted job 1332051
tools.inkobot has deleted job 1333404
tools.inkobot has deleted job 1334083
tools.inkobot has deleted job 1334291
tools.inkobot has deleted job 1334780
tools.inkobot has deleted job 1336172
tools.inkobot has deleted job 1337501
tools.inkobot has deleted job 1338832
tools.inkobot has deleted job 1340195
tools.inkobot has deleted job 1341575
tools.inkobot has deleted job 1342854
tools.inkobot has deleted job 1344331
tools.inkobot has deleted job 1345619
tools.inkobot has deleted job 1346953
tools.inkobot has deleted job 1348354
tools.inkobot@tools-sgebastion-11:~$ qstat
[..nothing..]
hgzh closed this task as Resolved.Mar 31 2023, 2:19 PM

Thanks! Task should be resolved then.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits