Page MenuHomePhabricator
Create Task
Maniphest T341066

[jobs-api,jobs-cli] Support multiple replicas of continuous jobs
Closed, ResolvedPublicFeature
Actions

Description

Feature summary (what you would like to be able to do and where):
Allow configuring a continuous job to have multiple replicas.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):
Sometimes there's a need to run multiple instances of the exact same thing, for example for multiple runner processes.

Benefits (why should this be implemented?):
Reduces duplicate work.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
d/changelog: bump to 16.1.3repos/cloud/toolforge/jobs-cli!69raymond-ndibebump_versionmain
d/changelog: bump to 16.1.2repos/cloud/toolforge/jobs-cli!68raymond-ndibebump_versionmain
jobs-api: bump to 0.0.336-20240919135748-c8ffa589repos/cloud/toolforge/toolforge-deploy!523ghostbump_jobs-apimain
[toolforge-deploy] test multi-replica support for continuous jobsrepos/cloud/toolforge/toolforge-deploy!521raymond-ndibetest_multi_replica_support_for_continuous_jobmain
[jobs-cli] remove unknown keys from dumprepos/cloud/toolforge/jobs-cli!64raymond-ndiberemove_unknown_keys_in_dumpmain
[jobs-cli] multi-replica support for continuous jobsrepos/cloud/toolforge/jobs-cli!63raymond-ndibemulti-replica-cont-jobmain
[jobs-api] multi-replica support for continuous jobsrepos/cloud/toolforge/jobs-api!115raymond-ndibemulti-replica-cont-jobmain
Draft: [maintain-kubeusers] increment default quota for pods, cpu, memrepos/cloud/toolforge/maintain-kubeusers!58raymond-ndibeincrement_default_quota_pod_cpu_memmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

taavi created this task.Jul 4 2023, 1:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 4 2023, 1:13 PM
taavi added a parent task: T348755: [jobs-api,webservice] Run webservices via the jobs framework.Oct 12 2023, 2:20 PM
Tacsipacsi subscribed.Jan 20 2024, 2:02 PM

I committed https://github.com/toollabs/Rotatebot/commit/8938d15165acc2c8cd4689da48a61dbeb84b1d80 on the assumption that this won’t happen. Others may have done the same, so please make it opt-in if it happens.

dcaro subscribed.Mar 11 2024, 12:01 PM
In T341066#9474531, @Tacsipacsi wrote:

I committed https://github.com/toollabs/Rotatebot/commit/8938d15165acc2c8cd4689da48a61dbeb84b1d80 on the assumption that this won’t happen. Others may have done the same, so please make it opt-in if it happens.

This task is only for continuous jobs, scheduled jobs will continue having the same parallelism limitations (just 1 essentially), so don't be worried :)

dcaro renamed this task from Support multiple replicas of continuous jobs to [jobs-api,jobs-cli] Support multiple replicas of continuous jobs.Mar 11 2024, 12:02 PM
dcaro triaged this task as Medium priority.
dcaro raised the priority of this task from Medium to High.
dcaro edited projects, added Toolforge; removed Toolforge Jobs framework.
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
Raymond_Ndibe subscribed.Apr 4 2024, 10:25 PM

how will this affect the current 3 continuous jobs limit? does 2 replicas of a continuous job count as 1 or 2 when considering limits?

Raymond_Ndibe added a project: User-Raymond_Ndibe.Apr 4 2024, 10:25 PM
dcaro added a comment.Apr 5 2024, 2:27 PM
In T341066#9691067, @Raymond_Ndibe wrote:

how will this affect the current 3 continuous jobs limit? does 2 replicas of a continuous job count as 1 or 2 when considering limits?

I would consider it as one, that uses 2x the amount of resources.

Raymond_Ndibe added a comment.May 22 2024, 12:17 PM
In T341066#9692931, @dcaro wrote:
In T341066#9691067, @Raymond_Ndibe wrote:

how will this affect the current 3 continuous jobs limit? does 2 replicas of a continuous job count as 1 or 2 when considering limits?

I would consider it as one, that uses 2x the amount of resources.

mmm, yeaaa makes sense. On the kubernetes level this will just be a deployment with 2 or more pods I guess

Raymond_Ndibe claimed this task.Jul 11 2024, 11:47 AM
Raymond_Ndibe moved this task from Ready to be worked on to Toolforge iteration 14 on the Toolforge board.Aug 7 2024, 5:10 AM
Raymond_Ndibe edited projects, added Toolforge (Toolforge iteration 14); removed Toolforge.
CodeReviewBot added a project: Patch-For-Review.Aug 7 2024, 11:23 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/58

Draft: [maintain-kubeusers] increment default quota for pods, cpu, mem

CodeReviewBot added a comment.Aug 7 2024, 11:24 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/115

Draft: [jobs-api] multi-replica support for continuous jobs

Raymond_Ndibe mentioned this in T372025: [jobs-api] prepend date and pod name to filelog lines .Aug 8 2024, 12:12 AM
CodeReviewBot added a comment.Aug 12 2024, 12:06 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/63

Draft: [jobs-cli] multi-replica support for continuous jobs

CodeReviewBot added a comment.Aug 12 2024, 1:12 PM

raymond-ndibe closed https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/58

Draft: [maintain-kubeusers] increment default quota for pods, cpu, mem

Raymond_Ndibe changed the task status from Open to In Progress.Aug 12 2024, 5:26 PM
Raymond_Ndibe moved this task from Next Up to In Review on the Toolforge (Toolforge iteration 14) board.
CodeReviewBot added a comment.Aug 17 2024, 11:32 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/64

[jobs-cli] remove unknown keys from dump

dcaro moved this task from In Review to In Progress on the Toolforge (Toolforge iteration 14) board.Sep 3 2024, 8:07 AM
Raymond_Ndibe moved this task from In Progress to In Review on the Toolforge (Toolforge iteration 14) board.Sep 3 2024, 8:46 PM
CodeReviewBot added a comment.Sep 12 2024, 3:52 AM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/521

[toolforge-deploy] test multi-replica support for continuous jobs

CodeReviewBot added a comment.Sep 19 2024, 1:51 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/64

[jobs-cli] remove unknown keys from dump

CodeReviewBot added a comment.Sep 19 2024, 1:57 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/115

[jobs-api] multi-replica support for continuous jobs

CodeReviewBot added a comment.Sep 19 2024, 2:00 PM

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/523

jobs-api: bump to 0.0.336-20240919135748-c8ffa589

Stashbot added a comment.Sep 19 2024, 2:05 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T14:05:42Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 2:10 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T14:10:00Z] <wmbot~raymondndibe@wmf3402> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api (T341066)

CodeReviewBot added a comment.Sep 19 2024, 2:31 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/68

d/changelog: bump to 16.1.2

CodeReviewBot added a comment.Sep 19 2024, 2:32 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/68

d/changelog: bump to 16.1.2

Stashbot added a comment.Sep 19 2024, 2:42 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T14:42:50Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 2:47 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T14:47:36Z] <wmbot~raymondndibe@wmf3402> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 2:50 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T14:50:41Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 2:56 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T14:55:59Z] <wmbot~raymondndibe@wmf3402> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 2:57 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T14:57:08Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 3:01 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T15:01:22Z] <wmbot~raymondndibe@wmf3402> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 3:07 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T15:07:32Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 3:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T15:08:40Z] <wmbot~raymondndibe@wmf3402> END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 3:18 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T15:18:16Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 3:19 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T15:19:34Z] <wmbot~raymondndibe@wmf3402> END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 3:27 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T15:27:11Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 3:28 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T15:28:20Z] <wmbot~raymondndibe@wmf3402> END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 4:26 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T16:26:46Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 4:38 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T16:38:04Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 4:46 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T16:46:34Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 4:48 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T16:48:20Z] <wmbot~raymondndibe@wmf3402> END (ERROR) - Cookbook wmcs.toolforge.component.deploy (exit_code=97) for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 5:06 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:06:54Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T341066)

Stashbot added a comment.Sep 19 2024, 5:13 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:13:20Z] <wmbot~raymondndibe@wmf3402> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T341066)

CodeReviewBot added a comment.Sep 19 2024, 5:15 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/523

jobs-api: bump to 0.0.336-20240919135748-c8ffa589

CodeReviewBot added a comment.Sep 19 2024, 5:17 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/63

[jobs-cli] multi-replica support for continuous jobs

CodeReviewBot added a comment.Sep 19 2024, 5:26 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/69

d/changelog: bump to 16.1.3

Stashbot added a comment.Sep 19 2024, 5:26 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:26:17Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:26 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:26:21Z] <wmbot~raymondndibe@wmf3402> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:27 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:27:06Z] <wmbot~raymondndibe@wmf3402> START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:27 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:27:28Z] <wmbot~raymondndibe@wmf3402> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:28 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:28:52Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:34 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:34:25Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli (T341066)

CodeReviewBot added a comment.Sep 19 2024, 5:35 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/521

[toolforge-deploy] test multi-replica support for continuous jobs

Stashbot added a comment.Sep 19 2024, 5:35 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:35:57Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:41 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:41:08Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:41 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:41:44Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli (T341066)

Stashbot added a comment.Sep 19 2024, 5:47 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-19T17:47:30Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli (T341066)

CodeReviewBot added a comment.Sep 19 2024, 5:55 PM

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/69

d/changelog: bump to 16.1.3

Maintenance_bot removed a project: Patch-For-Review.Sep 19 2024, 6:30 PM
Stashbot added a comment.Sep 20 2024, 7:19 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T19:19:34Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 7:24 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T19:24:49Z] <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 7:26 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T19:26:29Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 7:30 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T19:30:29Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 7:31 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T19:31:08Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 7:36 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T19:36:23Z] <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 8:06 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T20:06:47Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 8:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T20:08:23Z] <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 8:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T20:08:29Z] <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component calico (T341066)

Stashbot added a comment.Sep 20 2024, 8:12 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2024-09-20T20:12:51Z] <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component calico (T341066)

Raymond_Ndibe closed this task as Resolved.Sep 23 2024, 3:45 PM
Raymond_Ndibe moved this task from In Review to Done on the Toolforge (Toolforge iteration 14) board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits