Page MenuHomePhabricator
Create Task
Maniphest T348755

[jobs-api,webservice] Run webservices via the jobs framework
Open, In Progress, HighPublicFeature
Actions

Description

We want to move the webservice features to be hosted inside the jobs-api itself.

Started a design document to bootstrap the list of options, will move here once the discussion stabilizes

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLucasWerkmeisterT320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedmatmarexT319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
 In ProgressdcaroT194332 [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
 ResolveddcaroT393564 [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features
 In ProgresskomlaT368600 [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform
 ResolveddcaroT375199 [Hypothesis] WE6.3.4 If we enable the automatic deployment of a minimal tool, we will be able to evaluate the end to end flow and set the groundwork for adding support for more complex tools and deployment flows.
 ResolveddcaroT362051 [components-api] First iteration of the component API
 ResolvedRaymond_NdibeT362072 [components-api] Add support for port/healthcheck for continuous jobs in tool config/depolyment
 OpendcaroT362077 [components-api] Add webservice support
 OpenNoneT401102 Prebuilt webservice containers should not log to NFS by default
 In ProgressFeatureRaymond_NdibeT348755 [jobs-api,webservice] Run webservices via the jobs framework
 ResolvedFeatureRaymond_NdibeT341066 [jobs-api,jobs-cli] Support multiple replicas of continuous jobs
 ResolvedFeatureRaymond_NdibeT348758 [jobs-api,jobs-cli] Support services in jobs
 OpenRaymond_NdibeT359804 [jobs-api] Refactor before webservice support
 ResolveddcaroT359806 [jobs-api] Remove flask-restful
 ResolvedRaymond_NdibeT359808 [jobs-api] Split the API, core, and storage and runtime modules
 In ProgressRaymond_NdibeT359650 [jobs-api] Create storage layer, and save business models in persistent storage
 OpenRaymond_NdibeT358815 [jobs-api] separate jobs-framework k8s object templates from code
 ResolvedRaymond_NdibeT390135 [jobs-api] Split the core layer and create the core models
 DeclinedNoneT360016 [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance
 ResolvedRaymond_NdibeT385904 [jobs-api] move jobs.toolforge.org/* labels to annotations
 ResolvedRaymond_NdibeT390136 [jobs-api] Split the `*Job` API models into three
 ResolvedRaymond_NdibeT390137 [jobs-api] Introduce deprecation metrics
 OpenRaymond_NdibeT390138 [jobs-api] Generate the openapi definition from the code
 ResolvedtaaviT401113 [jobs-api] Migrate to FastAPI
 ResolvedRaymond_NdibeT362520 [maintain-kubeusers] Increment default services quota
 In ProgressRaymond_NdibeT359649 [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version
 OpenNoneT311917 [webservice,toolforge-cli] Make `webservice shell` a standalone tool
 ResolvedFeatureRaymond_NdibeT362621 Support HTTP health checks in jobs framework
 ResolvedFeatureRaymond_NdibeT335592 [jobs-api,jobs-cli] Support job health checks
 InvalidRaymond_NdibeT388088 [jobs-api] add -l|--last to toolforge jobs logs ...
 InvalidRaymond_NdibeT388090 [jobs-api] jobs-api should be able to read webservices started with toolforge webservice
 DuplicateRaymond_NdibeT388091 [jobs-api] add toolforge jobs shell feature
 In ProgressRaymond_NdibeT388092 [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice
 InvalidRaymond_NdibeT392717 [jobs-api] check for diff in services when running diff_with_running_job
 OpenRaymond_NdibeT409728 [image-config] deprecate and move all data to builds-api
 OpenRaymond_NdibeT409725 [jobs-api,webservice] Fetch images from builds-api
 OpenRaymond_NdibeT409726 [builds-api] Add an endpoint to get all available images
 In ProgressRaymond_NdibeT409727 [builds-api,harbor,image-config] Move pre-built images to harbor
 DuplicateRaymond_NdibeT409900 [jobs-api,image-config] Deprecate/update the list of supported pre-built images
 In ProgressRaymond_NdibeT415322 Replace job image variants with webservice image variants
 ResolvedRaymond_NdibeT409191 [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
taavi added a subtask: T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs.Oct 12 2023, 2:20 PM
taavi added a subtask: T348758: [jobs-api,jobs-cli] Support services in jobs.Oct 12 2023, 2:28 PM
bd808 changed the subtype of this task from "Task" to "Feature Request".Jan 5 2024, 1:45 AM
bd808 added a project: Epic.
dcaro subscribed.Jan 31 2024, 2:06 PM

This is being discussed as part of the toolforge user stories 2024: https://docs.google.com/document/d/1sqo6YGRn9u-S7V0y9m07cYKA84vQlKa-7_F8p7eg80Y/edit

taavi mentioned this in T357915: [webservice,jobs-api] Move logic to an extendend continuous job.Feb 19 2024, 3:44 PM
dcaro merged a task: T357915: [webservice,jobs-api] Move logic to an extendend continuous job.Feb 19 2024, 3:46 PM
dcaro edited projects, added Toolforge; removed Toolforge Jobs framework.
dcaro updated the task description. (Show Details)
dcaro added a subscriber: Raymond_Ndibe.
Raymond_Ndibe added a project: User-Raymond_Ndibe.Feb 19 2024, 3:48 PM
dcaro triaged this task as High priority.Feb 21 2024, 9:23 AM
dcaro moved this task from Backlog to Workspace for triaging whenever needed on the Toolforge board.
dcaro moved this task from Workspace for triaging whenever needed to Ready to be worked on on the Toolforge board.Feb 21 2024, 4:03 PM
taavi mentioned this in T358175: dbreps job pending to start for 2d16h on Toolforge.Feb 23 2024, 10:11 AM
taavi added a subtask: T335592: [jobs-api,jobs-cli] Support job health checks.Feb 28 2024, 12:09 PM
Raymond_Ndibe changed the status of subtask T335592: [jobs-api,jobs-cli] Support job health checks from Open to In Progress.Mar 4 2024, 5:30 PM
Raymond_Ndibe claimed this task.Mar 5 2024, 1:29 PM
dcaro added a subtask: T359650: [jobs-api] Create storage layer, and save business models in persistent storage.Mar 11 2024, 11:18 AM
dcaro updated the task description. (Show Details)Mar 11 2024, 11:25 AM
dcaro renamed this task from Run webservices via the jobs framework to [jobs-api,webservice] Run webservices via the jobs framework.Mar 11 2024, 11:42 AM
dcaro removed a subtask: T358815: [jobs-api] separate jobs-framework k8s object templates from code.
dcaro removed a subtask: T359650: [jobs-api] Create storage layer, and save business models in persistent storage.
dcaro added a subtask: T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version.Mar 11 2024, 12:52 PM
taavi added a subtask: T311917: [webservice,toolforge-cli] Make `webservice shell` a standalone tool.Mar 13 2024, 9:38 AM
Raymond_Ndibe closed subtask T335592: [jobs-api,jobs-cli] Support job health checks as Resolved.Apr 4 2024, 9:57 PM
taavi reopened subtask T335592: [jobs-api,jobs-cli] Support job health checks as Open.Apr 5 2024, 11:53 AM
taavi mentioned this in T335592: [jobs-api,jobs-cli] Support job health checks.
dcaro mentioned this in T362051: [components-api] First iteration of the component API.Apr 8 2024, 12:30 PM
dcaro added a parent task: T362051: [components-api] First iteration of the component API.Apr 8 2024, 12:32 PM
dcaro changed the status of subtask T335592: [jobs-api,jobs-cli] Support job health checks from Open to In Progress.
dcaro added a parent task: T362077: [components-api] Add webservice support.Apr 8 2024, 1:21 PM
Raymond_Ndibe closed subtask T335592: [jobs-api,jobs-cli] Support job health checks as Resolved.Apr 16 2024, 2:59 AM
taavi removed a subtask: T335592: [jobs-api,jobs-cli] Support job health checks.Apr 16 2024, 9:08 AM
dcaro changed the status of subtask T348758: [jobs-api,jobs-cli] Support services in jobs from Open to In Progress.Apr 30 2024, 1:15 PM
taavi mentioned this in T362012: [jobs] Allow configuration of Promethus scraping of a specific endpoint for publication in grafana.wmcloud.org.Jun 7 2024, 4:12 PM
Raymond_Ndibe closed subtask T348758: [jobs-api,jobs-cli] Support services in jobs as Resolved.Jun 20 2024, 11:48 AM
• joanna_borun added a project: cloud-services-team (FY2024/2025-Q1-Q2).Jul 22 2024, 9:38 AM
Raymond_Ndibe changed the status of subtask T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs from Open to In Progress.Aug 12 2024, 5:26 PM
bd808 mentioned this in T374062: Support running a job using an alternate service account.Sep 4 2024, 9:36 PM
Raymond_Ndibe closed subtask T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs as Resolved.Sep 23 2024, 3:45 PM
Raymond_Ndibe changed the status of subtask T362621: Support HTTP health checks in jobs framework from Open to In Progress.Jan 16 2025, 4:16 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (FY2024/2025-Q1-Q2).Feb 5 2025, 6:19 PM
fnegri moved this task from FY2024/2025-Q1-Q2 to Inbox on the cloud-services-team board.
Raymond_Ndibe closed subtask T362621: Support HTTP health checks in jobs framework as Resolved.Feb 18 2025, 7:55 PM
Raymond_Ndibe moved this task from Ready to be worked on to Toolforge iteration 18 on the Toolforge board.Mar 13 2025, 2:47 PM
Raymond_Ndibe edited projects, added Toolforge (Toolforge iteration 18); removed Toolforge.
Raymond_Ndibe changed the task status from Open to In Progress.Mar 13 2025, 2:51 PM
Raymond_Ndibe moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 18) board.
dcaro edited projects, added Toolforge (Toolforge iteration 19); removed Toolforge (Toolforge iteration 18).Mar 18 2025, 3:56 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 19) board.
Raymond_Ndibe changed the status of subtask T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version from Open to In Progress.Apr 14 2025, 2:21 AM
Raymond_Ndibe changed the status of subtask T388092: [jobs-api] expose jobs-api continuous jobs to the internet via `toolname.toolforge.org`, just like webservice from Open to In Progress.Apr 25 2025, 3:23 AM
dcaro mentioned this in T341065: [builds-api,components-api] Automatically deploy the webservice when the image is built.May 7 2025, 8:39 AM
dcaro edited projects, added Toolforge (Toolforge iteration 20); removed Toolforge (Toolforge iteration 19).May 14 2025, 2:30 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 20) board.
dcaro edited projects, added Toolforge (Toolforge iteration 21); removed Toolforge (Toolforge iteration 20).Jun 11 2025, 10:38 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 21) board.
dcaro edited projects, added Toolforge (Toolforge iteration 22); removed Toolforge (Toolforge iteration 21).Jul 14 2025, 9:49 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 22) board.
Raymond_Ndibe changed the task status from In Progress to Stalled.Jul 15 2025, 11:30 AM
Raymond_Ndibe moved this task from In Progress to Blocked/Paused on the Toolforge (Toolforge iteration 22) board.
taavi closed subtask T388088: [jobs-api] add -l|--last to toolforge jobs logs ... as Invalid.Jul 24 2025, 6:20 PM
taavi added a parent task: T401102: Prebuilt webservice containers should not log to NFS by default.Aug 4 2025, 9:47 AM
dcaro edited projects, added Toolforge (Toolforge iteration 23); removed Toolforge (Toolforge iteration 22).Aug 5 2025, 3:45 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 23) board.
dcaro edited projects, added Toolforge (Toolforge iteration 24); removed Toolforge (Toolforge iteration 23).Aug 27 2025, 9:43 AM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 24) board.
fnegri moved this task from Inbox to Epics on the cloud-services-team board.Sep 5 2025, 8:41 AM
dcaro closed subtask T388090: [jobs-api] jobs-api should be able to read webservices started with toolforge webservice as Invalid.Oct 22 2025, 2:11 PM
dcaro updated the task description. (Show Details)Oct 22 2025, 2:15 PM
dcaro edited projects, added Toolforge (Toolforge iteration 25); removed Toolforge (Toolforge iteration 24).Oct 29 2025, 5:23 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 25) board.
dcaro added a subtask: T409191: [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images.Nov 4 2025, 5:13 PM
fnegri subscribed.Nov 6 2025, 5:05 PM
dcaro removed a subtask: T409725: [jobs-api,webservice] Fetch images from builds-api.Nov 10 2025, 1:19 PM
Raymond_Ndibe changed the status of subtask T409191: [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images from Open to In Progress.Nov 25 2025, 1:46 AM
fnegri changed the task status from Stalled to Open.Tue, Jan 13, 4:49 PM
fnegri edited projects, added cloud-services-team (FY2025/2026-Q3-Q4); removed cloud-services-team.
fnegri changed the status of subtask T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version from In Progress to Open.Tue, Jan 13, 5:20 PM
fnegri removed Raymond_Ndibe as the assignee of this task.Tue, Jan 13, 5:40 PM
taavi edited projects, added Toolforge; removed Toolforge (Toolforge iteration 25).Tue, Jan 13, 5:48 PM
fnegri changed the status of subtask T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version from Open to In Progress.Thu, Jan 15, 3:27 PM
Raymond_Ndibe changed the task status from Open to In Progress.Tue, Jan 27, 10:18 PM
Raymond_Ndibe claimed this task.
Raymond_Ndibe moved this task from Backlog to Toolforge iteration 25 on the Toolforge board.
Raymond_Ndibe edited projects, added Toolforge (Toolforge iteration 25); removed Toolforge.
Raymond_Ndibe moved this task from Blocked/Paused to In Progress on the Toolforge (Toolforge iteration 25) board.
CodeReviewBot added a comment.Tue, Jan 27, 11:07 PM

raymond-ndibe updated https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264

add webservice endpoint

CodeReviewBot added a project: Patch-For-Review.Tue, Jan 27, 11:10 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143

support --webservice option

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/2026-Q3-Q4) board.Wed, Jan 28, 4:51 PM
fnegri removed a subtask: T409191: [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images.Thu, Jan 29, 2:48 PM
fnegri added a subtask: T415322: Replace job image variants with webservice image variants.Thu, Jan 29, 2:50 PM
CodeReviewBot added a comment.Mon, Feb 2, 5:20 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131

test continuous job webservice

CodeReviewBot added a comment.Thu, Feb 5, 12:27 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95

add jobs-api backend to webservice-cli

CodeReviewBot added a comment.Thu, Feb 5, 2:02 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134

jobs-api: test for proper handling of the diff variations of the --image argument

CodeReviewBot added a comment.Thu, Feb 5, 2:11 PM

raymond-ndibe updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134

test for jobs-api backend in webservice-cli

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits