Page MenuHomePhabricator
Create Task
Maniphest T348755

[jobs-api,webservice] Run webservices via the jobs framework
Open, In Progress, HighPublicFeature
Actions

Description

We want to move the webservice features to be hosted inside the jobs-api itself.

Started a design document to bootstrap the list of options, will move here once the discussion stabilizes

High level tasks:

  • Change image-config to have a webservice-defaults entry in each image (see https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/18#note_196349), with the overrides for that image (replaces the extras and wstype in the web variant config)
  • Add endpoint in jobs-api, using that configuration to override the jobs defaults if the user did not pass anything, this includes:
    • Adding a new API model to handle validation LegacyWebservice
    • It has to override any defaults with the default from image-config (and flag them as user-provided, so when returning the DefinedContinuousJob the values are there).
  • Make sure that the API docs are clear and flag it as deprecated pointing to the continuous job + ingress instead

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedLucasWerkmeisterT320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedmatmarexT319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
 ResolvedLegoktmT320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
 In ProgressdcaroT194332 [builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
 ResolveddcaroT393564 [Hypothesis] WE6.3.10 start a beta for the push-to-deploy features
 In ProgresskomlaT368600 [KR] WE6.3 Introduce a sustainability scoring system for the Toolforge platform
 ResolveddcaroT375199 [Hypothesis] WE6.3.4 If we enable the automatic deployment of a minimal tool, we will be able to evaluate the end to end flow and set the groundwork for adding support for more complex tools and deployment flows.
 ResolveddcaroT362051 [components-api] First iteration of the component API
 ResolvedRaymond_NdibeT362072 [components-api] Add support for port/healthcheck for continuous jobs in tool config/depolyment
 OpenNoneT362077 [components-api] Add webservice support
 OpenNoneT401102 Prebuilt webservice containers should not log to NFS by default
 In ProgressFeatureRaymond_NdibeT348755 [jobs-api,webservice] Run webservices via the jobs framework
 ResolvedFeatureRaymond_NdibeT341066 [jobs-api,jobs-cli] Support multiple replicas of continuous jobs
 ResolvedFeatureRaymond_NdibeT348758 [jobs-api,jobs-cli] Support services in jobs
 StalledRaymond_NdibeT359804 [jobs-api] Refactor before webservice support
 ResolveddcaroT359806 [jobs-api] Remove flask-restful
 ResolvedRaymond_NdibeT359808 [jobs-api] Split the API, core, and storage and runtime modules
 In ProgressNoneT359650 [jobs-api] Create storage layer, and save business models in persistent storage
 OpenNoneT358815 [jobs-api] separate jobs-framework k8s object templates from code
 ResolvedRaymond_NdibeT390135 [jobs-api] Split the core layer and create the core models
 DeclinedNoneT360016 [jobs-api,buildservice-api,envvars-api] evaluate crossplane for composite objects creation and maintenance
 ResolvedRaymond_NdibeT385904 [jobs-api] move jobs.toolforge.org/* labels to annotations
 ResolvedRaymond_NdibeT390136 [jobs-api] Split the `*Job` API models into three
 ResolvedRaymond_NdibeT390137 [jobs-api] Introduce deprecation metrics
 OpenNoneT390138 [jobs-api] Generate the openapi definition from the code
 ResolvedtaaviT401113 [jobs-api] Migrate to FastAPI
 ResolvedRaymond_NdibeT362520 [maintain-kubeusers] Increment default services quota
 ResolvedRaymond_NdibeT359649 [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version
 OpenNoneT311917 [webservice,toolforge-cli] Make `webservice shell` a standalone tool
 ResolvedFeatureRaymond_NdibeT362621 Support HTTP health checks in jobs framework
 ResolvedFeatureRaymond_NdibeT335592 [jobs-api,jobs-cli] Support job health checks
 InvalidRaymond_NdibeT388088 [jobs-api] add -l|--last to toolforge jobs logs ...
 InvalidRaymond_NdibeT388090 [jobs-api] jobs-api should be able to read webservices started with toolforge webservice
 DuplicateRaymond_NdibeT388091 [jobs-api] add toolforge jobs shell feature
 In ProgressRaymond_NdibeT388092 [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice
 InvalidRaymond_NdibeT392717 [jobs-api] check for diff in services when running diff_with_running_job
 In ProgressRaymond_NdibeT423408 [jobs-api] support exposing continuous jobs to the internet
 In ProgressRaymond_NdibeT423410 [jobs-cli] support publishing continuous job to the internet
 OpenNoneT409728 [image-config] deprecate and move all data to builds-api
 OpenNoneT409725 [jobs-api,webservice] Fetch images from builds-api
 OpenNoneT409726 [builds-api] Add an endpoint to get all available images
 StalledRaymond_NdibeT409727 [builds-api,harbor,image-config] Move pre-built images to harbor
 DuplicateRaymond_NdibeT409900 [jobs-api,image-config] Deprecate/update the list of supported pre-built images
 ResolvedRaymond_NdibeT415322 [jobs-api] Use the same images as webservice
 ResolvedRaymond_NdibeT409191 [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dcaro renamed this task from Run webservices via the jobs framework to [jobs-api,webservice] Run webservices via the jobs framework.Mar 11 2024, 11:42 AM
dcaro removed a subtask: T358815: [jobs-api] separate jobs-framework k8s object templates from code.
dcaro removed a subtask: T359650: [jobs-api] Create storage layer, and save business models in persistent storage.
dcaro added a subtask: T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version.Mar 11 2024, 12:52 PM
taavi added a subtask: T311917: [webservice,toolforge-cli] Make `webservice shell` a standalone tool.Mar 13 2024, 9:38 AM
Raymond_Ndibe closed subtask T335592: [jobs-api,jobs-cli] Support job health checks as Resolved.Apr 4 2024, 9:57 PM
taavi reopened subtask T335592: [jobs-api,jobs-cli] Support job health checks as Open.Apr 5 2024, 11:53 AM
taavi mentioned this in T335592: [jobs-api,jobs-cli] Support job health checks.
dcaro mentioned this in T362051: [components-api] First iteration of the component API.Apr 8 2024, 12:30 PM
dcaro added a parent task: T362051: [components-api] First iteration of the component API.Apr 8 2024, 12:32 PM
dcaro changed the status of subtask T335592: [jobs-api,jobs-cli] Support job health checks from Open to In Progress.
dcaro added a parent task: T362077: [components-api] Add webservice support.Apr 8 2024, 1:21 PM
Raymond_Ndibe closed subtask T335592: [jobs-api,jobs-cli] Support job health checks as Resolved.Apr 16 2024, 2:59 AM
taavi removed a subtask: T335592: [jobs-api,jobs-cli] Support job health checks.Apr 16 2024, 9:08 AM
dcaro changed the status of subtask T348758: [jobs-api,jobs-cli] Support services in jobs from Open to In Progress.Apr 30 2024, 1:15 PM
taavi mentioned this in T362012: [jobs] Allow configuration of Promethus scraping of a specific endpoint for publication in grafana.wmcloud.org.Jun 7 2024, 4:12 PM
Raymond_Ndibe closed subtask T348758: [jobs-api,jobs-cli] Support services in jobs as Resolved.Jun 20 2024, 11:48 AM
• joanna_borun added a project: cloud-services-team (FY2024/2025-Q1-Q2).Jul 22 2024, 9:38 AM
Raymond_Ndibe changed the status of subtask T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs from Open to In Progress.Aug 12 2024, 5:26 PM
bd808 mentioned this in T374062: Support running a job using an alternate service account.Sep 4 2024, 9:36 PM
Raymond_Ndibe closed subtask T341066: [jobs-api,jobs-cli] Support multiple replicas of continuous jobs as Resolved.Sep 23 2024, 3:45 PM
Raymond_Ndibe changed the status of subtask T362621: Support HTTP health checks in jobs framework from Open to In Progress.Jan 16 2025, 4:16 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (FY2024/2025-Q1-Q2).Feb 5 2025, 6:19 PM
fnegri moved this task from FY2024/2025-Q1-Q2 to Inbox on the cloud-services-team board.
Raymond_Ndibe closed subtask T362621: Support HTTP health checks in jobs framework as Resolved.Feb 18 2025, 7:55 PM
Raymond_Ndibe moved this task from Ready to be worked on to Toolforge iteration 18 on the Toolforge board.Mar 13 2025, 2:47 PM
Raymond_Ndibe edited projects, added Toolforge (Toolforge iteration 18); removed Toolforge.
Raymond_Ndibe changed the task status from Open to In Progress.Mar 13 2025, 2:51 PM
Raymond_Ndibe moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 18) board.
dcaro edited projects, added Toolforge (Toolforge iteration 19); removed Toolforge (Toolforge iteration 18).Mar 18 2025, 3:56 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 19) board.
Raymond_Ndibe changed the status of subtask T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version from Open to In Progress.Apr 14 2025, 2:21 AM
Raymond_Ndibe changed the status of subtask T388092: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice from Open to In Progress.Apr 25 2025, 3:23 AM
dcaro mentioned this in T341065: [builds-api,components-api] Automatically deploy the webservice when the image is built.May 7 2025, 8:39 AM
dcaro edited projects, added Toolforge (Toolforge iteration 20); removed Toolforge (Toolforge iteration 19).May 14 2025, 2:30 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 20) board.
dcaro edited projects, added Toolforge (Toolforge iteration 21); removed Toolforge (Toolforge iteration 20).Jun 11 2025, 10:38 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 21) board.
dcaro edited projects, added Toolforge (Toolforge iteration 22); removed Toolforge (Toolforge iteration 21).Jul 14 2025, 9:49 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 22) board.
Raymond_Ndibe changed the task status from In Progress to Stalled.Jul 15 2025, 11:30 AM
Raymond_Ndibe moved this task from In Progress to Blocked/Paused on the Toolforge (Toolforge iteration 22) board.
taavi closed subtask T388088: [jobs-api] add -l|--last to toolforge jobs logs ... as Invalid.Jul 24 2025, 6:20 PM
taavi added a parent task: T401102: Prebuilt webservice containers should not log to NFS by default.Aug 4 2025, 9:47 AM
dcaro edited projects, added Toolforge (Toolforge iteration 23); removed Toolforge (Toolforge iteration 22).Aug 5 2025, 3:45 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 23) board.
dcaro edited projects, added Toolforge (Toolforge iteration 24); removed Toolforge (Toolforge iteration 23).Aug 27 2025, 9:43 AM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 24) board.
fnegri moved this task from Inbox to Epics on the cloud-services-team board.Sep 5 2025, 8:41 AM
dcaro closed subtask T388090: [jobs-api] jobs-api should be able to read webservices started with toolforge webservice as Invalid.Oct 22 2025, 2:11 PM
dcaro updated the task description. (Show Details)Oct 22 2025, 2:15 PM
dcaro edited projects, added Toolforge (Toolforge iteration 25); removed Toolforge (Toolforge iteration 24).Oct 29 2025, 5:23 PM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 25) board.
dcaro added a subtask: T409191: [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images.Nov 4 2025, 5:13 PM
fnegri subscribed.Nov 6 2025, 5:05 PM
dcaro removed a subtask: T409725: [jobs-api,webservice] Fetch images from builds-api.Nov 10 2025, 1:19 PM
Raymond_Ndibe changed the status of subtask T409191: [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images from Open to In Progress.Nov 25 2025, 1:46 AM
fnegri changed the task status from Stalled to Open.Jan 13 2026, 4:49 PM
fnegri edited projects, added cloud-services-team (FY2025/2026-Q3-Q4); removed cloud-services-team.
fnegri changed the status of subtask T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version from In Progress to Open.Jan 13 2026, 5:20 PM
fnegri removed Raymond_Ndibe as the assignee of this task.Jan 13 2026, 5:40 PM
taavi edited projects, added Toolforge; removed Toolforge (Toolforge iteration 25).Jan 13 2026, 5:48 PM
fnegri changed the status of subtask T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version from Open to In Progress.Jan 15 2026, 3:27 PM
Raymond_Ndibe changed the task status from Open to In Progress.Jan 27 2026, 10:18 PM
Raymond_Ndibe claimed this task.
Raymond_Ndibe moved this task from Backlog to Toolforge iteration 25 on the Toolforge board.
Raymond_Ndibe edited projects, added Toolforge (Toolforge iteration 25); removed Toolforge.
Raymond_Ndibe moved this task from Blocked/Paused to In Progress on the Toolforge (Toolforge iteration 25) board.
CodeReviewBot added a comment.Jan 27 2026, 11:07 PM

raymond-ndibe updated https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264

add webservice endpoint

CodeReviewBot added a project: Patch-For-Review.Jan 27 2026, 11:10 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143

support --webservice option

fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/2026-Q3-Q4) board.Jan 28 2026, 4:51 PM
fnegri removed a subtask: T409191: [jobs-api] Investigate if we can reuse the 'web' flavour pre-built images as regular images.Jan 29 2026, 2:48 PM
fnegri added a subtask: T415322: [jobs-api] Use the same images as webservice.Jan 29 2026, 2:50 PM
CodeReviewBot added a comment.Feb 2 2026, 5:20 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131

test continuous job webservice

CodeReviewBot added a comment.Feb 5 2026, 12:27 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95

add jobs-api backend to webservice-cli

CodeReviewBot added a comment.Feb 5 2026, 2:02 PM

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134

jobs-api: test for proper handling of the diff variations of the --image argument

CodeReviewBot added a comment.Feb 5 2026, 2:11 PM

raymond-ndibe updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1134

test for jobs-api backend in webservice-cli

dcaro changed the task status from In Progress to Stalled.Mar 3 2026, 2:33 PM
dcaro moved this task from In Progress to Blocked/Paused on the Toolforge (Toolforge iteration 25) board.
dcaro changed the status of subtask T359804: [jobs-api] Refactor before webservice support from Open to Stalled.Mar 3 2026, 3:26 PM
dcaro closed subtask T359649: [jobs-api,infra] upgrade all the existing toolforge jobs to the latest job version as Resolved.Mar 3 2026, 5:49 PM
dcaro edited projects, added Toolforge (Toolforge iteration 26); removed Toolforge (Toolforge iteration 25).Mar 5 2026, 9:35 AM
dcaro moved this task from Next Up to Blocked/Paused on the Toolforge (Toolforge iteration 26) board.
dcaro added a project: tools-platform-team.Apr 2 2026, 12:18 PM
dcaro updated the task description. (Show Details)Apr 2 2026, 12:48 PM
aputhin changed the status of subtask T388092: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice from In Progress to Stalled.Apr 8 2026, 2:54 PM
taavi edited projects, added Toolforge; removed Toolforge (Toolforge iteration 26).Apr 9 2026, 3:30 PM
fnegri mentioned this in T127367: [toolforge,jobs-api,webservice,storage] Provide modern, non-NFS log solution for Toolforge tools.Apr 24 2026, 2:23 PM
Count_Count subscribed.Apr 24 2026, 2:33 PM
aputhin moved this task from Backlog to Todos on the tools-platform-team board.Apr 29 2026, 2:18 PM
fnegri removed a project: cloud-services-team (FY2025/2026-Q3-Q4).Apr 29 2026, 2:53 PM
aputhin mentioned this in T425215: toolforge webservice restart does not wait for pod to be ready, only running.Wed, May 13, 2:31 PM
Raymond_Ndibe closed subtask T415322: [jobs-api] Use the same images as webservice as Resolved.Wed, May 20, 1:38 AM
Raymond_Ndibe changed the task status from Stalled to In Progress.Tue, Jun 2, 3:35 PM
Raymond_Ndibe moved this task from Todos to In progress on the tools-platform-team board.
Raymond_Ndibe changed the status of subtask T388092: [jobs-api] allow exposing continuous jobs to the internet via `toolname.toolforge.org`, just like webservice from Stalled to In Progress.
aputhin edited projects, added Toolforge (Push-to-Deploy); removed Toolforge.Wed, Jun 3, 10:13 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits