Page MenuHomePhabricator
Create Task
Maniphest T194332

[Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
Open, MediumPublic
Actions

Description

Make usage of Toolforge more familiar to users of commercial Platform as a Service (PaaS) products (Heroku, DigitalOcean, Google App Engine, etc) by implementing a Container based PaaS product that provides "build pack" and "push to deploy" workflows for deploying and scaling Tool owned processes.

  • roughly estimated 9-12 month project
  • initial: 1 root, 1 SWE, .25 tech writer, .25 community liaison
  • sustaining: .5 root, .5 SWE, .10 tech writer, .10 community liaison

https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_push_to_deploy

Design Documentation: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Buildpack_Implementation

Details

SubjectRepoBranchLines +/-
toolsbeta: For testing, I'd rather not upload tekton to the internal reglabs/tools/registry-admission-webhookmaster+5 -0
Customize query in gerrit

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
ResolvedLucasWerkmeisterT320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes
ResolvedmatmarexT319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes
ResolvedLegoktmT320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes
ResolvedLegoktmT320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes
OpendcaroT194332 [Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs
DeclinedNoneT136265 Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions
DeclinedNoneT246163 Replace toollabs-docker-images with cloud-services-buildpacks
Declined• BstormT249946 Basic evaluation of gitea as a repo setup
ResolveddcaroT117071 Figure out a git hosting solution for tools/kubernetes
Resolved• BstormT265684 Figure out CI solution for building buildpack-based images for Toolforge
ResolvedLegoktmT265686 Create Python 3.7 buildpack builder for Toolforge
ResolvedLegoktmT266270 Install pack on Toolforge servers
ResolveddcaroT266283 Figure out how buildpack versioning is supposed to work
ResolvedLegoktmT266692 Allow Toolforge's service.template to be in webservice code directories
DuplicateNoneT266715 Allow Toolforge users to install apt packages in their buildpacks
ResolveddcaroT266716 Use a proper yaml parser in Toolforge buildpacks
ResolveddcaroT266901 Ensure webservice plays nicely with Toolforge tools using buildpack images
ResolveddcaroT267374 [tbs.beta] Create a toolforge build service beta release
ResolvedSlst2020T325799 [tbs] User story - I can use multiple language stacks for my application
ResolveddcaroT325382 Decision request - Choose a lang for the toolforge build service API
ResolveddcaroT327057 [toolforge] repositories move to gitlab
ResolveddcaroT334081 [tbs] Define what NFS access to enable and how users will interact with it
ResolveddcaroT334585 [cookbooks.wmcs.toolforge.component.deploy] Add secrets support when deploying
ResolveddcaroT304532 buildservice: migrate to helmfile from of kustomize
DuplicateNoneT334587 [builds-api] Add triggering support
ResolvedRaymond_NdibeT334588 [toolforge-cli.build] Split the build subcommand to it's own package
OpenNoneT334589 [toolforge-cli.build] Implement a --json flag to output machine-readable output
ResolveddcaroT334590 [buildservice] Create a buildservice API and move any logic from the client to it
DuplicatedcaroT334596 [buildservice] Cleanup old runs periodically
ResolveddcaroT293668 [tbs]Find a way to easy pull logs from build process
ResolveddcaroT338142 [toolforge-envvars.api,toolforge-build.api] Support using custom environment variables at build time
ResolvedNoneT341065 [builds-api,components-api] Automatically deploy the webservice when the image is built
ResolvedSlst2020T342077 Toolforge beyond build service
ResolvedaborreroT342853 toolforge lima-kilo: introduce harbor support
ResolvedSecuritydcaroT346311 [builds-api] Harbor secret is public if using env vars, move it to a Secret resource
ResolveddcaroT368602 [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment
OpendcaroT362051 [components-api] First iteration of the component API
· · ·

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
• bd808 mentioned this in T308988: Add make and gdal package to tf-bullseye-std toolforge-jobs image.Jun 21 2022, 3:10 PM
• bd808 closed subtask T136265: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions as Declined.Jun 29 2022, 9:32 PM
• bd808 mentioned this in T136265: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions.
• bd808 mentioned this in T136264: Evaluate Kubernetes based workflow replacement options for SGE.Jun 29 2022, 9:35 PM
AntiCompositeNumber subscribed.Jun 29 2022, 9:53 PM
Slst2020 subscribed.Jul 25 2022, 7:52 AM
dcaro moved this task from Backlog to To be Discussed on the Toolforge Build Service board.Aug 25 2022, 10:36 AM
dcaro added a subtask: T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions.Aug 25 2022, 2:48 PM
dcaro added a subtask: T316231: [tbs.cli.build-logs] Show a warning to the user if it’s not admin saying that they will not be able to read the logs.Aug 25 2022, 3:24 PM
dcaro added a subtask: T316232: [tbs.harbor.toolsbeta] Create a postgres DB with trove and plug the current instance there.Aug 25 2022, 3:29 PM
dcaro added a subtask: T316233: [tbs.harbor.toolsbeta] Create a cinder volume and put there the storage for the container images.Aug 25 2022, 3:35 PM
Slst2020 changed the status of subtask T316222: [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info) from Open to In Progress.Aug 26 2022, 8:04 AM
Slst2020 changed the status of subtask T316222: [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info) from In Progress to Open.Aug 26 2022, 8:21 AM
dcaro added a subtask: T316323: [tbs.harbor.tools] Replicate toolsbeta harbor deployment.Aug 26 2022, 8:26 AM
dcaro closed subtask T117071: Figure out a git hosting solution for tools/kubernetes as Resolved.Aug 26 2022, 8:30 AM
dcaro removed a subtask: T316233: [tbs.harbor.toolsbeta] Create a cinder volume and put there the storage for the container images.Aug 26 2022, 8:50 AM
dcaro removed a subtask: T316232: [tbs.harbor.toolsbeta] Create a postgres DB with trove and plug the current instance there.
dcaro removed a subtask: T316323: [tbs.harbor.tools] Replicate toolsbeta harbor deployment.
dcaro removed a subtask: T265681: [tbs.tools] Deploy a tool using a buildpack-based image on Toolforge.
dcaro removed a subtask: T265557: [tbs.maintainkubeusers] Create security policies for running buildpack images.Aug 26 2022, 8:52 AM
dcaro removed a subtask: T265685: [tbs] Set up CI for cloud/toolforge/buildpacks repository.Aug 26 2022, 8:55 AM
dcaro removed a subtask: T265691: [tbs.maintaindbusers] Update maintain-dbusers to create an envvar with replica.my.cnf to be used by buildpack images.
dcaro removed a subtask: T265781: [tbs] Upgrade Toolforge buildpacks to api 0.8 (latest).Aug 26 2022, 9:12 AM
dcaro removed a subtask: T265780: [tbs.docs]Figure out naming conventions for Toolforge buildpack related stuff.
dcaro removed a subtask: T316231: [tbs.cli.build-logs] Show a warning to the user if it’s not admin saying that they will not be able to read the logs.Aug 26 2022, 9:15 AM
dcaro removed a subtask: T316228: [tbs.cli] Define the process to build and deploy the CLI on toolforge bastions.
dcaro removed a subtask: T316226: [tbs.cli]: toolforge build-show shows unknown for steps that are actually ok.
dcaro removed a subtask: T316222: [tbs.cli]: Create reasonable defaults for the build command (so the users have to add only the minimum needed info).
dcaro removed a subtask: T267062: Allow Toolforge users to manually start/restart/stop buildpack based web servers.
KHernandez-WMF moved this task from To be Discussed to Backlog on the Toolforge Build Service board.Oct 5 2022, 4:38 PM
LucasWerkmeister added a parent task: T320140: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes.Oct 7 2022, 8:10 AM
• bd808 added a parent task: T319707: Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes.Oct 7 2022, 5:29 PM
Legoktm added parent tasks: T320062: Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes, T320011: Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes.Oct 15 2022, 3:32 AM
• bd808 mentioned this in T304983: tf-bullseye-std is missing libssl-dev.Oct 17 2022, 4:17 PM
• nskaggs mentioned this in T320118: Migrate vcat from Toolforge GridEngine to Toolforge Kubernetes.Oct 17 2022, 10:28 PM
• nskaggs mentioned this in T320157: Migrate wikihistory from Toolforge GridEngine to Toolforge Kubernetes.
thcipriani subscribed.Oct 25 2022, 2:16 PM
dcaro closed subtask T266901: Ensure webservice plays nicely with Toolforge tools using buildpack images as Resolved.Nov 18 2022, 12:10 PM
dcaro removed a subtask: T323555: [tbs][cli] Implement the global --verbose flag as an env var.Nov 22 2022, 2:46 PM
dcaro added a subtask: T267374: [tbs.beta] Create a toolforge build service beta release.Dec 13 2022, 5:54 PM
dcaro added a subtask: T325799: [tbs] User story - I can use multiple language stacks for my application.Dec 22 2022, 8:41 AM
dcaro added a subtask: T325382: Decision request - Choose a lang for the toolforge build service API.Dec 23 2022, 12:13 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:45 PM
fnegri moved this task from Kanban to Epics on the cloud-services-team board.
dcaro changed the status of subtask T267374: [tbs.beta] Create a toolforge build service beta release from Open to In Progress.Jan 31 2023, 9:57 AM
dcaro closed subtask T325382: Decision request - Choose a lang for the toolforge build service API as Resolved.Feb 23 2023, 4:55 PM
dcaro moved this task from Backlog to Epics on the Toolforge Build Service board.Feb 28 2023, 10:47 AM
dcaro added a subtask: T327057: [toolforge] repositories move to gitlab.Mar 6 2023, 5:23 PM
Legoktm mentioned this in T319724: Migrate enwikt-translations from Toolforge GridEngine to Toolforge Kubernetes.Apr 8 2023, 7:20 PM
dcaro added a subtask: T304532: buildservice: migrate to helmfile from of kustomize.Apr 13 2023, 2:08 PM
dcaro added a subtask: T293668: [tbs]Find a way to easy pull logs from build process.
taavi removed a subtask: T257386: Fix the proxy behavior for Kubernetes on toolsbeta.May 7 2023, 12:39 PM
dcaro moved this task from To-do to Implementation on the Cloud Services Proposals board.May 26 2023, 7:51 AM
dcaro closed subtask T293668: [tbs]Find a way to easy pull logs from build process as Resolved.Jun 2 2023, 7:35 AM
KHernandez-WMF changed the status of subtask T334590: [buildservice] Create a buildservice API and move any logic from the client to it from Open to In Progress.Jun 13 2023, 1:43 PM
KHernandez-WMF changed the status of subtask T334588: [toolforge-cli.build] Split the build subcommand to it's own package from Open to In Progress.
KHernandez-WMF changed the status of subtask T304532: buildservice: migrate to helmfile from of kustomize from Open to In Progress.
dcaro changed the status of subtask T304532: buildservice: migrate to helmfile from of kustomize from In Progress to Open.Jul 4 2023, 3:21 PM
dcaro changed the status of subtask T325799: [tbs] User story - I can use multiple language stacks for my application from Open to In Progress.Jul 4 2023, 3:31 PM
dcaro changed the status of subtask T325799: [tbs] User story - I can use multiple language stacks for my application from In Progress to Open.Jul 4 2023, 3:35 PM
Raymond_Ndibe closed subtask T334588: [toolforge-cli.build] Split the build subcommand to it's own package as Resolved.Jul 19 2023, 6:03 PM
dcaro closed subtask T266716: Use a proper yaml parser in Toolforge buildpacks as Resolved.Aug 3 2023, 4:51 PM
dcaro changed the status of subtask T341065: [builds-api,components-api] Automatically deploy the webservice when the image is built from Open to Stalled.Aug 8 2023, 9:58 AM
dcaro changed the status of subtask T334587: [builds-api] Add triggering support from Open to Stalled.
dcaro changed the status of subtask T334585: [cookbooks.wmcs.toolforge.component.deploy] Add secrets support when deploying from Open to In Progress.Aug 8 2023, 12:49 PM
dcaro closed subtask T334585: [cookbooks.wmcs.toolforge.component.deploy] Add secrets support when deploying as Resolved.Aug 16 2023, 9:50 AM
dcaro changed the status of subtask T342077: Toolforge beyond build service from Open to In Progress.Sep 1 2023, 12:55 PM
aborrero closed subtask T342853: toolforge lima-kilo: introduce harbor support as Resolved.Sep 8 2023, 10:37 AM
dcaro changed the status of subtask T327057: [toolforge] repositories move to gitlab from Open to In Progress.Sep 11 2023, 11:11 AM
dcaro changed the status of subtask T304532: buildservice: migrate to helmfile from of kustomize from Open to In Progress.
Slst2020 changed the status of subtask T325799: [tbs] User story - I can use multiple language stacks for my application from Open to In Progress.Sep 11 2023, 1:25 PM
dcaro closed subtask T266283: Figure out how buildpack versioning is supposed to work as Resolved.Sep 13 2023, 12:02 PM
aborrero added a subtask: T346311: [builds-api] Harbor secret is public if using env vars, move it to a Secret resource.Sep 14 2023, 9:23 AM
dcaro closed subtask T346311: [builds-api] Harbor secret is public if using env vars, move it to a Secret resource as Resolved.Sep 14 2023, 10:46 AM
dcaro closed subtask T334081: [tbs] Define what NFS access to enable and how users will interact with it as Resolved.Sep 26 2023, 10:24 AM
dcaro closed subtask T304532: buildservice: migrate to helmfile from of kustomize as Resolved.Sep 26 2023, 2:25 PM
Slst2020 changed the status of subtask T342077: Toolforge beyond build service from In Progress to Stalled.Sep 28 2023, 8:54 AM
dcaro closed subtask T327057: [toolforge] repositories move to gitlab as Resolved.Oct 9 2023, 1:49 PM
Slst2020 closed subtask T342077: Toolforge beyond build service as Resolved.Oct 12 2023, 1:38 PM
Slst2020 changed the status of subtask T325799: [tbs] User story - I can use multiple language stacks for my application from In Progress to Open.Oct 13 2023, 10:20 AM
Slst2020 closed subtask T325799: [tbs] User story - I can use multiple language stacks for my application as Resolved.Oct 13 2023, 11:18 AM
dcaro closed subtask T334590: [buildservice] Create a buildservice API and move any logic from the client to it as Resolved.Oct 24 2023, 9:17 AM
dcaro changed the status of subtask T338142: [toolforge-envvars.api,toolforge-build.api] Support using custom environment variables at build time from Open to In Progress.Dec 11 2023, 2:42 PM
taavi closed subtask T267374: [tbs.beta] Create a toolforge build service beta release as Resolved.Dec 12 2023, 7:17 PM
dcaro closed subtask T338142: [toolforge-envvars.api,toolforge-build.api] Support using custom environment variables at build time as Resolved.Dec 13 2023, 1:56 PM
fnegri edited projects, added Goal, cloud-services-team (FY2023/2024-Q3-Q4); removed cloud-services-team.Feb 1 2024, 11:20 AM
Count_Count subscribed.Feb 14 2024, 6:31 AM
dcaro changed the status of subtask T341065: [builds-api,components-api] Automatically deploy the webservice when the image is built from Stalled to In Progress.Feb 20 2024, 2:29 PM
dcaro edited projects, added Toolforge; removed Toolforge Build Service.Mar 5 2024, 1:19 PM
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
dcaro changed the status of subtask T341065: [builds-api,components-api] Automatically deploy the webservice when the image is built from In Progress to Stalled.Mar 5 2024, 2:40 PM
dcaro renamed this task from [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs to [Epic,builds-api,orchestrator,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs.Mar 5 2024, 3:55 PM
Slst2020 changed the status of subtask T341065: [builds-api,components-api] Automatically deploy the webservice when the image is built from Duplicate to Resolved.Apr 11 2024, 3:30 PM
dcaro renamed this task from [Epic,builds-api,orchestrator,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs to [Epic,builds-api,component-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs.Apr 16 2024, 12:27 PM
dcaro renamed this task from [Epic,builds-api,component-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs to [Epic,builds-api,components-api,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build packs.
dcaro added a subtask: T368602: [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment.Jul 1 2024, 1:59 PM
dcaro added a subtask: T362051: [components-api] First iteration of the component API.Jul 3 2024, 8:58 AM
dcaro changed the status of subtask T368602: [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment from Open to In Progress.Jul 15 2024, 12:41 PM
fnegri edited projects, added cloud-services-team (FY2024/2025-Q1-Q2); removed cloud-services-team (FY2023/2024-Q3-Q4).Aug 19 2024, 9:19 AM
dcaro closed subtask T368602: [Hypothesis] WE6.3.2 Create "standard" tool (Sample Complex Tool, SCT) to measure the number of steps for a deployment as Resolved.Tue, Sep 10, 9:19 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits