Page MenuHomePhabricator

[components-api] First iteration of the component API
Open, HighPublic

Description

Original doc: https://docs.google.com/document/d/1y7cIX3oiqOH8hEuPhSEuqWx-yElxTq9Ga8qJRvqAJjY/edit

This is an epic task to keep track of all the subtasks needed for a first iteration of the Toolforge Component API.

After this iteration, the user should be able to:

  • Define build configuration and trigger the build with one or more triggers (TBD) in a yaml file and persist it in toolforge
  • Define any kind of job (webservice included) in that yaml file that will be created/restarted when depolyed (toolforge deploy)
  • Redeploy automatically components (jobs) when their images are built by a trigger
  • Specify the dependencies between the jobs to deploy in a specific order
  • Specify ports to expose to other components defined in the yaml (internal services)

This will be open only as 'preview' for users.

Potential example:

$ cat mytool.yaml
config-version: 0.1

metadata:
 name: Toolhunt
 description: A web application for editing Toolhub records in a fun and easy way.
 version: v1.0.0

components:
 toolhunt-ui:
   description: a vue.js frontend
   type: webservice
   build:
     repository: https://github.com/wikimedia/toolhunt-ui
     triggers:
       - webhook
   wait-for:
      - toolhunt-api

 toolhunt-api:
   description: a python-flask api
   type: continuous
   build:
     repository: https://github.com/wikimedia/toolhunt
   run:
     port: 5000

 toolhunt-flower:
   description: dashboard for monitoring and managing celery
   type: continuous
   build:
     reuse-from: toolhunt-api
   run:
     port: 5555
     command: flower
   wait-for:
      - toolhunt-api


$ toolforge deploy mytool.yaml  # or toolforge component deploy mytool.yaml if we don't have a single cli yet
Starting deploy for toolhunt v1.0.0...
Building images:
   [✅] toolhunt-ui
   [✅] toolhunt-api
Deploying components:
   [✅] toolhunt-api
   [✅] toolhunt-ui (waiting for toolhunt-api)
   [✅] toolhunt-flower (waiting for toolhunt-api)
All components deployed



$ toolforge status  # very speculative, might be toolforge components status if we don't have a single cli yet
Current components:  # similar to `toolforge jobs list` with deployment info
---------------------------------------------------------------------------
name            status     build-id         deploy time  ports   type
---------------------------------------------------------------------------
toolhunt-ui     RUNNING    build-id-12345   12:15:00     -       webservice
toolhunt-api    RUNNING    build-id-12344   12:12:00     5000    continuous
toolhunt-flower ERROR      build-id-12344   12:12:00     5555    continuous


Last triggered builds:  # similar to `toolforge build list` with deployment info
   2024-04-04 12:11:00 UTC  OK  build-id-12345   toolhunt-ui    triggerd by webhook   some more information
   2024-04-04 12:00:00 UTC  OK  build-id-12344   toolhunt-api   triggerd manually    some more information


Last deployments:
   toolhunt-ui
      2024-04-04 12:15:00 UT build-id-12345 Current
      2024-03-02 12:00:00 UT build-id-12342 OK

   toolhunt-api
      2024-04-04 12:15:00 UT build-id-12344 Current
      2024-03-02 12:00:00 UT build-id-12222 FAILED

   toolhunt-flower (image: toolhunt-api)
      2024-04-04 12:15:00 UT build-id-12344 Current
      2024-03-02 12:00:00 UT build-id-12322 OK

The details on how that configuration file looks like and the command line interface are to be defined.

Direct subtasks:

  1. T362070: [components-api] Get a minimal version of the config with build-only data
  2. T362069: [components-api] Get a skeleton of API webservice and implement `/tool/<toolname>/deploy` with build-only features
  3. T362066: [components-api] Develop the webhook mechanism to trigger a deployment
  4. T362075: [components-api] add one-off, scheduled and continuous jobs support to the yaml + api
  5. T348755: [jobs-api,webservice] Run webservices via the jobs framework
  6. T362077: [components-api] Add webservice support (to refine)
  7. T362072: [components-api] Add support for non-public services
  8. T362076: [components-api] Add support for pre-built images (ex. python3.11, to refine)
  9. T362071: [components-api] Extend the list of build triggers (unrefined)

Related Objects

StatusSubtypeAssignedTask
Opendcaro
Opendcaro
Opendcaro
Opendcaro
Opendcaro
Opendcaro
OpenNone
Opendcaro
OpenFeatureRaymond_Ndibe
OpenFeatureNone
In ProgressFeatureRaymond_Ndibe
OpenRaymond_Ndibe
OpenRaymond_Ndibe
OpenRaymond_Ndibe
Resolveddcaro
In Progressdcaro
OpenNone
ResolvedRaymond_Ndibe
OpenNone
OpenNone
OpenFeatureNone
ResolvedFeatureRaymond_Ndibe
OpenNone
Opendcaro
Opendcaro
In ProgressSlst2020
In ProgressSlst2020
In ProgressSlst2020
In Progressdcaro
OpenNone
Opendcaro
Opendcaro
Opendcaro
Opendcaro

Event Timeline

dcaro triaged this task as High priority.Apr 8 2024, 10:12 AM
dcaro created this task.
dcaro renamed this task from [component] First iteration of the component API to [component-api] First iteration of the component API.Apr 8 2024, 12:47 PM

@dcaro I edited the description of this task to reflect what we discussed in the Toolforge Monthly Meeting.

It's somewhat duplicating the information in the Phabricator Task Graph, but I think listing the main sub-tasks in the task description is more readable. In the task graph, it's hard to see which tasks are the top-level subtasks (there's an icon in the first column, but you still have the noise of all the other tasks in between).

I hope this helps, but feel free to undo my change and explore different ways to organize things.

@dcaro I edited the description of this task to reflect what we discussed in the Toolforge Monthly Meeting.

It's somewhat duplicating the information in the Phabricator Task Graph, but I think listing the main sub-tasks in the task description is more readable. In the task graph, it's hard to see which tasks are the top-level subtasks (there's an icon in the first column, but you still have the noise of all the other tasks in between).

I hope this helps, but feel free to undo my change and explore different ways to organize things.

Thanks, I agree that is more readable, but it's harder to maintain xd

dcaro renamed this task from [component-api] First iteration of the component API to [components-api] First iteration of the component API.Apr 16 2024, 12:28 PM