Page MenuHomePhabricator

[components-api] First iteration of the component API
Closed, ResolvedPublic

Description

Original doc: https://docs.google.com/document/d/1y7cIX3oiqOH8hEuPhSEuqWx-yElxTq9Ga8qJRvqAJjY/edit
Current working doc: https://docs.google.com/document/d/116eZZcEPpm-YCV25qjK3Iv8LFj2OBA4wkwYMTeJcYWo/edit#heading=h.uhbe07clzg9n

This is an epic task to keep track of all the subtasks needed for a first iteration of the Toolforge Component API.

After this iteration, the user should be able to:

  • Define build configuration and trigger the build with one or more triggers (TBD) in a yaml file and persist it in toolforge
  • Define any kind of job (webservice included) in that yaml file that will be created/restarted when depolyed (toolforge deploy)
  • Redeploy automatically components (jobs) when their images are built by a trigger
  • Specify the dependencies between the jobs to deploy in a specific order
  • Specify ports to expose to other components defined in the yaml (internal services)

This will be open only as 'preview' for users.

Potential example:

$ cat mytool.yaml
config-version: 0.1

metadata:
 name: Toolhunt
 description: A web application for editing Toolhub records in a fun and easy way.
 version: v1.0.0

components:
 toolhunt-ui:
   description: a vue.js frontend
   type: webservice
   build:
     repository: https://github.com/wikimedia/toolhunt-ui
     triggers:
       - webhook
   wait-for:
      - toolhunt-api

 toolhunt-api:
   description: a python-flask api
   type: continuous
   build:
     repository: https://github.com/wikimedia/toolhunt
   run:
     port: 5000

 toolhunt-flower:
   description: dashboard for monitoring and managing celery
   type: continuous
   build:
     reuse-from: toolhunt-api
   run:
     port: 5555
     command: flower
   wait-for:
      - toolhunt-api


$ toolforge deploy mytool.yaml  # or toolforge component deploy mytool.yaml if we don't have a single cli yet
Starting deploy for toolhunt v1.0.0...
Building images:
   [✅] toolhunt-ui
   [✅] toolhunt-api
Deploying components:
   [✅] toolhunt-api
   [✅] toolhunt-ui (waiting for toolhunt-api)
   [✅] toolhunt-flower (waiting for toolhunt-api)
All components deployed



$ toolforge status  # very speculative, might be toolforge components status if we don't have a single cli yet
Current components:  # similar to `toolforge jobs list` with deployment info
---------------------------------------------------------------------------
name            status     build-id         deploy time  ports   type
---------------------------------------------------------------------------
toolhunt-ui     RUNNING    build-id-12345   12:15:00     -       webservice
toolhunt-api    RUNNING    build-id-12344   12:12:00     5000    continuous
toolhunt-flower ERROR      build-id-12344   12:12:00     5555    continuous


Last triggered builds:  # similar to `toolforge build list` with deployment info
   2024-04-04 12:11:00 UTC  OK  build-id-12345   toolhunt-ui    triggerd by webhook   some more information
   2024-04-04 12:00:00 UTC  OK  build-id-12344   toolhunt-api   triggerd manually    some more information


Last deployments:
   toolhunt-ui
      2024-04-04 12:15:00 UT build-id-12345 Current
      2024-03-02 12:00:00 UT build-id-12342 OK

   toolhunt-api
      2024-04-04 12:15:00 UT build-id-12344 Current
      2024-03-02 12:00:00 UT build-id-12222 FAILED

   toolhunt-flower (image: toolhunt-api)
      2024-04-04 12:15:00 UT build-id-12344 Current
      2024-03-02 12:00:00 UT build-id-12322 OK

The details on how that configuration file looks like and the command line interface are to be defined.

Direct subtasks:

  1. T362070: [components-api] Get a minimal version of the config with build-only data
  2. T362069: [components-api] Get a skeleton of API webservice and implement `/tool/<toolname>/deploy` with single continuous job deployment only
  3. T362066: [components-api] Develop the webhook mechanism to trigger a deployment
  4. T362075: [components-api] add order to the components deployment
  5. T348755: [jobs-api,webservice] Run webservices via the jobs framework
  6. T362077: [components-api] Add webservice support
  7. T362072: [components-api] Add support for port/healthcheck for continuous jobs in tool config/depolyment
  8. T362076: [components-api] Add support for pre-built images (ex. python3.11, to refine)
  9. T362071: [components-api] Add source polling build trigger

Related Objects

StatusSubtypeAssignedTask
ResolvedLucasWerkmeister
Resolvedmatmarex
ResolvedLegoktm
ResolvedLegoktm
In Progressdcaro
Resolveddcaro
Resolveddcaro
ResolvedRaymond_Ndibe
Resolveddcaro
OpenNone
ResolvedRaymond_Ndibe
Resolveddcaro
Resolveddcaro
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
Resolved Chuckonwumelu
Duplicate Chuckonwumelu
Resolvedtaavi
Resolveddcaro
Resolveddcaro
ResolvedFeaturedcaro
Resolveddcaro
OpenNone
OpenNone
ResolvedDamianZaremba
OpenNone
StalledRaymond_Ndibe
OpenBUG REPORTNone
Opendcaro
In Progresskomla
Resolveddcaro
Resolveddcaro
Resolveddcaro
Opendcaro
ResolvedRaymond_Ndibe
Resolveddcaro
Opendcaro
Resolveddcaro
Opendcaro
In ProgressFeatureRaymond_Ndibe
ResolvedFeatureRaymond_Ndibe
ResolvedFeatureRaymond_Ndibe
OpenRaymond_Ndibe
Resolveddcaro
ResolvedRaymond_Ndibe
In ProgressRaymond_Ndibe
OpenRaymond_Ndibe
ResolvedRaymond_Ndibe
DeclinedNone
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
ResolvedRaymond_Ndibe
OpenRaymond_Ndibe
Resolvedtaavi
ResolvedRaymond_Ndibe
In ProgressRaymond_Ndibe
OpenNone
ResolvedFeatureRaymond_Ndibe
ResolvedFeatureRaymond_Ndibe
InvalidRaymond_Ndibe
InvalidRaymond_Ndibe
DuplicateRaymond_Ndibe
In ProgressRaymond_Ndibe
InvalidRaymond_Ndibe
OpenRaymond_Ndibe
OpenRaymond_Ndibe
OpenRaymond_Ndibe
In ProgressRaymond_Ndibe
DuplicateRaymond_Ndibe
In ProgressRaymond_Ndibe
ResolvedRaymond_Ndibe
Resolveddcaro
ResolvedSlst2020
Resolveddcaro
ResolvedSlst2020
ResolvedSlst2020
Resolveddcaro
ResolvedSlst2020
OpenNone
Resolved aborrero
Resolveddcaro
Resolveddcaro
Resolveddcaro
Resolveddcaro
Invaliddcaro
Resolveddcaro
ResolvedSlst2020
Resolveddcaro
ResolvedSlst2020
Resolveddcaro
Resolveddcaro
Resolveddcaro
Resolveddcaro
ResolvedNone
Resolveddcaro

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dcaro triaged this task as High priority.Apr 8 2024, 10:12 AM
dcaro renamed this task from [component] First iteration of the component API to [component-api] First iteration of the component API.Apr 8 2024, 12:47 PM

@dcaro I edited the description of this task to reflect what we discussed in the Toolforge Monthly Meeting.

It's somewhat duplicating the information in the Phabricator Task Graph, but I think listing the main sub-tasks in the task description is more readable. In the task graph, it's hard to see which tasks are the top-level subtasks (there's an icon in the first column, but you still have the noise of all the other tasks in between).

I hope this helps, but feel free to undo my change and explore different ways to organize things.

@dcaro I edited the description of this task to reflect what we discussed in the Toolforge Monthly Meeting.

It's somewhat duplicating the information in the Phabricator Task Graph, but I think listing the main sub-tasks in the task description is more readable. In the task graph, it's hard to see which tasks are the top-level subtasks (there's an icon in the first column, but you still have the noise of all the other tasks in between).

I hope this helps, but feel free to undo my change and explore different ways to organize things.

Thanks, I agree that is more readable, but it's harder to maintain xd

dcaro renamed this task from [component-api] First iteration of the component API to [components-api] First iteration of the component API.Apr 16 2024, 12:28 PM
dcaro changed the task status from Open to In Progress.Oct 8 2024, 8:38 AM
dcaro moved this task from Ready to be worked on to Toolforge iteration 15 on the Toolforge board.
dcaro edited projects, added Toolforge (Toolforge iteration 15); removed Toolforge.
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 15) board.

This task is not useful anymore, I'll close (the subtasks are still useful, it just makes no sense anymore to address them as 'first iteration').