Change Details

== Problem == To be able to manage pipelines in the components api (build 1 + build 2 -> deploy component 1 -> deploy component 2 for example) we need some way to handle execution pipelines. === Constraints and risks === * The pipelines don't need to be huge, dozens of components would be the biggest == Decision record == In progress https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T302593_What_to_use_for_toolforge_components_api_task_execution == Options == === Option 1 === No asynchronous task processing Pros: * Easiest to implement Cons: * If the request breaks, the whole pipeline fails === Option 2 === After-request asynchronous task processing on the same thread (ex. https://fastapi.tiangolo.com/tutorial/background-tasks/) Pros: * Easy to implement and setup (no extra components/services/etc. needed) Cons: * If the service gets restarted (ex. OOM, moving to a different worker) the pipeline breaks === Option 3 === Using tekton pipelines Pros: * Already used for build service * Good pipeline support (made for it) Cons: * Relatively complex to setup * Pipelines written in tekton yaml + custom image/shell script * Needs interfacing with tekton (similar to what we do with builds service) === Option 4 === Using celery with redis Pros: * Very common pattern * Good pipeline support (made for it) * Pipelines written in the same code than the service Cons: * Needs a redis instance (we should not reuse the user-facing one for security reasons I think) === Option N === Add you option here!

== Problem == To be able to manage pipelines in the components api (build 1 + build 2 -> deploy component 1 -> deploy component 2 for example) we need some way to handle execution pipelines. === Constraints and risks === * The pipelines don't need to be huge, dozens of components would be the biggest == Decision record == In progress https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T362224_What_to_use_for_toolforge_components_api_task_execution == Options == === Option 1 === No asynchronous task processing Pros: * Easiest to implement Cons: * If the request breaks, the whole pipeline fails === Option 2 === After-request asynchronous task processing on the same thread (ex. https://fastapi.tiangolo.com/tutorial/background-tasks/) Pros: * Easy to implement and setup (no extra components/services/etc. needed) Cons: * If the service gets restarted (ex. OOM, moving to a different worker) the pipeline breaks === Option 3 === Using tekton pipelines Pros: * Already used for build service * Good pipeline support (made for it) Cons: * Relatively complex to setup * Pipelines written in tekton yaml + custom image/shell script * Needs interfacing with tekton (similar to what we do with builds service) === Option 4 === Using celery with redis Pros: * Very common pattern * Good pipeline support (made for it) * Pipelines written in the same code than the service Cons: * Needs a redis instance (we should not reuse the user-facing one for security reasons I think) === Option N === Add you option here!

== Problem == To be able to manage pipelines in the components api (build 1 + build 2 -> deploy component 1 -> deploy component 2 for example) we need some way to handle execution pipelines. === Constraints and risks === * The pipelines don't need to be huge, dozens of components would be the biggest == Decision record == In progress https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T30259362224_What_to_use_for_toolforge_components_api_task_execution == Options == === Option 1 === No asynchronous task processing Pros: * Easiest to implement Cons: * If the request breaks, the whole pipeline fails === Option 2 === After-request asynchronous task processing on the same thread (ex. https://fastapi.tiangolo.com/tutorial/background-tasks/) Pros: * Easy to implement and setup (no extra components/services/etc. needed) Cons: * If the service gets restarted (ex. OOM, moving to a different worker) the pipeline breaks === Option 3 === Using tekton pipelines Pros: * Already used for build service * Good pipeline support (made for it) Cons: * Relatively complex to setup * Pipelines written in tekton yaml + custom image/shell script * Needs interfacing with tekton (similar to what we do with builds service) === Option 4 === Using celery with redis Pros: * Very common pattern * Good pipeline support (made for it) * Pipelines written in the same code than the service Cons: * Needs a redis instance (we should not reuse the user-facing one for security reasons I think) === Option N === Add you option here!