Page MenuHomePhabricator

[builds-api] Add triggering support
Closed, DuplicatePublic

Description

To be able to be a proper push-to-deploy, we have to allow triggering the build when a change happens on the original git repository.

This task is to brainstorm ideas on how to do that.

Webhook

This allows a generic way of triggering the build by using a webhook, potentially with a token/generated url to prevent spamming.

Git pull + diff trigger (not full pull/diff, just the idea)

This would be some kind of daemon that periodically pulls the repositories and checks if they changed, and triggers a build if so.

Add you ideas here

Event Timeline

Yes thanks! I wanted to focus first on the user side of things for starters, and leaving the "how" to after we decided "what" xd

Based on the few tools I've migrated to buildpacks, I think the most important features I would like are:

  • Ability to trigger builds when I push a commit to a Git repository. Most of those repos are in Wikimedia GitLab so for that I'd like it to be as simple to set up as possible, for example if it'll be webhook based I'd prefer if that was automated for repos under /toolforge-repos.
  • Ability to restart the webservice or any continuous jobs when a build completes. For cron jobs it's fine to let them finish with the old image, but I don't want to restart any long-running jobs manually.
  • Some tools need database migrations. For those I'd like to automatically trigger an one-off job with a specific command after a build has completed. Preferrably the automatic restarts for continuous jobs would only happen after these complete.

All these look great :)

Ability to restart the webservice or any continuous jobs when a build completes. For cron jobs it's fine to let them finish with the old image, but I don't want to restart any long-running jobs manually.

This should probably be an opt-in setup right? I think that a lot of users would not want to restart webservices with the new code (ex. if there's a manual intervention needed)

Whatever works for me, as long as it's possible to auto restart containers (since that's pretty important for the "push-to-deploy" part) and it's clearly documented.

One more thing that's worth stating out loud: I want the ability to keep the toolforge-jobs configuration for my tools in the Git repository itself and have it automatically applied on push.

One more thing that's worth stating out loud: I want the ability to keep the toolforge-jobs configuration for my tools in the Git repository itself and have it automatically applied on push.

This seems unrelated to the triggering, can you open a new task for it? Probably on the toolforge jobs side of things.

dcaro renamed this task from [buildservice] Add triggering support to [builds-api] Add triggering support.Jul 4 2023, 1:11 PM

Ability to restart the webservice or any continuous jobs when a build completes. For cron jobs it's fine to let them finish with the old image, but I don't want to restart any long-running jobs manually.

Created T341065: [builds-api,components-api] Automatically deploy the webservice when the image is built for this

Some tools need database migrations. For those I'd like to automatically trigger an one-off job with a specific command after a build has completed. Preferrably the automatic restarts for continuous jobs would only happen after these complete.

This one is interesting, I'm thinking that might be worth re-thinking if toolforge build is the place for all this, instead of toolforge app deploy or toolforge deploy to handle the whole pipeline

Some tools need database migrations. For those I'd like to automatically trigger an one-off job with a specific command after a build has completed. Preferrably the automatic restarts for continuous jobs would only happen after these complete.

This one is interesting, I'm thinking that might be worth re-thinking if toolforge build is the place for all this, instead of toolforge app deploy or toolforge deploy to handle the whole pipeline

The one-off job would correspond to the release job type in the Heroku procfile, I think. I like this approach, it's simple and user-friendly.

release: python manage.py migrate
web: gunicorn myproject.wsgi

https://devcenter.heroku.com/articles/release-phase

Some tools need database migrations. For those I'd like to automatically trigger an one-off job with a specific command after a build has completed. Preferrably the automatic restarts for continuous jobs would only happen after these complete.

This one is interesting, I'm thinking that might be worth re-thinking if toolforge build is the place for all this, instead of toolforge app deploy or toolforge deploy to handle the whole pipeline

The one-off job would correspond to the release job type in the Heroku procfile, I think. I like this approach, it's simple and user-friendly.

release: python manage.py migrate
web: gunicorn myproject.wsgi

https://devcenter.heroku.com/articles/release-phase

That sounds good to me :)

dcaro removed dcaro as the assignee of this task.Jul 4 2023, 3:35 PM
dcaro changed the task status from Open to Stalled.Aug 8 2023, 9:59 AM
  • Ability to trigger builds when I push a commit to a Git repository. Most of those repos are in Wikimedia GitLab so for that I'd like it to be as simple to set up as possible, for example if it'll be webhook based I'd prefer if that was automated for repos under /toolforge-repos.

Based on a #wikimedia-cloud discussion today: Some people will only want to deploy a new version if it passes the automated tests. Unless we think the tests should be run inside the image build process (which taavi and I are both inclined against), that means we probably want the ability to trigger builds based on GitLab CI. I imagine this could be triggered by a successful pipeline completion, or explicitly by a job inside the pipeline.