Page MenuHomePhabricator

toolforge components - support providing ref in webhook
Open, LowPublicFeature

Description

Feature summary (what you would like to be able to do and where):

When using a deploy token (https://wikitech.wikimedia.org/wiki/Help:Toolforge/Deploy_your_tool#Triggering_a_deployment_from_your_CI_runner) it would be very useful to allow the ref to be provided as a URL parameter, overriding the ref specified on the component config.

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

I have a number of tools which are deployed from git tags, which are created from the main (default) branch by a human. This is preferable to deploying directly from main as it allows testing both multiple merged changes and pushing the tool to a staging user outside of the production user.

--ref is allowed in the build start command today and I use it (from a fabric based script) to handle building/restarting aka deploying.

Benefits (why should this be implemented?):

This would generally ease the deployment of tools (e.g. not having to maintain a production branch), additionally providing a means to roll forwards/back or deploy test releases.

For my tools it would remove a significant dependency on the bastion hosts.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
components-api: bump to 0.0.156-20250916125822-74722783repos/cloud/toolforge/toolforge-deploy!968group_203_bot_f4d95069bb2675e4ce1fff090c1c1620bump_components-apimain
Customize query in GitLab

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Note that this has a non-obvious security implication: Currently deployment tokens are relatively harmless as they only let you trigger a new build off of the currently configured repo and reference, so compromising that token doesn't directly give one access to run arbitrary code as the tool without write access to the repo itself.

But if a reference can be specified, that expands the scope of a token to basically include arbitrary code in the built container as GitLab and other forges tend to make merge requests available with a special reference even if they're not a part of the main repository.

This is a good point regarding refs, however the same is true if for example an SSH key or LDAP password was compromised.

Having other authentication opens - to be able to deal with the APIs externally - would be useful; in this case it would also avoid having to deal with creating a deployment token per tool as secrets could just be specified in the GitHub org covering all tool repos. Using such access to deal with builds directly, rather than the webhook would be perfectly acceptable for me.

I could make something work with that, but with the current tag based deployment it would mean having to keep the configs off on the side and then updating them on deploy (I can handle that in https://github.com/cluebotng/ci-toolforge-deploy).

I use a similar pattern elsewhere with ArgoCD and a 'config repo' (updated with kustomize).

Though I do think being able to just target the ref on an existing config (similar to the existing force flags) would be a lot simpler from the consumer side.

I guess one issue is that different components on the same tool might have different repositories, so you would need a way to specify also which component that ref is and such too (something that it's not supported, yet at least).

I guess one issue is that different components on the same tool might have different repositories, so you would need a way to specify also which component that ref is and such too (something that it's not supported, yet at least).

Indeed I do use multiple repos for different components within 1 tool

I mocked this up for cluebotng staging with:
https://github.com/cluebotng/component-configs [source for toolforge components configs]
https://github.com/cluebotng/ci-update-component-ref [action to run the shell script in the above to update the yaml with yq]
https://github.com/cluebotng/ci-toolforge-deploy [action to trigger the deploy and wait for it to finish]

It's working however the builds fail due to a super old go version.

I mocked this up for cluebotng staging with:
https://github.com/cluebotng/component-configs [source for toolforge components configs]
https://github.com/cluebotng/ci-update-component-ref [action to run the shell script in the above to update the yaml with yq]
https://github.com/cluebotng/ci-toolforge-deploy [action to trigger the deploy and wait for it to finish]

It's working however the builds fail due to a super old go version.

I see use_latest_versions was literally added today (yay), so actually this is now working (https://github.com/cluebotng/botng/actions/runs/16883509095).

If this keeps going ok, then I'll see about moving some other jobs over to components for non critical things.

group_203_bot_f4d95069bb2675e4ce1fff090c1c1620 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/968

components-api: bump to 0.0.156-20250916125822-74722783