Page MenuHomePhabricator

[Deployment pipeline] Support Toolforge?
Closed, DeclinedPublic

Description

Could the Wikimedia Deployment pipeline support deploying Toolforge tools?

Background: I’m an external volunteer developer working on some minor tool running on Toolforge Kubernetes. The tool happens to be written in Go, a compiled language. To deploy my tool, I’m currently cross-compiling my binaries for Linux on a Mac laptop (a random uncontrolled environment), then I use scp to copy the binary to the Toolforge bastion server, and finally I manually install the binary into the tool’s home directory. Nobody else can reproduce my compiles, or even just tell what version of which software went into my artifacts. From a release engineering (and also security) perspective, my current “process” is the exact opposite of good industry practice. Is there a better way? Glancing over the documentation for the Wikimedia Deployment pipeline, it sounds almost perfect, but it doesn’t seem to be available for Toolforge. Apologies if this has been discussed already, or if there’s other plans.

This came up, a little off-topic, in T277457#6925568 in reply to a question by @aborrero. Filing this as a separate ticket to split off the conversation.

Event Timeline

Yes, the goal is basically to be able to control the builders available in order to reduce the amount of duplication created by custom images. The deployment pipeline currently has the ability to publish images, as I understand, but before it goes into Toolforge, it should be restricted to defined builders, implementing the buildpack standard in it. That sounds like enough of a fork of pipelinelib that I don't think anyone is planning on using it directly so far.

I haven't synced up on the latest of what's being done there, but I imagine it's likely to be implemented as an add-on to the Foundation's future CI/git offerings or in a separate CI in Toolforge itself to prevent leaks.

hashar subscribed.

Per the last comments. Seems like Toolforge will have a dedicated solution (based on Tekton? T265684#7080312).

Per the last comments. Seems like Toolforge will have a dedicated solution (based on Tekton? T265684#7080312).

I agree with declining this task since it's a bit outdated but I still think the basic idea is great. I would love to support a continuous delivery workflow to toolforge that uses the same mechanisms as continuous delivery to wiki-prod