Page MenuHomePhabricator

Figure out naming conventions for Toolforge buildpack related stuff
Open, MediumPublic

Description

buildpacks introduce a bunch of new things that we should figure out naming conventions for. Some prior discussion happened in https://gerrit.wikimedia.org/r/c/operations/docker-images/toollabs-images/+/634349 and between @Legoktm and @Bstorm on IRC.

We'll probably want to create the buildpacks.toolforge.org tool as a placeholder or info/status page at some point.

stacks

A stack is the name of an ABI implemented through docker images. Each stack will have at least a build, builder and run image, but might have more in the future for mixins. For the most part, these names should only be used in the buildpacks' config files, not by end users.

  • org.toolforge.buildpacks.stack.buster1 - A stack based on Debian Buster (note: we'd probably start with 0 for the PoC stage)
    • docker-registry.tools.wmflabs.org/toolforge-buster1-run:latest
      • Base layer that all applications run on top of
    • docker-registry.tools.wmflabs.org/toolforge-buster1-build:latest
      • Base layer the builders run on top of
    • docker-registry.tools.wmflabs.org/toolforge-buster1-builder:latest
      • The image that contains the buildpacks that is run by pack to do the actual building

Using :latest is convenient because we can automatically push security updates of apt packages to users on the same tag. Instead we put the version in the image name, so if we need to make an ABI-incompatible change, we can create a new image.

Prior art: Heroku names its stacks in the form heroku-18 or heroku-16 where 16 and 18 refer to the Ubuntu LTS version being used. We could do something similar, e.g. toolforge-10 (Buster = 10) but I think we should have our own Toolforge version in case we do need to make ABI-incompatible changes.

buildpacks

buildpacks have a name and version. Advanced users could use these names in their project.toml files.

  • org.toolforge.buildpacks.python37
    • This buildpack is fixed at python37 because it's using Debian Buster's packages. Plus we expect to have multiple Python versions. A python38 buildpack would need to be implemented differently.
  • org.toolforge.buildpacks.pip
    • This is expected to be rolling and follow the latest upstream, so there's no versioning in the name. In theory this is stack agnostic, so we can reuse the same buildpack for python37 and python38.
  • org.toolforge.buildpacks.uwsgi
    • This is (currently) pinned to a specific version mostly for stability, but we only expect one as it's also stack agnostic.

images

Each repository will result in a docker image being built. I think a naming convention like tool-$TOOL if there's only one repository for that tool, or tool-$TOOL-$SUFFIX where the operators can pick an arbitrary [A-z0-9] suffix if the tool plans on having multiple repositories/images. I don't know if that has enough conflict protection though...

Event Timeline

Bstorm triaged this task as Medium priority.Oct 19 2020, 7:07 PM
[16:46]  <    bd808> legoktm: LGTM honestly. The final image naming thing... probably could use names tied to whatever the naming for the origin git repos is I guess.
[16:47]  <    bd808> the "tool-{name}" prefix thing is what Striker does for Diffusion repos and Phab projects. It is possible to have collisions in that scheme, but it has not become an active problem yet.

On images, I think tool-$TOOL-$SUFFIX is enough collision protection if we have a reasonable auth scheme. I cannot imagine a way that a per-tool auth scheme doesn't end up as a requirement unless the remote repo is entirely controlled from within Toolforge and forces naming accordingly.

I started looking into creating a k8s deployment, I noticed that webservice creates some labels:

# Labels for all objects created by this webservice
self.webservice_labels = {
    "tools.wmflabs.org/webservice": "true",
    "tools.wmflabs.org/webservice-version": "1",
    "toolforge": "tool",
    "name": self.tool.name,
}

We need to come up with new labels due to the domain change and then also because this is not actually using webservice. (Note: we don't want to add toolforge: tool because that enables a pod preset for read/write NFS, which we don't want per @Bstorm.)

@bd808 suggested using https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/

So:

app.kubernetes.io/component: web
app.kubernetes.io/managed-by: tfb

Theoretically we could also update webservice to match this labeling system (managed-by: webservice), but that's not a constraint.