Page MenuHomePhabricator

Decision: Evaluate tooling for building and deploying OCI images
Open, Needs TriagePublicSpike

Description

Situation summary:

We have an increasingly lean, but still entirely custom shell script way of building our Docker Images. These script clearly overlap the functionality of existing and robust tooling for , configuring and initiating Docker Image builds through to releasing them to Dockerhub.

Evaluating our requirements and considering some of these tools will allow us to remove some unnecessary complexity in our setup, and inherit some needed/wanted functionality.

The main thing driving this consideration is in answer to the question of how we can streamline our release process such that versioning, CHANGELOG generation, and publishing/releasing is consistent and largely automatic. Using one of these existing build tools in lieu of our own custom code, has the dual benefit of giving us access to a much larger feature-set towards this end, as well as reducing our maintenance burden.

Two key options to consider:

  1. Docker Compose file with build specs + env vars (docker compose build)
    • Pro: We are already getting familiar with Docker Compose
    • Pro: Conceivably could become our standard Deploy Compose configuration
    • Con: Potentially too limited for some of our build/tagging requirements
  2. Skaffold:
    • Pro: Additonal (possibly needed) build features around dynamic tagging, etc
    • Pro: Skaffold originates as a K8 deployment tool, so has built-in the option of a step to deploy to a Kubernetes cluster. This could become a key feature in our product, even potentially the place we integrate better with Wikibase Cloud needs.
    • Con: A new tool to learn
    • Con: Somewhat more complex than Docker Compose

Additionally, we need to switch to using buildx when we move towards multi-platform builds (if we're not already using it).

Acceptance criteria:

  • We have evaluated available tooling for configuring, building, and publishing (to DockerHub), WBS Images. With consideration to at least the above two suggested tools.
  • We have decided whether to move forward with any of the options, and, if so, have created at least one follow-up ticket related to implementation.

Event Timeline

Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptTue, Jun 25, 9:25 AM
lojo_wmde renamed this task from Evaluate tooling for building and deploying OCI images to Decision: Evaluate tooling for building and deploying OCI images.Tue, Jun 25, 10:11 AM
lojo_wmde removed a project: Spike.
darthmon_wmde subscribed.

We have now a github action to publish our container images - the whole download locally and upload the images is over. This turns this ticket, as I understand it, as deprecated .

Ah, no actually this is very recent thinking and I see needs more context.

The consideration here isn't about our Github actions use, but about our build.sh and the associated variables.env file. This is the layer which would be replaced with more standard tooling for the configuration and initiation of multiple related image builds. Just like our current build.sh script, this tool would be the thing that gets ran for both local builds as well as within our Github Actions configuration.

What led to me formulating this is awareness of two things:

  1. A short-mid term need/want for automating our versioning + release process which just got longer into something shorter, closer to a "one click" proper CI workflow. The tooling I'm mentioning, along with re-organizing our repo a bit, are what I see us needing to squash the release checklist into something nearly fully automatic. Code, test, release. After this first release we're due for a round of release engineering to accommodate the next releases, including a coherent mechanism to handle versioning in a consistent way. I can unpack this more, but that is the primary reason I have been thinking about this and formulated this spike to continue exploring options. It is in what I'm seeing about to come up. We can wait for it to, and I can bring this up then...
  1. What the "Skaffold" tool mentioned above is designed for is something that in the mid-longer term, but maybe as soon as later this year, we may want to center our product on more: A build + test + deploy pipeline for a truly customised Wikibase deployment. With our increasingly slim, digestible, and standard way of building WBS Images we are very near having in place what a user needs to customise the images themselves, build, and deploy in a simple set of commands. This I think is not something someone who is just "trying out Wikibase" cares about nor will need to, but if we succeed at getting more adoption I think our users will be needing and wanting this option. Dev+Build+Test+Deply to Dockerhub+Deploy into a Kubernetes (K8) Cluster is the workflow Skaffold is designed for, and as I understand it, the exact workflow being used by Wikibase Cloud. It is possible they are already using Skaffold or similar for that (we should ask). Us creating a well thought through stack which handles this whole workflow as the larger scope of our product I think is to me a near inevitability, and the swan song of our product. I don't think we're far from it.

So, feel free to keep this closed for now, but I do think it will come up again sooner than later in a brief form as it along with the potential mono-repo work relates to how we make versioned independent releases which sync well with our Deploy configuration/s.