Page MenuHomePhabricator

Give Wikibase Suite reprobuilds™
Closed, ResolvedPublic8 Estimated Story Points

Description

Current situation:

  • At the moment, the release pipeline (a. k. a. RP) checks the head commits every time that it runs to make the packages. it usually happens that more commits have made it into the branch or the tag. That results into the packages being different depending on the time they are made. Both core code and extensions are included here.
  • Docker base images are also not referenced by explicit hashes, but tags that might get repushed to the container registries. This allows easy access to updated base images during build, but also might lead to different build results on every run

Ideal situation:

  • the first time that we run a build the hash commits of the required repositories are stored. On posterior builds those same hashes will be used.

Acceptance Criteria:

  • given a mw and wb version we release the same package no matter the moment it happens

Notes:

  • it may make sense to timebox a short investigation into Earthly (T343424) or other tools that may make our lives easier in this regard

Some idea was sketched here https://github.com/wmde/wikibase-release-pipeline/pull/422/files

Event Timeline

If a tag was used though that doesn't move, right? I mean barring a force push that redefines a tag, which I think can be considered out-of-scope.

Yes, but in the end it just depends on the way the upstream team uses git. I force pushed tags in the past ^^. So I'd prefer we record commit hashes and rely on them, this way we can be sure to always get the same thing, regardless of what people upstream do. And in case of a weird rebase upstream that removes the revision, we would get an explicit failure.

You're probably right, but I do want more information as to whether the upstream tags are stable are not, as they may be.

Like do other things upstream also depend on their stability? If so then I'd not want to reproduce a "git tag like" layer here that requires us to keep an extra and potentially extraneous bit of meta data for our builds.

I do want more information as to whether the upstream tags are stable are not

I think for some sources there are not even tags https://github.com/wmde/wikibase-release-pipeline/blob/main/versions/wmde11.env#L7

If so then I'd not want to reproduce a "git tag like" layer here that requires us to keep an extra and potentially extraneous bit of meta data for our builds.

Totally on your page, let's keep it as simple as possible

Interesting, but I wonder if we can't just create the tag for ourselves? It looks like this may have been the practice?: https://gerrit.wikimedia.org/r/admin/repos/wikidata/query/gui,tags

I think I'm still missing something here though. We can hash (🤣) it out when it gets in a sprint :)

We can hash (🤣) it out when it gets in a sprint :)

Agreed.

darthmon_wmde changed the point value for this task from 8 to 13.Sep 12 2023, 11:59 AM
roti_WMDE changed the point value for this task from 8 to 13.
roti_WMDE changed the point value for this task from 13 to 8.

I'd suggest that we refactor the build system a bit here as well. We can move almost everything of our building logic into dockerfiles, leaving only a minimal part in shell scripts. This way, the reproducibility/version locking could be implemented completely in docker files.

Example: https://github.com/wmde/wikibase-release-pipeline/pull/517

roti_WMDE renamed this task from Make wbSuite releases reproducible to Give Wikibase Suite reprobuilds™.Nov 21 2023, 2:44 PM