We probably want shellcheck for all the bash scripts we have.
And then it would be good if we could run something like pack create-builder toolforge-builder:testing --config builder.toml --verbose to verify all the configuration is compatible. This would pull the base images and build a new one.
@hashar is it possible for normal jobs to pull/build docker images? We don't actually need to use the images, but verify that pack can build one.
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| integration/config | master | +20 -0 | Add shellcheck Jenkins job and configure in Zuul | |
| integration/config | master | +31 -0 | Add shellcheck image |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Stalled | LucasWerkmeister | T320140 Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes | |||
| Stalled | matmarex | T319707 Migrate dtcheck from Toolforge GridEngine to Toolforge Kubernetes | |||
| Open | None | T320062 Migrate steve-adder from Toolforge GridEngine to Toolforge Kubernetes | |||
| Open | None | T320011 Migrate rfa-voting-history from Toolforge GridEngine to Toolforge Kubernetes | |||
| Open | dcaro | T194332 [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs | |||
| In Progress | dcaro | T267374 [tbs.beta] Create a toolforge build service beta release | |||
| Open | None | T265685 [tbs] Set up CI for cloud/toolforge/buildpacks repository | |||
| Declined | dduvall | T266081 Evaluate how we might run buildpack's `pack` in CI | |||
| Open | None | T283724 Tracking task to support running arbitrary Docker images |
Change 634568 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add shellcheck image
Change 634569 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Add shellcheck Jenkins job and configure in Zuul
Change 634568 merged by jenkins-bot:
[integration/config@master] Add shellcheck image
Change 634569 merged by jenkins-bot:
[integration/config@master] Add shellcheck Jenkins job and configure in Zuul
is it possible for normal jobs to pull/build docker images? We don't actually need to use the images, but verify that pack can build one.
Most of our jobs are Docker based now, and the only way to allow a Docker-based job to build images would be to give it access to the running daemon by bind mounting /var/run/docker.sock. That comes with security risks, however, as it would give the containerized process very broad access to Docker processes on the host. I don't know a ton about webpack builder configuration, but if there are any sort of hooks for scripting or shelling out, this might not be acceptable.
Another route to take would be for us to add webpack dependencies to CI hosts and write a one-off JJB job for it.
Does webpack depend on docker build for building images, or does it allow for other tooling? I know there are some other tools for building images without a running Docker daemon—and so they are easily run inside containers—but I'm not sure if that would be acceptable for your case, as it sounds like you're wanting to test the build process itself.
My bad, I should have explicitly stated that this is about cloud native buildpacks: https://buildpacks.io/ and pack is its CLI tool: https://buildpacks.io/docs/tools/pack/
Right. As far as I can tell, pack will just run over the TOML files in the git repository, but I doubt anyone has audited it for any type of such holes. It also downloads the docker images we specify and checks that various labels all match each other.
Another route to take would be for us to add webpack dependencies to CI hosts and write a one-off JJB job for it.
pack is a golang project that they distribute pre-built binaries for. I haven't yet looked into what it would take to package it.
Does webpack depend on docker build for building images, or does it allow for other tooling? I know there are some other tools for building images without a running Docker daemon—and so they are easily run inside containers—but I'm not sure if that would be acceptable for your case, as it sounds like you're wanting to test the build process itself.
It's currently docker specific unfortunately: https://github.com/buildpacks/pack/issues/564
Very cool. I'll definitely be jumping into that rabbit hole later. :) I do remember reading about buildpack a while back in the context of experimental BuildKit frontends. I'm planning on experimenting with the latter soon as a means to a more scalable image-build infrastructure, so maybe there will be opportunity to collaborate with WMCS there.
pack is a golang project that they distribute pre-built binaries for. I haven't yet looked into what it would take to package it.
I'd be willing to give it a... go. (Ugh.) We used to package Blubber (another golang program) before Blubberoid took over, and IIRC it wasn't too difficult.
After thinking about this a bit more, I'm not sure packaging pack and running it on a bare agent will give us a better situation as it would still have direct access to Docker and more. I can't think of an obvious path here other than auditing pack more closely and determining whether it has anything that could be used for shell/command injection.
The architecture looks fairly straightforward at the surface as does the builder config reference. Is create-builder the only subcommand the job would run?
I'm deep enough in the hole that when I see the word "pack" it's only about buildpacks now :P FWIW https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_push_to_deploy is the overall project which is going to use buildpacks.
Yeah, that makes sense.
The architecture looks fairly straightforward at the surface as does the builder config reference. Is create-builder the only subcommand the job would run?
As a nice to have, I'd also want it to run pack inspect-builder [image] to print out a representation of the builder.
I think that would be it for now though. Any further CI needs (e.g. running the builder against a sample repository) would probably be part of our larger CD discussion: T265684: Figure out CI solution for building buildpack-based images for Toolforge.
:) That looks like a really fun/challenging/valuable project. I'll definitely be keeping tabs on it.
As a nice to have, I'd also want it to run pack inspect-builder [image] to print out a representation of the builder.
Right on. I'll open a task for someone in Release-Engineering-Team (and maybe Security-Team?) to see how we might build a buildpack image and safely give it access to Docker in the CI cluster.
> I think that would be it for now though. Any further CI needs (e.g. running the builder against a sample repository) would probably be part of our larger CD discussion: T265684: Figure out CI solution for building buildpack-based images for Toolforge.
Oh! Well, if you have questions about Argo related projects, I evaluated it pretty thoroughly to be our new CI/CD system before GitLab took center stage on that. I didn't evaluate Argo CD exactly (it was an Argo Workflow + Argo Events system) but they're closely related. See here for the remnants of that work in case any of it is useful.
https://www.mediawiki.org/wiki/Draft:Wikimedia_Release_Engineering_Team/Seakeeper_proposal_(FY2019-20Q4_rework)
https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Seakeeper_proposal
I've done this in toolsbeta by using tekton pipelines. The way it works is using an unprivileged pod that uses a new container for each phase of the buildpacks lifecycle so that the build doesn't have access to the image push secrets. That could be translated to gitlab if people want to take a look at it for T291017. I've pushed the manifests which are mostly edited from upstream tekton stuff to Github here https://github.com/toolforge/buildservice/blob/main/admin-manifests/deploy/task-combined.yaml
That version also did require a few docker capabilities (also FYI) https://github.com/toolforge/buildservice/blob/main/admin-manifests/deploy/psp.yaml#L54-L65. It needs them to run a chown on the volume data. Buildpack builders always expect one consistent UID that is typically defined by the builder, so if you use Paketo or any publicly known buildpack builders, probably UID 1000. If you use the Toolforge python one that @Legoktm made, it's UID 61312. Depending on how you get the code and what owner the code has, you may or may not need those capabilities for the container.