Page MenuHomePhabricator

Establish image trust system for GitLab/Blubber
Closed, DeclinedPublic8 Estimated Story Points

Description

How do we decide whether an image is a trustworthy artifact for production use?

Currently we have a trusted registry, but we could trust individual artifacts.

Scope of this task

  • Outline potential future problems with the trusted registry model
  • Talk through alternative models with ServiceOpsen
  • Come to an agreement about workflow

Event Timeline

That's related to T295481.

Trust is most probably established by using dedicated and more secure runners. So this builds will need to use the Trusted Runners (or other, Docker build-specific runners).

Things to look investigate:

  • ContentTrustCLI
  • Concept of using signing to establish trust
  • CNCF paper about image signing

We (@brennen and others but I don't want to subscribe everyone to the task, Ahmon, Jeena, Jaime) talked a bit more about signing this morning in a breakout following our GitLab-a-thon checkin. It sounds like signing would likely provide additional verification mechanisms to the pipeline which we don't have at all in our current system, but it is very much out of scope for this sprint.

I just want to add briefly that signing on its own would not provide total security. What it could provide is a chain of verifiable attestations, that a container running in production is using an image that was looked at by a number of trusted parties (people and subsystems, trusted in aggregate as a chain, not trusted independently), that each party's signature is real, and that its contents are the same as when each party signed off. Think of signing as a tamper proof shipping label that is scanned at multiple locations with additional metadata attached along the way—it doesn't inherently preclude the contents from being dangerous, it just provides some degree of assurance that the box that reaches its destination is the same box (and contents) that was labeled and scanned at its origin.

So what could our chain of "verifiable attestation" look like and what kind of trust would each attestation provide to the eventual runtime system (WMF prod k8s)? Here's one possibility that I'm imagining. But again, this goes way outside the scope of this sprint and possibly out of scope for this task. I just want to leave it here for reference.

  • party developer attests to contents of their commit by signing commit and pushing for review
  • party reviewer attests to acceptability of commit(s) by signing all commits in merge request and merging
  • party gitlab pipeline attests to validity of commit signatures and that the image is a result of those commits by signing image and publishing
  • party deployer attests to validity of image signature (and implicitly the source commits) and by signing image and deploying
  • party k8s attests to validity of deployer signature (and implicitly the image and source commits) by running the image

Out of scope for GitLab migration. We have trusted runners. They will be able to publish to the registry using JWT. (See T308501: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token)