Page MenuHomePhabricator

Assess GitLab Container Registry as a default for container build processes
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

We're looking at this during the Release-Engineering-Team (GitLab-a-thon 🦊) sprint as a potential default for building / publishing images from GitLab CI. In particular, the docs mention per-job registry users and tokens and make this seem like an intended workflow. After some discussion, we're leaning towards the idea that this would make sense:

  1. As an intermediary publication target for images to later be imported into the production registry.
  2. For caching purposes.
  3. Explicitly not as a permanent / guaranteed nonvolatile store of images.

Notes

Upstream documentation here:

Our omnibus installation currently has the registry disabled instance-wide with registry['enable'] = false in gitlab.rb. This is modules/gitlab/templates/gitlab.rb.erb in ops/puppet.

The registry defaults to local filesystem storage in /var/opt/gitlab/gitlab-rails/shared/registry, but other backends are supported:

DriverDescription
filesystemUses a path on the local file system
azureMicrosoft Azure Blob Storage
gcsGoogle Cloud Storage
s3Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct s3 Permission Scopes.
swiftOpenStack Swift Object Storage
ossAliyun OSS

There's not currently a way to impose a storage quota.

Questions

A starting list:

  1. What would it take to do storage correctly?
    • As I understand it, docker-registry.wikimedia.org is backed by Swift. Could/should we do the same here?
    • Alternatively, would filesystem storage be good enough?
      • If so, how much space would we need?
    • Is it actually safe to assume we wouldn't need to handle backups here?
  2. How would we handle garbage collection?

A lot of these are questions for serviceops folks.

Details

ReferenceSource BranchDest BranchAuthorTitle
repos/releng/gitlab-settings!35review/brennen/revert-container-registrymainbrennenconfigure-projects: default registry access to allowed
Customize query in GitLab

Event Timeline

brennen changed the task status from Open to In Progress.May 10 2022, 5:27 PM
brennen raised the priority of this task from Medium to High.
brennen updated the task description. (Show Details)
brennen updated the task description. (Show Details)
brennen added subscribers: Jelto, Dzahn.

@Jelto, @Dzahn - we'd appreciate your thoughts, particularly on the storage questions outlined in the description.

For "docker-registry.wikimedia.org is backed by Swift. Could/should we do the same here?" I would like to bring this up in the serviceops meeting and ask @JMeybohm and others

Change 790778 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/puppet@production] WIP: GitLab: enable container registry (experimental)

https://gerrit.wikimedia.org/r/790778

For "docker-registry.wikimedia.org is backed by Swift. Could/should we do the same here?" I would like to bring this up in the serviceops meeting

Thanks!

In the meanwhile, we talked this through on IRC:

16:15 <+brennen> the little docker replicator approach really does seem like the most practical way forward, assuming we have somewhere for it to replicate _from_.
16:26 <+brennen> we could enable the registry but require people to turn it on: https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-for-new-projects-site-wide
16:28 <dduvall> brennen: ooh. would turning it on require our approval?
16:29 <dduvall> or does it just require project owners to know where to look?
16:33 <+brennen> i think it just requires project owners to know where to look.
16:34 <+brennen> we could add it to the list of project settings that get auto-disabled by https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/configure-projects and actually get that running in a scheduled job.

So we're looking at enabling the registry on an experimental basis to make progress on T308080. See above patch.

Any serious use of it will obviously require answering the storage questions.

brennen renamed this task from Assess GitLab-provided docker container registry as a default for docker-in-docker build processes to Assess GitLab Container Registry as a default for container build processes.May 11 2022, 11:30 PM
brennen changed the task status from In Progress to Stalled.May 11 2022, 11:33 PM

We enabled the container registry on the GitLab test instance in devtools.

We confirmed registries aren't enabled for newly created projects, tested enabling the registry on a new project, and tested building / pushing an image to a registry from a pipeline. All work.

Mentioned in SAL (#wikimedia-releng) [2022-05-12T00:46:03Z] <brennen> gitlab: disabling container registries on all existing projects (T307537)

To save splitting discussion here, I'm going to roll this into T304845: gitlab: consider enabling docker container registry. We've assessed the GL-provided registry for this purpose and concluded that:

  • It'd be nice to have. There are useful features here.
  • It's not a blocker to our current plan for publishing images to the Wikimedia production registry, although it would be useful for caching and other purposes.
  • If we enable it for this narrow use case, it will most likely be used for other purposes as well.

(Please correct me if I've missed any important details here.)

brennen claimed this task.

Change 790778 abandoned by Brennen Bearnes:

[operations/puppet@production] GitLab: enable container registry

Reason:

https://gerrit.wikimedia.org/r/790778