Assess GitLab Container Registry as a default for container build processes
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	• brennen
	May 4 2022, 4:29 AM

Description

Background

We're looking at this during the Release-Engineering-Team (GitLab-a-thon 🦊) sprint as a potential default for building / publishing images from GitLab CI. In particular, the docs mention per-job registry users and tokens and make this seem like an intended workflow. After some discussion, we're leaning towards the idea that this would make sense:

As an intermediary publication target for images to later be imported into the production registry.
For caching purposes.
Explicitly not as a permanent / guaranteed nonvolatile store of images.

Notes

Upstream documentation here:

Our omnibus installation currently has the registry disabled instance-wide with registry['enable'] = false in gitlab.rb. This is modules/gitlab/templates/gitlab.rb.erb in ops/puppet.

The registry defaults to local filesystem storage in /var/opt/gitlab/gitlab-rails/shared/registry, but other backends are supported:

Driver	Description
filesystem	Uses a path on the local file system
azure	Microsoft Azure Blob Storage
gcs	Google Cloud Storage
s3	Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct s3 Permission Scopes.
swift	OpenStack Swift Object Storage
oss	Aliyun OSS

There's not currently a way to impose a storage quota.

Questions

A starting list:

What would it take to do storage correctly?
- As I understand it, docker-registry.wikimedia.org is backed by Swift. Could/should we do the same here?
- Alternatively, would filesystem storage be good enough?
  - If so, how much space would we need?
- Is it actually safe to assume we wouldn't need to handle backups here?
How would we handle garbage collection?
- There's a command for this, but note: "You must set the Registry to read-only mode. Running garbage collection causes downtime for the Container Registry."
- "Retention policies within your object storage provider, such as Amazon s3 Lifecycle, may prevent objects from being properly deleted."
- Deleting images from the existing registry is tricky; would the same be true if we used Swift for storage here?
  - https://wikitech.wikimedia.org/wiki/Docker-registry#Deleting_images
  - T242604: Remove obsoleted docker images

A lot of these are questions for serviceops folks.

Details

	Subject	Repo	Branch	Lines +/-
	GitLab: enable container registry	operations/puppet	production	+35 -3

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Declined		None	T308080 Replicate select published images from GitLab container registry to WMF prod registry
Stalled	Feature	None	T304845 gitlab: consider enabling docker container registry
Resolved		dduvall	T301168 Migrate Blubber project to GitLab
Resolved		dduvall	T307536 Build and Run Blubber test variant on GitLab untrusted runners
Resolved		• brennen	T307537 Assess GitLab Container Registry as a default for container build processes

Event Timeline

• brennen triaged this task as Medium priority.May 4 2022, 4:29 AM

• brennen created this task.

• brennen added a parent task: T307536: Build and Run Blubber test variant on GitLab untrusted runners.

Ottomata awarded a token.May 4 2022, 1:23 PM

thcipriani moved this task from Backlog to Ready on the Release-Engineering-Team (GitLab-a-thon 🦊) board.May 4 2022, 4:42 PM

• brennen changed the task status from Open to In Progress.May 10 2022, 5:27 PM

• brennen raised the priority of this task from Medium to High.

• brennen updated the task description. (Show Details)

• brennen moved this task from Ready to In progress on the Release-Engineering-Team (GitLab-a-thon 🦊) board.

@Jelto, @Dzahn - we'd appreciate your thoughts, particularly on the storage questions outlined in the description.

Dzahn added a project: serviceops.May 10 2022, 6:55 PM

dancy updated the task description. (Show Details)May 10 2022, 6:58 PM

For "docker-registry.wikimedia.org is backed by Swift. Could/should we do the same here?" I would like to bring this up in the serviceops meeting and ask @JMeybohm and others

Change 790778 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/puppet@production] WIP: GitLab: enable container registry (experimental)

https://gerrit.wikimedia.org/r/790778

gerritbot added a project: Patch-For-Review.May 10 2022, 11:57 PM

For "docker-registry.wikimedia.org is backed by Swift. Could/should we do the same here?" I would like to bring this up in the serviceops meeting

Thanks!

In the meanwhile, we talked this through on IRC:

16:15 <+brennen> the little docker replicator approach really does seem like the most practical way forward, assuming we have somewhere for it to replicate _from_.
16:26 <+brennen> we could enable the registry but require people to turn it on: https://docs.gitlab.com/ee/administration/packages/container_registry.html#disable-container-registry-for-new-projects-site-wide
16:28 <dduvall> brennen: ooh. would turning it on require our approval?
16:29 <dduvall> or does it just require project owners to know where to look?
16:33 <+brennen> i think it just requires project owners to know where to look.
16:34 <+brennen> we could add it to the list of project settings that get auto-disabled by https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/configure-projects and actually get that running in a scheduled job.

So we're looking at enabling the registry on an experimental basis to make progress on T308080. See above patch.

Any serious use of it will obviously require answering the storage questions.

• brennen renamed this task from Assess GitLab-provided docker container registry as a default for docker-in-docker build processes to Assess GitLab Container Registry as a default for container build processes.May 11 2022, 11:30 PM

We enabled the container registry on the GitLab test instance in devtools.

We confirmed registries aren't enabled for newly created projects, tested enabling the registry on a new project, and tested building / pushing an image to a registry from a pipeline. All work.

Mentioned in SAL (#wikimedia-releng) [2022-05-12T00:46:03Z] <brennen> gitlab: disabling container registries on all existing projects (T307537)

configure-projects: disable container registries by default

akosiaris subscribed.May 17 2022, 12:57 PM

• jnuche subscribed.May 18 2022, 7:14 AM

LSobanski subscribed.May 18 2022, 10:09 AM

To save splitting discussion here, I'm going to roll this into T304845: gitlab: consider enabling docker container registry. We've assessed the GL-provided registry for this purpose and concluded that:

It'd be nice to have. There are useful features here.
It's not a blocker to our current plan for publishing images to the Wikimedia production registry, although it would be useful for caching and other purposes.
If we enable it for this narrow use case, it will most likely be used for other purposes as well.

(Please correct me if I've missed any important details here.)

• brennen closed this task as Resolved.May 25 2022, 10:04 PM

• brennen claimed this task.

• brennen mentioned this in T304845: gitlab: consider enabling docker container registry.May 25 2022, 10:08 PM

• brennen mentioned this in Blog Post: GitLab-a-thon!.May 31 2022, 6:30 PM

Change 790778 abandoned by Brennen Bearnes:

[operations/puppet@production] GitLab: enable container registry

Reason:

https://gerrit.wikimedia.org/r/790778

Maintenance_bot removed a project: Patch-For-Review.Sep 20 2022, 9:30 PM

brennen opened https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/35

configure-projects: default registry access to allowed