Page MenuHomePhabricator

GitLab runners: allowed_images patterns need to be loosened to include subdirectories
Closed, ResolvedPublic2 Estimated Story PointsBUG REPORT

Description

This is followup from T291978: Limit GitLab shared runners to images from Wikimedia Docker registry. Needs changes in both ops/puppet and cloud runner config.

Capturing this from IRC:

<gmodena> Hi folks, I have a CI pipeline that started failing with ERROR: The "maven:3.3.9-jdk-8" image is not present on list of allowed images. (https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-stream-enrichment/-/jobs/207010). I've seen this error in the past, but it used to go away after a retry. Has the list of allowed images permanently changed?

<gmodena> and FWIW using our internal docker image (which i should have done to begin with), triggers a similar error: https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-stream-enrichment/-/jobs/20712

<gmodena> ERROR: The "docker-registry.wikimedia.org/releng/maven-java8:1.0.0-s1" image is not present on list of allowed images:

<dancy> gmodena: https://gerrit.wikimedia.org/r/c/operations/puppet/+/724472 was merged on June 9 (last Thursday) which restricted what images can be run.

<dancy> The topic is discussed in https://phabricator.wikimedia.org/T291978.  That might be a reasonable place for you to request that maven:* be added to the trusted list.

<+brennen> gmodena, dancy: that seems like a bug - or at least the internal docker image part of it does - docker-registry.wikimedia.org images should all be allowed.  i'll look at this after current meeting.

<dduvall> maybe it requires a pattern of more depth, `docker-registry.wikimedia.org/*/*` or `docker-registry.wikimedia.org/**/*`?

<dduvall> for the allow list that is

<+brennen> yeah, i wonder
<+brennen> seems pretty likely
<+brennen> https://github.com/bmatcuk/doublestar#patterns

From the doublestar docs:

Special TermsMeaning
*matches any sequence of non-path-separators
/**/matches zero or more directories
?matches any single non-path-separator character
[class]matches any single non-path-separator character against a class of characters (see "character classes")
{alt1,...}matches a sequence of characters if one of the comma-separated alternatives matches

So it does look like we need to add /**/ to include paths with a directory separator.

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
allowed_images: Allow subdirs in image pathsrepos/releng/gitlab-cloud-runner!3brennenbrennen/allowed-images-wildcardmain
Customize query in GitLab

Event Timeline

brennen set the point value for this task to 2.
brennen added a subscriber: gmodena.

Change 805247 had a related patch set uploaded (by Brennen Bearnes; author: Brennen Bearnes):

[operations/puppet@production] gitlab_runner: Allow subdirs in image paths

https://gerrit.wikimedia.org/r/805247

Change 805247 merged by Dzahn:

[operations/puppet@production] gitlab_runner: Allow subdirs in image paths

https://gerrit.wikimedia.org/r/805247

@sbassett: With that patch merged, this should take effect once runners are re-registered. I can probably get to that after train.

@sbassett: With that patch merged, this should take effect once runners are re-registered. I can probably get to that after train.

Great, thanks!

brennen closed this task as Resolved.EditedJun 21 2022, 10:16 PM
brennen claimed this task.
brennen moved this task from Backlog to Done or Declined on the User-brennen board.

Re-registered by adding:

profile::gitlab::runner::ensure: absent

To project puppet config in gitlab-runners, then running:

dsh \
  -c \
  -m runner-1021.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1022.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1023.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1024.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1025.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1026.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1027.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1028.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1029.gitlab-runners.eqiad1.wikimedia.cloud \
  -m runner-1030.gitlab-runners.eqiad1.wikimedia.cloud \
  sudo run-puppet-agent

Followed by removing the config and running the above again. A spot check says config is as-expected. Let me know if you run into further trouble.

Hi @brennen, it seems that the issue is resolved now.

Thank you so much.

I can confirm, pipeline works again for me. Re-configuring the Cloud Runners was block because of a failing pipeline. But this works again and Cloud Runner are using the new double star config now.

I'm going to re-register Protected Runners too soon.