Page MenuHomePhabricator

Explicitly config buildkitd with internal DNS nameserver
Closed, ResolvedPublic

Description

(renamed from: Remove use custom network for GitLab runners and buildkitd in favor of a fixed IP; see update below)

See T308271#8225621 for background as to why this is needed. The tl;dr is that buildkitd needs to know our host's resolv.conf configuration in order to pass it along to containers that it spawns, yet buildkitd is currently running within a Docker container that uses a custom Docker network which overwrites resolv.conf with an internal Docker DNS proxy.

The custom Docker network is currently serving only one purpose which is to provide a way to consistently reference the buildkitd server (by name). To retain this consistency, we'll assign a fixed IP to the buildkitd container within the default Docker network and set a runner level environment variable for BUILDKIT_HOST that obviates hardcoding of the IP in CI jobs.

update:

Unfortunately a fixed IP cannot be allocated to containers on the default network, so we can't abandon the custom network. However, there appears to be an undocumented config field for buildkitd to configure DNS options. This will allow us to pass along DNS options which buildkitd will use when spawning build containers. \o/

Event Timeline

Well, I seem to be running into a catch 22 here: Fixed IPs are only supported when using a custom Docker network.

dduvall renamed this task from Remove use custom network for GitLab runners and buildkitd in favor of a fixed IP to Explicitly config buildkitd with internal DNS nameserver.Sep 15 2022, 9:48 PM
dduvall updated the task description. (Show Details)

I've updated the task as abandoning the custom network no longer seems feasible, and there appears to be a way to configure buildkitd with our internal nameservers for use by the OCI executor. (See new description.)

dduvall triaged this task as Medium priority.Sep 15 2022, 9:50 PM

Change 832584 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] buildkitd: Support configuration of OCI executor nameservers

https://gerrit.wikimedia.org/r/832584

Change 832584 merged by Dzahn:

[operations/puppet@production] buildkitd: Support configuration of OCI executor nameservers

https://gerrit.wikimedia.org/r/832584

Mentioned in SAL (#wikimedia-operations) [2022-09-16T17:01:45Z] <mutante> gitlab-runner*: deployed gerrit:832584 and systemctl restart buildkitd on 6 hosts for T317904

Change 832694 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] buildkitd: Add missing `--config` argument

https://gerrit.wikimedia.org/r/832694

Change 832694 merged by Dzahn:

[operations/puppet@production] buildkitd: Add missing `--config` argument

https://gerrit.wikimedia.org/r/832694

Change 833046 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] P:gitlab::runner: $nameservers parameter type should match aliased

https://gerrit.wikimedia.org/r/833046

Change 833046 merged by Dzahn:

[operations/puppet@production] P:gitlab::runner: $nameservers parameter type should match aliased

https://gerrit.wikimedia.org/r/833046