Page MenuHomePhabricator

Manually provision Zuul v9 services in devtools as Docker based systemd services
Closed, ResolvedPublic

Description

Before puppetizing (see parent task), let's manually provision the Zuul services using Docker and systemd service files on an instance in the devtools project. This will help us to tease out any strangeness and system level dependencies before diving into puppet code.

Details

TitleReferenceAuthorSource BranchDest Branch
wmf: Script to dynamically generate tenant configrepos/releng/zuul/zuul!7dduvallreview/gitlab-tenant-config-scriptwmf/9.3.0
Customize query in GitLab

Event Timeline

dduvall changed the task status from Open to In Progress.Nov 15 2023, 5:03 PM
dduvall claimed this task.
dduvall created this task.

Change 975360 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] gitlab_runner: Allow rsyncd access to zuul.devtools.wmcloud.org

https://gerrit.wikimedia.org/r/975360

Change 975360 merged by Jelto:

[operations/puppet@production] gitlab_runner: Allow rsyncd access to zuul.devtools.wmcloud.org

https://gerrit.wikimedia.org/r/975360

Zuul is fully provisioned in devtools on zuul-1001.devtools.eqiad1.wikimedia.cloud. The Zuul services are currently run via systemd/docker.

dduvall@zuul-1001:~$ sudo docker ps
CONTAINER ID   IMAGE                                                                                 COMMAND                  CREATED       STATUS       PORTS            
                           NAMES
1cc11f74b591   docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-executor:wmf-9.2.0-2    "/entrypoint -d"         12 days ago   Up 12 days   0.0.0.0:873->873/
tcp, :::873->873/tcp       zuul-executor
2888c119631d   docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-fingergw:wmf-9.2.0-2    "/srv/app/venv/bin/z…"   12 days ago   Up 12 days                    
                           zuul-fingergw
7d820bbbb54a   docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-web:wmf-9.2.0-2         "/srv/app/venv/bin/z…"   12 days ago   Up 12 days   0.0.0.0:9000->900
0/tcp, :::9000->9000/tcp   zuul-web
82ea84f20970   docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-scheduler:wmf-9.2.0-2   "/srv/app/venv/bin/z…"   12 days ago   Up 12 days                    
                           zuul-scheduler
4f53cd4fbe60   0af4d0a0d72c                                                                          "/docker-entrypoint.…"   12 days ago   Up 12 days   2181/tcp, 2888/tc
p, 3888/tcp, 8080/tcp      zookeeper

Zookeeper is also running via docker, but I suspect we'll want to provision it via puppet in production. MariaDB is running on the host.

The unit files are straightforward and should map well to the service::docker operations/puppet resource when we start puppetizing (see T350814: Puppetize Zuul v9). Here's a dump of most of the relevant configuration. There are other moving parts of course, most notably the zuul-config repo and a script that dynamically generates the Zuul tenant config based on repos under given GitLab namespaces. I will add the latter to our zuul repo, so it's always present in the images.

Services

zuul-executor.service
[Unit]
Description=Docker service %N
After=docker.service
BindsTo=docker.service

[Service]
EnvironmentFile=/etc/zuul/env
ExecStartPre=-/usr/bin/docker stop %N
ExecStartPre=-/usr/bin/docker rm %N
ExecStart=/usr/bin/docker run --rm --name %N \
	--privileged \
	--user root \
	--env-file /etc/zuul/env \
	--network zuul \
	-v /etc/zuul/:/etc/zuul/ \
	-v zuul-executor:/var/lib/zuul \
	-v zuul-ssh-keys:/var/ssh \
	-v zuul-certs:/etc/ssl/zuul:ro \
	-p 873:873 \
	-v /etc/zuul/rsyncd.conf:/etc/rsyncd.conf \
	${ZUUL_EXECUTOR_IMAGE} -d
Restart=always
RestartSec=10s
NotifyAccess=all
# Let docker do its things on its own terms
TimeoutStartSec=120
TimeoutStopSec=15
SyslogIdentifier=%N

[Install]
WantedBy=multi-user.target
zuul-fingergw.service
[Unit]
Description=Docker service %N
After=docker.service
BindsTo=docker.service

[Service]
EnvironmentFile=/etc/zuul/env
ExecStartPre=-/usr/bin/docker stop %N
ExecStartPre=-/usr/bin/docker rm %N
ExecStart=/usr/bin/docker run --rm --name %N \
	--user root \
	--env-file /etc/zuul/env \
	--network zuul \
	-v /etc/zuul/:/etc/zuul/ \
	-v zuul-fingergw:/var/lib/zuul \
	-v zuul-certs:/etc/ssl/zuul:ro \
	${ZUUL_FINGERGW_IMAGE}
Restart=always
RestartSec=10s
NotifyAccess=all
# Let docker do its things on its own terms
TimeoutStartSec=120
TimeoutStopSec=15
SyslogIdentifier=%N

[Install]
WantedBy=multi-user.target
zuul-scheduler.service
[Unit]
Description=Docker service %N
After=docker.service
BindsTo=docker.service

[Service]
EnvironmentFile=/etc/zuul/env
ExecStartPre=-/usr/bin/docker stop %N
ExecStartPre=-/usr/bin/docker rm %N
ExecStart=/usr/bin/docker run --rm --name %N \
	--user root \
	--env-file /etc/zuul/env \
	--network zuul \
	-v /etc/zuul/:/etc/zuul/ \
	-v zuul-scheduler:/var/lib/zuul \
	-v zuul-ssh-keys:/var/ssh \
	-v zuul-certs:/etc/ssl/zuul:ro \
	${ZUUL_SCHEDULER_IMAGE} -d
Restart=always
RestartSec=10s
NotifyAccess=all
# Let docker do its things on its own terms
TimeoutStartSec=120
TimeoutStopSec=15
SyslogIdentifier=%N

[Install]
WantedBy=multi-user.target
zuul-web.service
[Unit]
Description=Docker service %N
After=docker.service
BindsTo=docker.service

[Service]
EnvironmentFile=/etc/zuul/env
ExecStartPre=-/usr/bin/docker stop %N
ExecStartPre=-/usr/bin/docker rm %N
ExecStart=/usr/bin/docker run --rm --name %N \
	--user root \
	--env-file /etc/zuul/env \
	--network zuul \
	-v /etc/zuul/:/etc/zuul/ \
	-v zuul-web:/var/lib/zuul \
	-v zuul-certs:/etc/ssl/zuul:ro \
	-p ${ZUUL_WEB_PORT}:${ZUUL_WEB_PORT} \
	${ZUUL_WEB_IMAGE}
Restart=always
RestartSec=10s
NotifyAccess=all
# Let docker do its things on its own terms
TimeoutStartSec=120
TimeoutStopSec=15
SyslogIdentifier=%N

[Install]
WantedBy=multi-user.target

Configuration

/etc/zuul/zuul.conf
[zookeeper]
hosts=%(ZUUL_ZOOKEEPER_HOSTS)s
tls_cert=%(ZUUL_ZOOKEEPER_TLS_CERT)s
tls_key=%(ZUUL_ZOOKEEPER_TLS_KEY)s
tls_ca=%(ZUUL_ZOOKEEPER_TLS_CA)s

[keystore]
password=%(ZUUL_KEYSTORE_PASSWORD)s

[scheduler]
tenant_config_script=%(ZUUL_TENANT_CONFIG_SCRIPT)s

[connection "gitlab"]
name=gitlab
driver=gitlab
server=%(ZUUL_GITLAB_SERVER)s
baseurl=%(ZUUL_GITLAB_BASE_URL)s
api_token_name=%(ZUUL_GITLAB_TOKEN_NAME)s
api_token=%(ZUUL_GITLAB_TOKEN)s
webhook_token=%(ZUUL_GITLAB_WEBHOOK_TOKEN)s
sshkey=%(ZUUL_GITLAB_SSH_KEY)s

[database]
dburi=mysql+pymysql://%(ZUUL_DB_USERNAME)s:%(ZUUL_DB_PASSWORD)s@%(ZUUL_DB_HOST)s/%(ZUUL_DB_DATABASE)s

[web]
listen_address=0.0.0.0
port=%(ZUUL_WEB_PORT)s
root=%(ZUUL_WEB_ROOT)s

[executor]
private_key_file=%(ZUUL_GITLAB_SSH_KEY)s
trusted_rw_paths=%(ZUUL_EXECUTOR_TRUSTED_RW_PATHS)s
hostname=%(ZUUL_EXECUTOR_HOSTNAME)s

[webclient]
url=http://zuul-web:9000
/etc/zuul/env
ZUUL_EXECUTOR_IMAGE=docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-executor:wmf-9.2.0-2
ZUUL_FINGERGW_IMAGE=docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-fingergw:wmf-9.2.0-2
ZUUL_SCHEDULER_IMAGE=docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-scheduler:wmf-9.2.0-2
ZUUL_WEB_IMAGE=docker-registry.wikimedia.org/repos/releng/zuul/zuul/wmf/zuul-web:wmf-9.2.0-2

ZUUL_ZOOKEEPER_HOSTS=zookeeper:2281
ZUUL_ZOOKEEPER_TLS_CERT=/etc/ssl/zuul/certs/client.pem
ZUUL_ZOOKEEPER_TLS_KEY=/etc/ssl/zuul/keys/clientkey.pem
ZUUL_ZOOKEEPER_TLS_CA=/etc/ssl/zuul/certs/cacert.pem

ZUUL_KEYSTORE_PASSWORD=[omitted]

ZUUL_TENANT_CONFIG_SCRIPT=/etc/zuul/gitlab-tenant-config-script
ZUUL_GITLAB_TENANT_CONFIG_PROJECT=29
ZUUL_GITLAB_TENANT_CONFIG_GROUPS=repos/mediawiki

ZUUL_GITLAB_SERVER=gitlab.devtools.wmcloud.org
ZUUL_GITLAB_BASE_URL=https://gitlab.devtools.wmcloud.org/
ZUUL_GITLAB_TOKEN_NAME=zuul
ZUUL_GITLAB_TOKEN=[omitted]
ZUUL_GITLAB_SSH_KEY=/var/ssh/zuul
ZUUL_GITLAB_WEBHOOK_TOKEN=[omitted]

ZUUL_DB_USERNAME=zuul
ZUUL_DB_PASSWORD=[omitted]
ZUUL_DB_HOST=172.19.0.1
ZUUL_DB_DATABASE=zuul

ZUUL_WEB_PORT=9000
ZUUL_WEB_ROOT=https://zuul-devtools.wmcloud.org/

ZUUL_EXECUTOR_TRUSTED_RW_PATHS=/var/lib/zuul/rsyncd.d
ZUUL_EXECUTOR_HOSTNAME=zuul-executor

# Variables without a ZUUL_ prefix will be accessible to the trusted project playbooks
GITLAB_TOKEN=[omitted]
RSYNCD_CONF_DIR=/var/lib/zuul/rsyncd.d
PUBLIC_ZUUL_EXECUTOR_HOSTNAME=zuul.devtools.wmcloud.org
/etc/zuul/rsyncd.conf
use chroot = yes
hosts allow = *
list = no
read only = yes
uid = nobody

&include /var/lib/zuul/rsyncd.d