Page MenuHomePhabricator

Provide separate/larger volume for /var/lib/docker on GitLab runners
Closed, ResolvedPublic2 Estimated Story Points

Description

Provide more capacity on Docker based GitLab runners for built images, container filesystems, and temporary volumes using a separate cinder volume mounted at /var/lib/docker. See T291221: runner-1002 is out of space for recent out-of-space issues.

A quota increase for volume storage and volumes is necessary to unblock this task. See T293832: Request increased quota for gitlab-runners Cloud VPS project.

Event Timeline

dduvall triaged this task as Medium priority.Oct 19 2021, 10:03 PM

Change 732392 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] gitlab: Refactor docker volume parameters to use cinder

https://gerrit.wikimedia.org/r/732392

Change 732392 merged by Dzahn:

[operations/puppet@production] gitlab: Refactor docker volume parameters to use cinder

https://gerrit.wikimedia.org/r/732392

Puppet patches are tested and merge. runner-1002.gitlab-runners.eqiad1.wikimedia.cloud now has a 60G cinder volume mounted at /var/lib/docker. Waiting on an increase to quotas before provisioning/attaching volumes and reconfiguring all runners.

dduvall changed the task status from Stalled to In Progress.Oct 21 2021, 6:05 PM

With T293832: Request increased quota for gitlab-runners Cloud VPS project closed, I'll perform a rolling deletion/creation of runner instances—realized I'll have to do this since the estimate was based off a different flavor than is used by current instances.

Mentioned in SAL (#wikimedia-releng) [2021-10-21T18:07:03Z] <dduvall> replacing runner-1001 with new instance (T293835)

dduvall changed the task status from In Progress to Stalled.Oct 21 2021, 10:35 PM

Waiting on the new flavor.

dduvall changed the task status from Stalled to In Progress.Oct 25 2021, 4:31 PM

Moving forward with runner re-provisioning.

Mentioned in SAL (#wikimedia-releng) [2021-10-25T23:27:23Z] <dduvall> fully provisioned runner-{1008,1011,1012,1013,1014,1015,1016,1017,1018,1019} instances for use as new gitlab runners and removed old instances (T293835)

We now have 10 g3.cores8.ram24.disk20.ephemeral40.4xiops instances and each runs an executor with a max concurrency of 4, for a total of 40 available concurrent jobs.

Note that the flavor made available to us comes with a high-iops 40G ephemeral volume which was chosen by the cinderutils::ensure resource during provisioning (because it resolves by a size range and it favored the first volume found). This was a mistake and yields less space than the 60G volume I was planning to use for each instance.

However, the 40G volumes are much much faster when it comes to IOPS and throughput. I ran a quick benchmark using fio to verify this:

60G standard IOPS volume

test: (groupid=0, jobs=1): err= 0: pid=13532: Mon Oct 25 23:21:09 2021
  read: IOPS=1493, BW=5974KiB/s (6117kB/s)(3070MiB/526262msec)
   bw (  KiB/s): min= 3808, max= 8136, per=100.00%, avg=5972.81, stdev=594.98, samples=1052
   iops        : min=  952, max= 2034, avg=1493.15, stdev=148.74, samples=1052
  write: IOPS=499, BW=1996KiB/s (2044kB/s)(1026MiB/526262msec); 0 zone resets
   bw (  KiB/s): min= 1496, max= 2488, per=99.99%, avg=1995.72, stdev=113.54, samples=1052
   iops        : min=  374, max=  622, avg=498.91, stdev=28.37, samples=1052
  cpu          : usr=2.66%, sys=7.83%, ctx=856527, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=5974KiB/s (6117kB/s), 5974KiB/s-5974KiB/s (6117kB/s-6117kB/s), io=3070MiB (3219MB), run=526262-526262msec
  WRITE: bw=1996KiB/s (2044kB/s), 1996KiB/s-1996KiB/s (2044kB/s-2044kB/s), io=1026MiB (1076MB), run=526262-526262msec

Disk stats (read/write):
  sdc: ios=785920/263175, merge=0/140, ticks=1179322/32106947, in_queue=33279168, util=100.00%

40G 4xIOPS volume

test: (groupid=0, jobs=1): err= 0: pid=13617: Mon Oct 25 23:24:46 2021
  read: IOPS=5893, BW=23.0MiB/s (24.1MB/s)(3070MiB/133363msec)
   bw (  KiB/s): min=  600, max=31448, per=100.00%, avg=23573.22, stdev=2637.22, samples=266
   iops        : min=  150, max= 7862, avg=5893.29, stdev=659.30, samples=266
  write: IOPS=1969, BW=7878KiB/s (8067kB/s)(1026MiB/133363msec); 0 zone resets
   bw (  KiB/s): min=  128, max= 9608, per=100.00%, avg=7877.08, stdev=815.33, samples=266
   iops        : min=   32, max= 2402, avg=1969.27, stdev=203.83, samples=266
  cpu          : usr=7.07%, sys=19.90%, ctx=693863, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=23.0MiB/s (24.1MB/s), 23.0MiB/s-23.0MiB/s (24.1MB/s-24.1MB/s), io=3070MiB (3219MB), run=133363-133363msec
  WRITE: bw=7878KiB/s (8067kB/s), 7878KiB/s-7878KiB/s (8067kB/s-8067kB/s), io=1026MiB (1076MB), run=133363-133363msec

Disk stats (read/write):
  sdb: ios=785530/262538, merge=0/58, ticks=1160955/7313880, in_queue=8465012, util=100.00%

Based on the massive performance difference, I think we should stick with the 40G volumes. If we need additional space or performance, we could conceivably partition the /var/lib/docker partition further to use the 60G volume for Docker volumes and the 40G volume for container/image filesystems. Just a thought.

We have additional space now, more theoretical performance, and we have options for further expansion. Closing this out.