Page MenuHomePhabricator

Increase maximum artifacts size for gitlab CI
Closed, ResolvedPublic

Description

The default max artifacts size is 100MB
https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-cicd

Artifacts are used by the mwcli repository as part of the release process.

Specifically artifacts are built in a stage and then for tags these are added as a package.
See https://gitlab.wikimedia.org/releng/cli/-/blob/4a1086c56ab14f3400fd6f781ac4d55398a5597e/.gitlab-ci.yml#L98-128

The current size of all artifacts is around 36*7 or 250MB (as of MR https://gitlab.wikimedia.org/releng/cli/-/merge_requests/47) for T288373: [mwcli] Include a command for interacting with WM gitlab instance
And in the future we would see additional binaries T292016: mwcli: build darwin/arm64 binary

~/go/src/gerrit.wikimedia.org/r/mediawiki/tools/cli (glab) $ ls -lahr ./_release/latest/
total 234M
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_windows_amd64.exe.sha256
-rwxr-xr-x 1 adam adam  36M Oct  3 21:15 mw_latest_windows_amd64.exe
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_ppc64le.sha256
-rwxr-xr-x 1 adam adam  35M Oct  3 21:15 mw_latest_linux_ppc64le
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_arm64.sha256
-rwxr-xr-x 1 adam adam  34M Oct  3 21:15 mw_latest_linux_arm64
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_arm.sha256
-rwxr-xr-x 1 adam adam  31M Oct  3 21:15 mw_latest_linux_arm
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_amd64.sha256
-rwxr-xr-x 1 adam adam  36M Oct  3 21:15 mw_latest_linux_amd64
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_386.sha256
-rwxr-xr-x 1 adam adam  32M Oct  3 21:15 mw_latest_linux_386
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_darwin_amd64.sha256
-rwxr-xr-x 1 adam adam  35M Oct  3 21:15 mw_latest_darwin_amd64

It looks like there is some discussion around increasing the default in Gitlab too https://gitlab.com/gitlab-org/gitlab/-/issues/283981

Another alternative for now for the mwcli project would be to avoid uploading these artifacts as CI artifacts, instead combining the 2 jobs into a single job and checking some possible set Gitlab CI environment variable for if the files should be uploaded for release or not.
Though I expect another project further down the line will also end up needing some more artifact space.

Another thing to note about artifacts is that they expire in 30 days by default anyway, so perhaps increasing the max size is no issue?

Event Timeline

brennen added subscribers: Jelto, Arnoldokoth, brennen.

From the list of settings: https://docs.gitlab.com/ee/api/settings.html#list-of-settings-that-can-be-accessed-via-api-calls

AttributeTypeRequiredDescription
max_artifacts_sizeintegernoMaximum artifacts size in MB.

We've currently got:

brennen@gitlab1001:~$ sudo du -hs /var/opt/gitlab/gitlab-rails/shared/artifacts
14G     /var/opt/gitlab/gitlab-rails/shared/artifacts

...I suspect we might need to start thinking about storage for these sooner rather than later.

(Tagging under "GitLab (CI & Job Runners)", but I guess this could also be "Settings" or "Infrastructure".)

We've currently got:

brennen@gitlab1001:~$ sudo du -hs /var/opt/gitlab/gitlab-rails/shared/artifacts
14G     /var/opt/gitlab/gitlab-rails/shared/artifacts

...I suspect we might need to start thinking about storage for these sooner rather than later.

(Tagging under "GitLab (CI & Job Runners)", but I guess this could also be "Settings" or "Infrastructure".)

Almost all, if not all of that, will be coming from the mwcli project right now

image.png (325×1 px, 42 KB)

The default artifact purge time is probably another thing worth looking at, the default is 30 days, be perhaps that could be decreased.
It can also be set to a lower value per project, and I'd more than happily set it to 7d for this project.
I also wonder if the maximum artifact size per project can be configured? which might lead to earlier purging?

default_artifacts_expire_in -> The syntax of duration is described in artifacts:expire_in and the default value is 30 days. From https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#default-artifacts-expiration

I think Jenkins generally keeps artifacts for 2 weeks currently?

In the short term, I'll put together a MR on gitlab-settings here to bump artifact size and decrease retention a bit. Long term we probably need to come up with a more deliberate policy for exactly what we want to support.

In the short term, I'll put together a MR on gitlab-settings here to bump artifact size and decrease retention a bit. Long term we probably need to come up with a more deliberate policy for exactly what we want to support.

Sounds great to me, and will unblock me in my trailblazing through the gitlab features.

Another interesting point here is that the package registry part of gitlab, which is more permeant than these artifacts from CI appears to have higher limits (perhaps because they general happen less often).
See https://gitlab.wikimedia.org/releng/cli/-/packages

In the long term I saw various things in gitlab docs about hooking gitlab up to object storage options, which probably makes sense for this instance.

In the short term, I'll put together a MR on gitlab-settings here to bump artifact size and decrease retention a bit. Long term we probably need to come up with a more deliberate policy for exactly what we want to support.

Sounds great to me, and will unblock me in my trailblazing through the gitlab features.

Another interesting point here is that the package registry part of gitlab, which is more permeant than these artifacts from CI appears to have higher limits (perhaps because they general happen less often).
See https://gitlab.wikimedia.org/releng/cli/-/packages

This is what I've been using/prototyping in my Rust CI pipeline on gitlab.com to eventually use on Wikimedia's Gitlab. It uses https://docs.gitlab.com/ee/user/packages/generic_packages/index.html for persistent storage of generated binaries. Do we know if that'll have a size limit too? I'm hoping we can keep these kind of CI-built release binaries around...forever.

Mentioned in SAL (#wikimedia-releng) [2021-10-15T16:14:46Z] <brennen> gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for T292372

Addshore claimed this task.

LGTM!