Increase maximum artifacts size for gitlab CI
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Addshore
	Oct 3 2021, 8:27 PM

Description

The default max artifacts size is 100MB
https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-cicd

Artifacts are used by the mwcli repository as part of the release process.

Specifically artifacts are built in a stage and then for tags these are added as a package.
See https://gitlab.wikimedia.org/releng/cli/-/blob/4a1086c56ab14f3400fd6f781ac4d55398a5597e/.gitlab-ci.yml#L98-128

The current size of all artifacts is around 36*7 or 250MB (as of MR https://gitlab.wikimedia.org/releng/cli/-/merge_requests/47) for T288373: [mwcli] Include a command for interacting with WM gitlab instance
And in the future we would see additional binaries T292016: mwcli: build darwin/arm64 binary

~/go/src/gerrit.wikimedia.org/r/mediawiki/tools/cli (glab) $ ls -lahr ./_release/latest/
total 234M
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_windows_amd64.exe.sha256
-rwxr-xr-x 1 adam adam  36M Oct  3 21:15 mw_latest_windows_amd64.exe
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_ppc64le.sha256
-rwxr-xr-x 1 adam adam  35M Oct  3 21:15 mw_latest_linux_ppc64le
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_arm64.sha256
-rwxr-xr-x 1 adam adam  34M Oct  3 21:15 mw_latest_linux_arm64
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_arm.sha256
-rwxr-xr-x 1 adam adam  31M Oct  3 21:15 mw_latest_linux_arm
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_amd64.sha256
-rwxr-xr-x 1 adam adam  36M Oct  3 21:15 mw_latest_linux_amd64
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_linux_386.sha256
-rwxr-xr-x 1 adam adam  32M Oct  3 21:15 mw_latest_linux_386
-rw-r--r-- 1 adam adam   65 Oct  3 21:15 mw_latest_darwin_amd64.sha256
-rwxr-xr-x 1 adam adam  35M Oct  3 21:15 mw_latest_darwin_amd64

It looks like there is some discussion around increasing the default in Gitlab too https://gitlab.com/gitlab-org/gitlab/-/issues/283981

Another alternative for now for the mwcli project would be to avoid uploading these artifacts as CI artifacts, instead combining the 2 jobs into a single job and checking some possible set Gitlab CI environment variable for if the files should be uploaded for release or not.
Though I expect another project further down the line will also end up needing some more artifact space.

Another thing to note about artifacts is that they expire in 30 days by default anyway, so perhaps increasing the max size is no issue?

Details

	Title	Reference	Author	Source Branch	Dest Branch
	Use artifacts as part of the release process once more!	repos/releng/cli!74	addshore	release-with-artifacts-again	main

Customize query in GitLab

Related Objects

Mentioned In: T292381: mwcli release-publish doesnt add binary artifacts on main
Mentioned Here: T288373: [mwcli] Include a command for interacting with WM gitlab instance
T292016: mwcli: build darwin/arm64 binary

Event Timeline

Addshore created this task.Oct 3 2021, 8:27 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 3 2021, 8:27 PM

Addshore moved this task from Inbox to Watching & Blocked on the mwcli board.Oct 3 2021, 8:27 PM

Addshore mentioned this in T292381: mwcli release-publish doesnt add binary artifacts on main.Oct 4 2021, 12:04 AM

I was directed to the settings repo https://gitlab.wikimedia.org/releng/gitlab-settings

And found the docs for changing the setting
https://docs.gitlab.com/ee/administration/job_artifacts.html#set-the-maximum-file-size-of-the-artifacts

But seemingly it is only in the UI? Or at least I can't find the setting value!
https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#maximum-artifacts-size

From the list of settings: https://docs.gitlab.com/ee/api/settings.html#list-of-settings-that-can-be-accessed-via-api-calls

Attribute Type Required Description

max_artifacts_size integer no Maximum artifacts size in MB.

We've currently got:

brennen@gitlab1001:~$ sudo du -hs /var/opt/gitlab/gitlab-rails/shared/artifacts
14G     /var/opt/gitlab/gitlab-rails/shared/artifacts

...I suspect we might need to start thinking about storage for these sooner rather than later.

(Tagging under "GitLab (CI & Job Runners)", but I guess this could also be "Settings" or "Infrastructure".)

In T292372#7398731, @brennen wrote:
We've currently got:
brennen@gitlab1001:~$ sudo du -hs /var/opt/gitlab/gitlab-rails/shared/artifacts
14G     /var/opt/gitlab/gitlab-rails/shared/artifacts
...I suspect we might need to start thinking about storage for these sooner rather than later.

(Tagging under "GitLab (CI & Job Runners)", but I guess this could also be "Settings" or "Infrastructure".)

Almost all, if not all of that, will be coming from the mwcli project right now

The default artifact purge time is probably another thing worth looking at, the default is 30 days, be perhaps that could be decreased.
It can also be set to a lower value per project, and I'd more than happily set it to 7d for this project.
I also wonder if the maximum artifact size per project can be configured? which might lead to earlier purging?

default_artifacts_expire_in -> The syntax of duration is described in artifacts:expire_in and the default value is 30 days. From https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#default-artifacts-expiration

I think Jenkins generally keeps artifacts for 2 weeks currently?

In the short term, I'll put together a MR on gitlab-settings here to bump artifact size and decrease retention a bit. Long term we probably need to come up with a more deliberate policy for exactly what we want to support.

In T292372#7399078, @brennen wrote:

In the short term, I'll put together a MR on gitlab-settings here to bump artifact size and decrease retention a bit. Long term we probably need to come up with a more deliberate policy for exactly what we want to support.

Sounds great to me, and will unblock me in my trailblazing through the gitlab features.

Another interesting point here is that the package registry part of gitlab, which is more permeant than these artifacts from CI appears to have higher limits (perhaps because they general happen less often).
See https://gitlab.wikimedia.org/releng/cli/-/packages

In the long term I saw various things in gitlab docs about hooking gitlab up to object storage options, which probably makes sense for this instance.

brennen moved this task from Backlog to Next on the User-brennen board.Oct 4 2021, 6:31 PM

Will test this tomorrow: https://gitlab.wikimedia.org/releng/gitlab-settings/-/merge_requests/8

brennen moved this task from INBOX to Doing on the Release-Engineering-Team board.Oct 4 2021, 11:15 PM

brennen edited projects, added Release-Engineering-Team (Doing); removed Release-Engineering-Team.

In T292372#7399089, @Addshore wrote:

In T292372#7399078, @brennen wrote:

In the short term, I'll put together a MR on gitlab-settings here to bump artifact size and decrease retention a bit. Long term we probably need to come up with a more deliberate policy for exactly what we want to support.

Sounds great to me, and will unblock me in my trailblazing through the gitlab features.

Another interesting point here is that the package registry part of gitlab, which is more permeant than these artifacts from CI appears to have higher limits (perhaps because they general happen less often).
See https://gitlab.wikimedia.org/releng/cli/-/packages

This is what I've been using/prototyping in my Rust CI pipeline on gitlab.com to eventually use on Wikimedia's Gitlab. It uses https://docs.gitlab.com/ee/user/packages/generic_packages/index.html for persistent storage of generated binaries. Do we know if that'll have a size limit too? I'm hoping we can keep these kind of CI-built release binaries around...forever.

In T292372#7400629, @brennen wrote:

Will test this tomorrow: https://gitlab.wikimedia.org/releng/gitlab-settings/-/merge_requests/8

How did the testing go? ;)

brennen edited projects, added Release-Engineering-Team (Priority Backlog 📥); removed Release-Engineering-Team (Doing).Oct 13 2021, 5:01 PM

Mentioned in SAL (#wikimedia-releng) [2021-10-15T16:14:46Z] <brennen> gitlab.wikimedia.org: apply max artifact size of 350M and artifact expiry of 2 weeks for T292372

LGTM!

Restricted Application added a project: User-Addshore. · View Herald TranscriptOct 15 2021, 4:34 PM

Attribute	Type	Required	Description
max_artifacts_size	integer	no	Maximum artifacts size in MB.

	F34671794: image.png
	Oct 4 2021, 2:20 PM

Increase maximum artifacts size for gitlab CIClosed, ResolvedPublicActions

Description

Details

Related Objects

Event Timeline

Increase maximum artifacts size for gitlab CI
Closed, ResolvedPublic
Actions