Change Details

GitLab has accumulated over 400k CI artifacts which causes problems with storage operations like backups or object storage migration. GitLab is configured for a [7 day and 1 day retention](https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/gitlab2002.yaml?ref_type=heads#L127), https://gitlab.wikimedia.org/admin/application_settings/ci_cd also says `1 day`. So the number of artifacts //should// be limited to the most recent ones. However the growth of artifacts looks more linear to me and increases constantly. So we should double check * Default Artifact Expiration at the Instance Level * check default Artifact Expiration at the Project Level * artifacts are actually expired and deleted * ensure artifacts Are Deleted * maybe manually delete old artifacts with a script or custom job So there is a job for the artifact expiry `Sidekiq::Cron::Job.find('expire_build_artifacts_worker')` and recent logs in `grep 'ExpireBuildArtifactsWorker' /var/log/gitlab/sidekiq/current`.

GitLab has accumulated over 400k CI artifacts which causes problems with storage operations like backups or object storage migration. GitLab is configured for a [7 day and 1 day retention](https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/gitlab2002.yaml?ref_type=heads#L127), https://gitlab.wikimedia.org/admin/application_settings/ci_cd also says `1 day`. So the number of artifacts //should// be limited to the most recent ones. However the growth of artifacts looks more linear to me and increases constantly. So we should double check * Default Artifact Expiration at the Instance Level * check default Artifact Expiration at the Project Level * artifacts are actually expired and deleted * ensure artifacts Are Deleted * maybe manually delete old artifacts with a script or custom job So there is a job for the artifact expiry `Sidekiq::Cron::Job.find('expire_build_artifacts_worker')` and recent logs in `grep 'ExpireBuildArtifactsWorker' /var/log/gitlab/sidekiq/current`. Todos: [x] test lower `ci_delete_pipelines_in_seconds`, `default_artifacts_expire_in` and `archive_builds_in_human_readable` on the replica [x] set reasonable value (`6mo`) for settings above [x] replicas [x] production [x] update [gitlab-settings](https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/69) [] automate configuration of expiry policy somewhere (gitab-settings ci, puppet, configure-projects, ...?)

GitLab has accumulated over 400k CI artifacts which causes problems with storage operations like backups or object storage migration. GitLab is configured for a [7 day and 1 day retention](https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/gitlab2002.yaml?ref_type=heads#L127), https://gitlab.wikimedia.org/admin/application_settings/ci_cd also says `1 day`. So the number of artifacts //should// be limited to the most recent ones. However the growth of artifacts looks more linear to me and increases constantly. So we should double check * Default Artifact Expiration at the Instance Level * check default Artifact Expiration at the Project Level * artifacts are actually expired and deleted * ensure artifacts Are Deleted * maybe manually delete old artifacts with a script or custom job So there is a job for the artifact expiry `Sidekiq::Cron::Job.find('expire_build_artifacts_worker')` and recent logs in `grep 'ExpireBuildArtifactsWorker' /var/log/gitlab/sidekiq/current`. Todos: [x] test lower `ci_delete_pipelines_in_seconds`, `default_artifacts_expire_in` and `archive_builds_in_human_readable` on the replica [x] set reasonable value (`6mo`) for settings above [x] replicas [x] production [x] update [gitlab-settings](https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/69) [] automate configuration of expiry policy somewhere (gitab-settings ci, puppet, configure-projects, ...?)