GitLab minor version upgrade: 14.9.x
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Jelto
	Mar 24 2022, 4:09 PM

Description

GitLab is behind two new minor releases.

Currently we run gitlab-ce 14.7.4 and gitlab-runner 14.7.0.

We should upgrade gitlab-ce to 14.9.1 and gitlab-runer to 14.9.1.

I looked over the release and upgrade notes and could not find any critical migration/breaking change:
https://about.gitlab.com/releases/2022/03/23/gitlab-14-9-1-released/
https://about.gitlab.com/releases/2022/03/22/gitlab-14-9-released/#upgrade
https://about.gitlab.com/releases/2022/03/16/gitlab-14-8-4-released/
https://about.gitlab.com/releases/2022/03/15/gitlab-14-8-3-released/
https://about.gitlab.com/releases/2022/02/22/gitlab-14-8-released/#upgrade
https://about.gitlab.com/releases/2022/03/09/gitlab-14-7-5-released/

Runner Changelog: https://gitlab.com/gitlab-org/gitlab-runner/blob/v14.9.0/CHANGELOG.md

A update is not strictly necessary (no feature explicitly needed or security patch). However @brennen and I discussed that we should try to keep somewhat up to date with the GitLab upstream version. I added this to the upgrade description. GitLab releases a new minor version once a month. So we should try to update GitLab at least monthly. Security patches will get installed as soon as possible.

Details

Subject	Repo	Branch	Lines +/-
gitlab_runner: override ExecStart in service unit for non-root	operations/puppet	production	+6 -2
gitlab_runner: override User of default service unit file	operations/puppet	production	+3 -17
gitlab_runner: overwrite default service unit file	operations/puppet	production	+1 -0
aptrepo: update gitlab-ce & gitlab-runner to 14.9	operations/puppet	production	+2 -2
aptrepo::files::updates Update gitlab-ce and gitlab-runner to 14.9	operations/puppet	production	+2 -2

Customize query in gerrit

Related Objects

Mentioned Here: T272555: releases2002 ganeti VM not getting IP after reboot

Event Timeline

Jelto created this task.Mar 24 2022, 4:09 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 24 2022, 4:09 PM

Jelto triaged this task as Medium priority.Mar 24 2022, 4:09 PM

@Arnoldokoth and I will do the upgrade of GitLab + Runners on Monday after 4pm UTC.

brennen awarded a token.Mar 25 2022, 4:01 PM

This will happen tomorrow/Tuesday due to scheduling conflicts.

Change 774884 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] aptrepo::files::updates Update gitlab-ce and gitlab-runner to 14.9

https://gerrit.wikimedia.org/r/774884

gerritbot added a project: Patch-For-Review.Mar 29 2022, 1:05 PM

Change 774884 merged by AOkoth:

[operations/puppet@production] aptrepo::files::updates Update gitlab-ce and gitlab-runner to 14.9

https://gerrit.wikimedia.org/r/774884

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2022, 3:12 PM

Change 774905 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] aptrepo: update gitlab-ce & gitlab-runner to 14.9

https://gerrit.wikimedia.org/r/774905

Change 774905 merged by AOkoth:

[operations/puppet@production] aptrepo: update gitlab-ce & gitlab-runner to 14.9

https://gerrit.wikimedia.org/r/774905

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2022, 4:11 PM

@Arnoldokoth and I updated the test instance gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud and the replica gitlab2001.wikimedia.org to gitlab-ce 14.9.1-ce.0.

We will proceed with production instance gitlab1001 and gitlab-runners tomorrow 3pm UTC.

@Arnoldokoth and I updated production instance gitlab1001 and gitlab-runners successfully.

We encountered some unexpected behavior for the apt repo (reprepro) and for the cookbook sre.ganeti.reboot-vm (similar to T272555) which led to a slightly longer maintenance window.

However the update process could be improved by automating the update of GitLab Runners more (see wikitech.wikimedia.org/wiki/GitLab/Upgrade#Upgrading_GitLab_shared_runners) and by not having to pause replica restore (see 773783).

Thanks also to @Dzahn for troubleshooting the sre.ganeti.reboot-vm issue on gitlab2001 yesterday!

Change 775808 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: overwrite default service unit file

https://gerrit.wikimedia.org/r/775808

gerritbot added a project: Patch-For-Review.Mar 31 2022, 9:21 AM

Change 775808 merged by Jelto:

[operations/puppet@production] gitlab_runner: overwrite default service unit file

https://gerrit.wikimedia.org/r/775808

Change 775815 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: override User of default service unit file

https://gerrit.wikimedia.org/r/775815

Change 775815 merged by Jelto:

[operations/puppet@production] gitlab_runner: override User of default service unit file

https://gerrit.wikimedia.org/r/775815

Change 775821 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: override ExecStart of default service unit file

https://gerrit.wikimedia.org/r/775821

Change 775821 merged by Jelto:

[operations/puppet@production] gitlab_runner: override ExecStart in service unit for non-root