Note upgrade path docs here: https://docs.gitlab.com/ce/update/index.html#upgrade-paths

Change 716346 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update repository hook for Gitlab 14

https://gerrit.wikimedia.org/r/716346

Per IRC discussion, we're planning to conduct upgrade week of 2021-09-06. (I'm out 'til US morning of the 7th.)

Using the upstream package repositories, I did a cursory check on gitlab-ansible-test of:

sudo apt-get install gitlab-ce=14.0.9-ce.0
sudo apt-get install gitlab-ce=14.1.4-ce.0 
sudo apt-get install gitlab-ce=14.2.3-ce.0

These all completed without any obvious problems.

Change 716346 merged by Muehlenhoff:

[operations/puppet@production] Update repository hook for Gitlab 14

https://gerrit.wikimedia.org/r/716346

I've uploaded 14.0.10, we can bump the import hook after the initial update is complete.

I've uploaded 14.0.10, we can bump the import hook after the initial update is complete.

Thanks - today got away from me, planning to run through that around 7am MDT, so 2021-09-08 13:00 UTC.

@Jelto I'll coordinate with you tomorrow, but if you want to go ahead with the upgrade on gitlab2001 before I'm online, feel free.

Mentioned in SAL (#wikimedia-operations) [2021-09-08T10:03:21Z] <jelto@cumin1001> START - Cookbook sre.hosts.downtime for 2:00:00 on gitlab2001.wikimedia.org with reason: upgrade gitlab2001 to new version https://phabricator.wikmiedia.org/T289802

Mentioned in SAL (#wikimedia-operations) [2021-09-08T10:03:26Z] <jelto@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on gitlab2001.wikimedia.org with reason: upgrade gitlab2001 to new version https://phabricator.wikmiedia.org/T289802

@Jelto I'll coordinate with you tomorrow, but if you want to go ahead with the upgrade on gitlab2001 before I'm online, feel free.

I upgraded gitlab2001 to gitlab-ce=14.0.10-ce.0 successfully.

Here are the steps I've done:

#Downtime host in icinga
sudo cookbook sre.hosts.downtime -r "upgrade gitlab2001 to new version https://phabricator.wikmiedia.org/T289802" -H 2 gitlab2001.wikimedia.org

# check if all migrations are finished https://docs.gitlab.com/ee/update/index.html#checking-for-background-migrations-before-upgrading
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'

# create backups (may be duplicate, because upgrade also create backups)
sudo /usr/bin/gitlab-backup create CRON=1 STRATEGY=copy GZIP_RSYNCABLE=yes SKIP=builds,artifacts,registry GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1
sudo /usr/bin/gitlab-ctl backup-etc

#preload to-be-installed GitLab CE packages before upgrading (using 
sudo apt-get install gitlab-ce=14.0.10-ce.0 --download-only

# install new version
sudo apt-get install gitlab-ce=14.0.10-ce.0

I did not try zero-downtime upgrades. I've just done the above commands and GitLab runs all migrations on its own. We can try that out in the future when needed.

I will start to collect some information regarding upgrading here: https://wikitech.wikimedia.org/wiki/GitLab/Upgrade

Mentioned in SAL (#wikimedia-operations) [2021-09-08T12:45:07Z] <brennen> gitlab: pausing all runners in preparation for upgrade to 14.0.10 (T289802)

Change 719512 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Update repository hook for Gitlab 14.1

https://gerrit.wikimedia.org/r/719512

Change 719513 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] aptrepo::files::updates Update repository hook for gitlab-runner 14

https://gerrit.wikimedia.org/r/719513

Mentioned in SAL (#wikimedia-operations) [2021-09-08T13:13:17Z] <brennen> gitlab1001: downtiming alerts for 2.5 hours; upgrading to 14.0.10 (T289802)

Change 719512 merged by Jelto:

[operations/puppet@production] Update repository hook for Gitlab 14.1

https://gerrit.wikimedia.org/r/719512

Change 719513 merged by Jelto:

[operations/puppet@production] aptrepo::files::updates Update repository hook for gitlab-runner 14

https://gerrit.wikimedia.org/r/719513

I updated gitlab-runner to 14.2.0 and gitlab-ce to 14.1.5-ce.0 on apt1001.

Mentioned in SAL (#wikimedia-operations) [2021-09-08T13:38:49Z] <brennen> gitlab: upgrading gitlab2001, followed by gitlab1001, to 14.1.5 (T289802)

Change 719519 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] aptrepo::files::updates Update repository hook for Gitlab 14.2

https://gerrit.wikimedia.org/r/719519

Change 719519 merged by Jelto:

[operations/puppet@production] aptrepo::files::updates Update repository hook for Gitlab 14.2

https://gerrit.wikimedia.org/r/719519

Mentioned in SAL (#wikimedia-releng) [2021-09-08T14:05:11Z] <brennen> runner-1002.gitlab-runners: tested upgrade of gitlab-runner to 14.2.0, seemed to go fine, will do remaining runners (T289802)

I updated gitlab-ce to 14.2.3-ce.0 on apt1001

Mentioned in SAL (#wikimedia-operations) [2021-09-08T14:54:26Z] <brennen> gitlab: upgrading gitlab2001, followed by gitlab1001, to 14.2.3 (T289802)