Page MenuHomePhabricator
Create Task
Maniphest T426164

Upgrade GitLab to major version 19
Closed, ResolvedPublic
Actions

Description

In two month we have to upgrade to GitLab version 19. GitLab 19.0 is scheduled for May 21, 2026. This tasks tracks all deprecation.

A list of all deprecation in version 19: https://docs.gitlab.com/update/deprecations/?removal_milestone=19.0

See also update recommendations here: https://docs.gitlab.com/policy/maintenance/#upgrade-recommendations (which are not yet available for 19).

Deprecation affecting us:

Support for PostgreSQL 16 all instances updated to postgresql 17

Deprecation NOT affecting us:

ciJobTokenScopeAddProject GraphQL mutation is deprecated
Azure storage driver for the container registry
Container Registry AWS S3 Signature Version 2 support
Enforce page limit for unauthenticated Projects API requests
Linux package support for SUSE distributions
Linux package support for Ubuntu 20.04
Mattermost bundled with Linux package
Resource owner password credentials grant is deprecated
S3 storage driver (AWS SDK v1) for the container registry
Slack slash commands
Spamcheck support in the Linux package and GitLab Helm chart
Support for bundled PostgreSQL, Redis, and MinIO in GitLab Helm chart
Support for NGINX Ingress
Support for Redis 6
The ci_job_token_scope_enabled projects API attribute is deprecated
The heroku/builder:22 image is deprecated
Trending tab in Explore projects page is deprecated
Using app passwords with Bitbucket Cloud import GitLab API

Upgraded hosts:

Test instance:

  • gitlab-1001.devtools.eqiad1.wikimedia.cloud
  • gitlab-runner-1007.devtools.eqiad1.wikimedia.cloud
  • gitlab-runner-1008.devtools.eqiad1.wikimedia.cloud

Replicas:

  • gitlab1003.wikimedia.org
  • gitlab2002.wikimedia.org

Production:

  • gitlab1004.wikimedia.org
  • Trusted runners
  • Shared runners
  • Cloud runners MR opened

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
aptrepo: update gitlab and gitlab-runner to version 19.0Jeltooperations/puppetproduction+2 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
gitlab-runner: bump image version to alpine-v19.0.1repos/releng/gitlab-cloud-runner!618dancymain-I1890a725e7c1bbc65701526bb3cca9f0f707c8c1main
gitlab-runner: bump image version to alpine-v19.0.1repos/releng/gitlab-cloud-runner!615jeltogitlab-19-0main
Customize query in GitLab

Related Objects
Search...

Event Timeline

Jelto created this task.May 13 2026, 9:47 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptMay 13 2026, 9:47 AM
LSobanski subscribed.May 13 2026, 12:59 PM
Jdforrester-WMF updated the task description. (Show Details)May 13 2026, 1:52 PM
LSobanski triaged this task as High priority.May 19 2026, 9:35 AM
Jelto moved this task from Incoming to Work in Progress on the collaboration-services board.Jun 1 2026, 11:30 AM
Jelto updated the task description. (Show Details)Jun 1 2026, 11:54 AM
Jelto added a comment.Jun 1 2026, 12:01 PM

I went over all deprecation and removals and it looks like only Support for PostgreSQL 16 is affecting our installation.

GitLab follows an annual upgrade cadence for PostgreSQL.
Support for PostgreSQL 16 is scheduled for removal in GitLab 19.0. In GitLab 19.0, PostgreSQL 17 becomes the minimum required PostgreSQL version.
PostgreSQL 17 is available as of GitLab 18.9, and you can upgrade at any time before the removal of PostgreSQL 16 in GitLab 19.0.
If you are running a single PostgreSQL instance you installed by using the Linux package, an automatic upgrade may be attempted with 18.11. Make sure you have enough disk space to accommodate the upgrade.
For more information, see Upgrade packaged PostgreSQL server.

We are still using psql (PostgreSQL) 16.13 and the postgres version will be updated with GitLab 16.11 (so the next upgrade). It would be possible to upgrade Postgres independently with the following steps: https://docs.gitlab.com/omnibus/settings/database/#upgrade-packaged-postgresql-server. But I'd try doing the integrated upgrade with version 16.11. After we migrated to GitLab 16.11 we should see Postgres 17 and all blockers for the major version upgrade should be resolved.

Jelto updated the task description. (Show Details)Jun 12 2026, 11:49 AM

In T428842 postgresql was updated to version 17. So all blockers for the GitLab major version upgrade are resolved.

I'll update the test instance on Monday to version 19 for initial testing.

gerritbot added a comment.Jun 15 2026, 9:46 AM

Change #1302101 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] aptrepo: update gitlab and gitlab-runner to version 19.0

https://gerrit.wikimedia.org/r/1302101

gerritbot added a project: Patch-For-Review.Jun 15 2026, 9:46 AM
gerritbot added a comment.Jun 15 2026, 11:19 AM

Change #1302101 merged by Jelto:

[operations/puppet@production] aptrepo: update gitlab and gitlab-runner to version 19.0

https://gerrit.wikimedia.org/r/1302101

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2026, 11:31 AM
Jelto added a comment.Jun 15 2026, 11:55 AM

I updated gitlab-ce and gitlab-runner to 19.0 and proceed with updating the test hosts

Jelto added subscribers: dancy, brennen.Jun 15 2026, 2:56 PM

The GitLab test instance was upgraded to version v19.0.2 successfully.

@brennen @dancy feel free to test anything on the test instance https://gitlab.devtools.wmcloud.org. I did not spot any critical breaking changes so far. I'll proceed with the upgrade next Monday June 22nd.

Jelto updated the task description. (Show Details)Jun 15 2026, 2:56 PM
brennen added a comment.Jun 15 2026, 5:00 PM

@brennen @dancy feel free to test anything on the test instance https://gitlab.devtools.wmcloud.org. I did not spot any critical breaking changes so far. I'll proceed with the upgrade next Monday June 22nd.

Thanks - will poke around.

Arnoldokoth subscribed.Jun 16 2026, 3:23 PM
Jelto mentioned this in T429503: ProbeDown (gitlab1004).Jun 17 2026, 4:07 PM
Jelto mentioned this in T429505: SystemdUnitFailed ( gitlab-backup-restore.service on gitlab2002).
Jelto mentioned this in T429506: SystemdUnitFailed.Jun 17 2026, 4:10 PM
CodeReviewBot added a project: Patch-For-Review.Jun 17 2026, 4:13 PM

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/615

gitlab-runner: bump image version to alpine-v19.0.1

Jelto updated the task description. (Show Details)Jun 17 2026, 4:14 PM
Jelto added a comment.Jun 17 2026, 4:17 PM

All hosts were upgraded accidentally during an unrelated package upgrade. So all hosts are on 19.0, I also bumped the gitlab-runners to 19.0.1.

@brennen @dancy sorry for the unexpected upgrade.

I'll leave this task open in case there are any issues.

CodeReviewBot added a comment.Jun 17 2026, 5:42 PM

dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/618

gitlab-runner: bump image version to alpine-v19.0.1

CodeReviewBot added a comment.Jun 17 2026, 5:43 PM

dancy closed https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/615

gitlab-runner: bump image version to alpine-v19.0.1

CodeReviewBot added a comment.Jun 17 2026, 5:43 PM

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/618

gitlab-runner: bump image version to alpine-v19.0.1

Maintenance_bot removed a project: Patch-For-Review.Jun 17 2026, 6:31 PM
Dzahn subscribed.Jun 17 2026, 7:14 PM
  Today at 10:08 AM
Hmm.. Gitlab upgrade broke Gerritlab.
..
anything my team should do?

No. I prepared a fix for Gerritlab: https://github.com/yaoyuannnn/gerritlab/pull/86
Jelto created subtask T429595: Explore apt pinning / hold for gitlab-ce package.Jun 18 2026, 12:44 PM
Jelto mentioned this in T429595: Explore apt pinning / hold for gitlab-ce package.
Jelto closed this task as Resolved.Jun 19 2026, 7:12 AM

I'll close the task, all hosts are on version 19 and all breaking changes have been address (thanks for the gerritlab patch and the terraform update @dancy)

Jelto closed subtask T429595: Explore apt pinning / hold for gitlab-ce package as Resolved.Tue, Jun 23, 1:26 PM
Jelto mentioned this in T431966: Investigate GitLab backup size change Jun 2026.Mon, Jul 13, 7:09 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits