Page MenuHomePhabricator
Create Task
Maniphest T365675

Upgrade GitLab to major version 17
Open, HighPublic
Actions

Description

In two month we have to upgrade to GitLab version 17. This tasks tracks all deprecation.

A list of all deprecation in version 17: https://docs.gitlab.com/ee/update/deprecations.html?removal_milestone=17.0

Deprecation affecting us:

Deprecation NOT affecting us:

  • 'repository_download_operation’ audit event type for public projects
  • Agent for Kubernetes option ca-cert-file renamed
  • Auto DevOps support for Herokuish is deprecated
  • Autogenerated Markdown anchor links with dash (-) characters
  • CiRunner.projects default sort is changing to id_desc
  • Compliance framework in general settings
  • Container registry support for the Swift and OSS storage drivers
  • DAST ZAP advanced configuration variables deprecation
  • Dependency Scanning incorrect SBOM metadata properties
  • Dependency Scanning support for sbt 1.0.X
  • Deprecate GraphQL fields related to the temporary storage increase
  • Deprecate Grype scanner for Container Scanning
  • Deprecate License Scanning CI templates
  • Deprecate Python 3.9 in Dependency Scanning and License Scanning
  • Deprecate Windows CMD in GitLab Runner
  • Deprecate CiRunner GraphQL fields duplicated in CiRunnerManager
  • Deprecate fmt job in Terraform Module CI/CD template
  • Deprecate message field from Vulnerability Management features
  • Deprecate terminationGracePeriodSeconds in the GitLab Runner Kubernetes executor
  • Deprecate version field in feature flag API
  • Deprecate change vulnerability status from the Developer role
  • Deprecate custom role creation for group owners on self-managed
  • Deprecate field hasSolutions from GraphQL VulnerabilityType
  • Deprecate legacy shell escaping and quoting runner shell executor
  • Deprecating Windows Server 2019 in favor of 2022
  • DingTalk OmniAuth provider
  • Duplicate storages in Gitaly configuration
  • File type variable expansion fixed in downstream pipelines
  • Geo: Legacy replication details routes for designs and projects deprecated
  • GitLab Helm chart values gitlab.kas.privateApi.tls.* are deprecated
  • GitLab Runner provenance metadata SLSA v0.2 statement
  • GraphQL API access through unsupported methods
  • GraphQL deprecation of dependencyProxyTotalSizeInBytes field
  • GraphQL field confidential changed to internal on notes
  • GraphQL field registrySizeEstimated has been deprecated
  • GraphQL field totalWeight is deprecated
  • GraphQL networkPolicies resource deprecated
  • GraphQL type, RunnerMembershipFilter renamed to CiRunnerMembershipFilter
  • GraphQL: The DISABLED_WITH_OVERRIDE value of the SharedRunnersSetting enum is deprecated. Use DISABLED_AND_OVERRIDABLE instead
  • GraphQL: deprecate support for canDestroy and canDelete
  • HashiCorp Vault integration will no longer use CI_JOB_JWT by default
  • Heroku image upgrade in Auto DevOps build
  • Internal container registry API tag deletion endpoint
  • Legacy Geo Prometheus metrics
  • Deprecated metric Replacement metric
  • License List is deprecated
  • License Scanning support for sbt 1.0.X
  • Linux packages for Ubuntu 18.04
  • List repository directories Rake task
  • Maintainer role providing the ability to change Package settings using GraphQL API
  • Maven versions below 3.8.8 support in Dependency Scanning and License Scanning
  • Min concurrency and max concurrency in Sidekiq options
  • Offset pagination for /users REST API endpoint is deprecated
  • OmniAuth Facebook is deprecated
  • Package pipelines in API payload is paginated
  • PipelineSecurityReportFinding projectFingerprint GraphQL field
  • Proxy-based DAST deprecated
  • Queue selector for running Sidekiq is deprecated
  • Removal of tags from small SaaS runners on Linux
  • Rename the ‘require_password_to_approve’ field
  • Required Pipeline Configuration is deprecated
  • SAST analyzer coverage changing in GitLab 17.0
  • Scan execution policies enforcing scans with an _EXCLUDED_ANALYZERS variable will override project variables
  • Secure analyzers major version update
  • Security policy field match_on_inclusion is deprecated
  • Security policy field newly_detected is deprecated
  • Support for self-hosted Sentry versions 21.4.1 and earlier
  • The deprecated support is for GitLab instance error tracking features for administrators. The deprecated support does not relate to GitLab error tracking for developers’ own deployed applications.
  • Support for setting custom schema for backup is deprecated
  • The GitHub importer Rake task
  • The Visual Reviews tool is deprecated
  • The gitlab-runner exec command is deprecated
  • The pull-based deployment features of the GitLab agent for Kubernetes is deprecated
  • Trigger jobs can mirror downstream pipeline status exactly
  • Twitter OmniAuth login option is deprecated from self-managed GitLab
  • Unified approval rules are deprecated
  • Upgrading the operating system version of GitLab SaaS runners on Linux
  • Vulnerability confidence field
  • after_script keyword will run for cancelled jobs
  • dependency_files is deprecated
  • metric filter and value field for DORA API
  • omniauth-azure-oauth2 gem is deprecated
  • omnibus_gitconfig configuration item is deprecated
  • postgres_exporter['per_table_stats'] configuration setting
  • npm package uploads now occur asynchronously

Upgraded hosts:

Test instance:

  • gitlab-prod-1002.devtools.eqiad1.wikimedia.cloud
  • gitlab-runner-1002.devtools.eqiad1.wikimedia.cloud
  • gitlab-runner-1003.devtools.eqiad1.wikimedia.cloud
  • gitlab-runner-1005.devtools.eqiad1.wikimedia.cloud - This runner is on bookworm and is not using wmf apt package T367717

Replicas:

  • gitlab1003.wikimedia.org
  • gitlab1004.wikimedia.org

Production:

  • gitlab2002.wikimedia.org
  • Trusted runners
  • Shared runners - will be upgraded by unattended upgrades
  • Cloud runners

Details

SubjectRepoBranchLines +/-
aptrepo: bump gitlab-runner and gitlab-ce to 17.0operations/puppetproduction+2 -2
docker_registry_ha: replace deprecated /-/jwks endpoint on gitlaboperations/puppetproduction+1 -1
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
replace deprecated `/-/jkws` endpoint for server keysrepos/releng/jwt-authorizer!17jnucheT365675main
replace deprecated `/-/jkws` endpoint for server keysrepos/releng/reggie!78jnucheT365675main
Customize query in GitLab

Related Objects
Search...

Event Timeline

Jelto created this task.Thu, May 23, 8:30 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, May 23, 8:30 AM
LSobanski subscribed.Fri, May 24, 10:06 AM
Jelto moved this task from Incoming to Work in Progress on the collaboration-services board.Tue, May 28, 10:13 AM
Jelto triaged this task as High priority.Tue, May 28, 11:38 AM
Jelto updated the task description. (Show Details)
Jelto updated the task description. (Show Details)Tue, May 28, 11:45 AM
Jelto updated the task description. (Show Details)
Jelto updated the task description. (Show Details)Tue, May 28, 1:39 PM
Jelto updated the task description. (Show Details)Tue, May 28, 1:53 PM
Jelto updated the task description. (Show Details)Tue, May 28, 1:55 PM
brennen subscribed.Tue, May 28, 3:38 PM
Jelto updated the task description. (Show Details)Wed, May 29, 10:02 AM
gerritbot added a comment.Wed, May 29, 10:28 AM

Change #1037043 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] docker_registry_ha: replace deprecated /-/jwks endpoint on gitlab

https://gerrit.wikimedia.org/r/1037043

gerritbot added a project: Patch-For-Review.Wed, May 29, 10:28 AM
Jelto updated the task description. (Show Details)Wed, May 29, 10:32 AM
Jelto added a subscriber: jnuche.Wed, May 29, 10:34 AM

@jnuche just to be sure, you replaced the old versions of JSON web tokens (CI_JOB_JWT*) in T337474 and we should not be affected by the deprecation anymore?

https://docs.gitlab.com/ee/update/deprecations.html?removal_milestone=17.0#old-versions-of-json-web-tokens-are-deprecated

CodeReviewBot added a comment.Wed, May 29, 1:33 PM

jnuche opened https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/78

replace deprecated /-/jkws endpoint for server keys

CodeReviewBot added a comment.Wed, May 29, 1:33 PM

jnuche opened https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/17

replace deprecated /-/jkws endpoint for server keys

jnuche added a comment.Wed, May 29, 1:34 PM

@jnuche just to be sure, you replaced the old versions of JSON web tokens (CI_JOB_JWT*) in T337474 and we should not be affected by the deprecation anymore?

Can't speak for other tools, but Kokkuri uses the new id tokens instead of the old deprecated JWT variables. So at least I can say that Kokkuri is not blocking the upgrade.

For what it's worth, I've done some searching in GitLab and the only repos that I could find that are still using CI_JOB_JWT* tokens are an old fork of Kokkuri and two playgrounds.

A different issue is the deprecation of the /-/jwks endpoint, I know of at least two repos that are still using that, I've created MRs to update them. As in the case of the tokens, I'm not aware of whether we have other repos using the old endpoint:

  1. https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/78
  2. https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/17
CodeReviewBot added a comment.Wed, May 29, 3:48 PM

jnuche merged https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/78

replace deprecated /-/jkws endpoint for server keys

CodeReviewBot added a comment.Wed, May 29, 3:48 PM

jnuche merged https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/17

replace deprecated /-/jkws endpoint for server keys

Jelto updated the task description. (Show Details)Wed, May 29, 3:52 PM
Jelto updated the task description. (Show Details)Thu, May 30, 1:33 PM
Jelto updated the task description. (Show Details)Fri, May 31, 1:15 PM
gerritbot added a comment.Mon, Jun 3, 9:59 AM

Change #1037043 merged by Jelto:

[operations/puppet@production] docker_registry_ha: replace deprecated /-/jwks endpoint on gitlab

https://gerrit.wikimedia.org/r/1037043

Jelto updated the task description. (Show Details)Mon, Jun 3, 10:29 AM
Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 3, 10:30 AM
Jelto added a subscriber: eoghan.Mon, Jun 3, 10:35 AM
In T365675#9842110, @jnuche wrote:

Can't speak for other tools, but Kokkuri uses the new id tokens instead of the old deprecated JWT variables. So at least I can say that Kokkuri is not blocking the upgrade.

For what it's worth, I've done some searching in GitLab and the only repos that I could find that are still using CI_JOB_JWT* tokens are an old fork of Kokkuri and two playgrounds.

A different issue is the deprecation of the /-/jwks endpoint, I know of at least two repos that are still using that, I've created MRs to update them. As in the case of the tokens, I'm not aware of whether we have other repos using the old endpoint:

  1. https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/78
  2. https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/17

Thanks for finding the usages in reggie and the jwt-authorizer and uploading a patch! With the changes mentioned above and the change to update the registry nodes (https://gerrit.wikimedia.org/r/1037043) all (production) tools should use the new jwks endpoint.

So all deprecation affecting us are resolved and we should be ready to update to version 17.
@eoghan @jnuche @brennen if you have a few minutes I'd be happy if you go over https://docs.gitlab.com/ee/update/deprecations.html?removal_milestone=17.0 too, in case I missed anything.

brennen added a comment.Mon, Jun 3, 1:49 PM

@brennen if you have a few minutes I'd be happy if you go over https://docs.gitlab.com/ee/update/deprecations.html?removal_milestone=17.0 too, in case I missed anything.

I went through the doc.

The only thing I noticed was Offset pagination for /users REST API endpoint is deprecated. I wondered if this would hit gitlab-settings/group-management, but it looks to me based on the keyset-based pagination docs that it's already fine.

brennen added a subscriber: bd808.Mon, Jun 3, 1:52 PM

The only thing I noticed was Offset pagination for /users REST API endpoint is deprecated. I wondered if this would hit gitlab-settings/group-management, but it looks to me based on the keyset-based pagination docs that it's already fine.

@bd808 might be worth a glance at whether the account approval bot is hit by that one?

bd808 added a comment.Mon, Jun 3, 3:36 PM
In T365675#9854875, @brennen wrote:

The only thing I noticed was Offset pagination for /users REST API endpoint is deprecated. I wondered if this would hit gitlab-settings/group-management, but it looks to me based on the keyset-based pagination docs that it's already fine.

@bd808 might be worth a glance at whether the account approval bot is hit by that one?

Is there a v17 server that things can be tested against? The relevant gitlab-account-approval code does look like it may need updating to fit with https://docs.gitlab.com/ee/api/rest/index.html#keyset-based-pagination. Pretty weird that they would drop the default pagination method there without making the new method the default, but trying to understand the upstream reasoning can often be tricky.

brennen added a comment.Mon, Jun 3, 4:11 PM

We could probably bump https://gitlab.devtools.wmcloud.org/ to v17...

Jelto added a comment.EditedTue, Jun 4, 9:50 AM
In T365675#9855610, @brennen wrote:

We could probably bump https://gitlab.devtools.wmcloud.org/ to v17...

We are on 16.10 at the moment. The upgrade to 17 needs 16.11 as an intermediate step. We have to upgrade to 16.11 next week probably (depending on security updates). If all instances are on 16.11 I can bump the tests instance to 17.0 somewhere end of next week. It's a bit easier if we just have two active version (16.11 and 17.0), supporting 16.10, 16.11 and 17.0 would be a bit tricky regarding the apt package.

https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/?current=16.10.6

brennen updated the task description. (Show Details)Wed, Jun 5, 4:23 PM
gerritbot added a comment.Fri, Jun 14, 12:57 PM

Change #1043764 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] aptrepo: bump gitlab-runner and gitlab-ce to 17.0

https://gerrit.wikimedia.org/r/1043764

gerritbot added a project: Patch-For-Review.Fri, Jun 14, 12:57 PM
Jelto updated the task description. (Show Details)Fri, Jun 14, 1:52 PM
Jelto added a comment.Fri, Jun 14, 1:55 PM

I'm going to upgrade the test instance to 17.0 next Monday June 17th.

gerritbot added a comment.Mon, Jun 17, 7:59 AM

Change #1043764 merged by Jelto:

[operations/puppet@production] aptrepo: bump gitlab-runner and gitlab-ce to 17.0

https://gerrit.wikimedia.org/r/1043764

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 17, 8:30 AM
Jelto updated the task description. (Show Details)Mon, Jun 17, 9:18 AM
Jelto added a comment.Mon, Jun 17, 9:23 AM

I updated the test instance and test gitlab-runners to GitLab version 17. Everything looks good and the migration was successful. @brennen we can test the account approval bot on the test instance now.

gitlab-runner-1005 seems to be a new(er) runner on bookworm and we don't import the gitlab-runner package for bookworm. I'll create a follow up task to properly build or import the package for bookworm as well.

The Shared Runners in WMCS will update to 17 this night automatically due to unattended upgrades.

Jelto updated the task description. (Show Details)Mon, Jun 17, 9:35 AM
brennen mentioned this in T367765: Enable parse_ci_job_timestamps in gitlab-settings and document use of FF_TIMESTAMPS in GitLab CI.Mon, Jun 17, 2:48 PM
Jelto updated the task description. (Show Details)Tue, Jun 18, 9:03 AM
Jelto added a comment.Thu, Jun 20, 8:50 AM

I will upgrade gitlab1003 to version 17.0 later today. If everything goes as planned, I will update the other replica on Monday and the production system during the deployment window on Friday, June 28th.

Jelto added a comment.Thu, Jun 20, 12:53 PM

gitlab1003 was upgraded successfully to 17.0.2: https://gitlab-replica-b.wikimedia.org. I'll proceed with the other replica on Monday.

Jelto updated the task description. (Show Details)Thu, Jun 20, 12:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL