Page MenuHomePhabricator
Create Task
Maniphest T433028

Upgrade restbase cluster to Cassandra 5.0.8
Closed, ResolvedPublic
Actions

Description

  • restbase1031 (canary)
  • restbase2024 (canary)
  • restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet
  • JDK 17

Post-upgrade:

  • Configure storage_compatability_mode: UPGRADING
  • Perform rolling restart
  • Configure storage_compatability_mode: NONE
  • Perform rolling restart

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
restbase: Set storage compatibility to NONEEevansoperations/puppetproduction+1 -1
restbase: Set storage compatibility to UPGRADINGEevansoperations/puppetproduction+1 -0
restbase: Upgrade to JDK 17Eevansoperations/puppetproduction+1 -1
restbase: Upgrade Cassandra to 5.0.8Eevansoperations/puppetproduction+2 -52
restbase: Canary Cassandra 5.0.8 to restbase[1031,2024]Eevansoperations/puppetproduction+50 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Eevans created this task.Thu, Jul 23, 11:23 PM
gerritbot added a comment.Thu, Jul 23, 11:54 PM

Change #1315182 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Canary Cassandra 5.0.8 to restbase[1031,2024]

https://gerrit.wikimedia.org/r/1315182

gerritbot added a project: Patch-For-Review.Thu, Jul 23, 11:54 PM

Change #1315183 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade Cassandra to 5.0.8

https://gerrit.wikimedia.org/r/1315183

gerritbot added a comment.Thu, Jul 23, 11:54 PM

Change #1315184 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade to JDK 17

https://gerrit.wikimedia.org/r/1315184

gerritbot added a comment.Thu, Jul 23, 11:54 PM

Change #1315185 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Set storage compatibility to UPGRADING

https://gerrit.wikimedia.org/r/1315185

gerritbot added a comment.Thu, Jul 23, 11:54 PM

Change #1315186 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Set storage compatibility to NONE

https://gerrit.wikimedia.org/r/1315186

Eevans moved this task from Backlog to In-Progress on the User-Eevans board.Fri, Jul 24, 2:59 PM
gerritbot added a comment.Mon, Jul 27, 2:07 PM

Change #1315182 merged by Eevans:

[operations/puppet@production] restbase: Canary Cassandra 5.0.8 to restbase[1031,2024]

https://gerrit.wikimedia.org/r/1315182

Eevans triaged this task as Medium priority.Mon, Jul 27, 2:28 PM
Eevans updated the task description. (Show Details)

Canary nodes upgraded to: 5.0.8

gerritbot added a comment.Tue, Jul 28, 1:06 PM

Change #1315183 merged by Eevans:

[operations/puppet@production] restbase: Upgrade Cassandra to 5.0.8

https://gerrit.wikimedia.org/r/1315183

Stashbot added a comment.Tue, Jul 28, 1:14 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T13:14:08Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003

Stashbot added a comment.Tue, Jul 28, 3:11 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T15:11:33Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003

gerritbot added a comment.Tue, Jul 28, 4:50 PM

Change #1315184 merged by Eevans:

[operations/puppet@production] restbase: Upgrade to JDK 17

https://gerrit.wikimedia.org/r/1315184

Stashbot added a comment.Tue, Jul 28, 4:55 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T16:55:14Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003

Stashbot added a comment.Tue, Jul 28, 5:56 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T17:56:48Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003

Stashbot added a comment.Tue, Jul 28, 6:36 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T18:36:56Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003

Eevans updated the task description. (Show Details)Tue, Jul 28, 6:50 PM

The full cluster has now been upgraded to 5.0.8. Java upgrades are in progress...

Stashbot added a comment.Tue, Jul 28, 7:39 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-28T19:39:18Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Upgrade Java to 17.0.20 — T433028 - eevans@cumin1003

Eevans updated the task description. (Show Details)Tue, Jul 28, 8:02 PM
Eevans added a comment.Tue, Jul 28, 8:04 PM

Java upgrade is complete; All nodes are now running Cassandra 5.0.8 on Java 17.0.20.

The next step is to move storage compat to UPGRADING, and then NONE (after which a rollback to 4.1.11 will no longer be possible).

gerritbot added a comment.Wed, Aug 12, 3:17 PM

Change #1315185 merged by Eevans:

[operations/puppet@production] restbase: Set storage compatibility to UPGRADING

https://gerrit.wikimedia.org/r/1315185

Stashbot added a comment.Wed, Aug 12, 3:23 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T15:23:42Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2024,1031]*.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003

Stashbot added a comment.Wed, Aug 12, 3:33 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T15:33:41Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2024,1031]*.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003

Stashbot added a comment.Wed, Aug 12, 4:12 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T16:12:21Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003

Stashbot added a comment.Wed, Aug 12, 6:07 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T18:07:27Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase[2025-2038].codfw.wmnet,restbase[1032-1045].eqiad.wmnet: Set storage compatability to UPGRADING — T433028 - eevans@cumin1003

gerritbot added a comment.Wed, Aug 12, 6:14 PM

Change #1315186 merged by Eevans:

[operations/puppet@production] restbase: Set storage compatibility to NONE

https://gerrit.wikimedia.org/r/1315186

Eevans updated the task description. (Show Details)Wed, Aug 12, 6:16 PM
Stashbot added a comment.Wed, Aug 12, 6:18 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T18:18:28Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Set storage compatability to NONE — T433028 - eevans@cumin1003

Maintenance_bot removed a project: Patch-For-Review.Wed, Aug 12, 6:31 PM
Stashbot added a comment.Wed, Aug 12, 7:19 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T19:19:55Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Set storage compatability to NONE — T433028 - eevans@cumin1003

Stashbot added a comment.Wed, Aug 12, 8:09 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T20:09:38Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Set storage compatability to NONE — T433028 - eevans@cumin1003

Stashbot added a comment.Wed, Aug 12, 9:10 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-12T21:10:57Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Set storage compatability to NONE — T433028 - eevans@cumin1003

Eevans closed this task as Resolved.Wed, Aug 12, 9:13 PM
Eevans updated the task description. (Show Details)

{{done}}

Stashbot added a comment.Thu, Aug 13, 7:44 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-13T19:44:31Z] <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs[2001,1016]*: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003

Stashbot added a comment.Thu, Aug 13, 7:52 PM

Mentioned in SAL (#wikimedia-operations) [2026-08-13T19:52:45Z] <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs[2001,1016]*: Upgrade Cassandra to 5.0.8 — T433028 - eevans@cumin1003

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits