Page MenuHomePhabricator
Create Task
Maniphest T380420

Upgrade Cassandra clusters to v4.1.7
Closed, ResolvedPublic
Actions

Description

4.1.8 is currently being voted on
  • aqs
  • cassandra-dev
  • ml-cache
  • restbase
  • sessionstore
Post-upgrade
  • Copy 4.1.7 packages to component/cassandra41
  • Update dev target versions to 4.x

CHANGELOG

v4.1.7

Merged from 4.0:

v4.1.6

Merged from 4.0:

Merged from 3.0:

Configuration files w/ changes (minor)

  • conf/commitlog_archiving.properties
  • conf/jvm-server.options
  • conf/hotspot_compiler
  • conf/credentials.sample

See: https://gerrit.wikimedia.org/r/1100549

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Event Timeline

Eevans created this task.Nov 20 2024, 9:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2024, 9:04 PM
Eevans triaged this task as Medium priority.Nov 20 2024, 9:04 PM
Eevans added a project: Cassandra.
Eevans updated the task description. (Show Details)Nov 20 2024, 9:15 PM
Eevans updated the task description. (Show Details)Nov 20 2024, 9:18 PM
Reedy renamed this task from Upgrade Casssandra clusters to v4.1.7 to Upgrade Cassandra clusters to v4.1.7.Nov 21 2024, 5:05 AM
Eevans added a comment.Dec 4 2024, 9:26 PM

Cassandra 4.1.7 has been uploaded to apt repo's cassandradev component

gerritbot added a comment.Dec 4 2024, 10:11 PM

Change #1100549 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: configurations merged from upstream 4.1.7

https://gerrit.wikimedia.org/r/1100549

gerritbot added a project: Patch-For-Review.Dec 4 2024, 10:11 PM
gerritbot added a comment.Dec 5 2024, 8:12 PM

Change #1100859 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra_dev: upgrade Cassandra to 'dev' (aka 4.1.7)

https://gerrit.wikimedia.org/r/1100859

gerritbot added a comment.Dec 5 2024, 8:17 PM

Change #1100860 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: pin 'dev' to cassandra 4.1.7

https://gerrit.wikimedia.org/r/1100860

gerritbot added a comment.Dec 5 2024, 8:39 PM

Change #1100549 merged by Eevans:

[operations/puppet@production] cassandra: configurations merged from upstream 4.1.7

https://gerrit.wikimedia.org/r/1100549

gerritbot added a comment.Dec 5 2024, 8:42 PM

Change #1100860 merged by Eevans:

[operations/puppet@production] cassandra: pin 'dev' to cassandra 4.1.7

https://gerrit.wikimedia.org/r/1100860

gerritbot added a comment.Dec 5 2024, 11:25 PM

Change #1100859 merged by Eevans:

[operations/puppet@production] cassandra_dev: upgrade Cassandra to 'dev' (aka 4.1.7)

https://gerrit.wikimedia.org/r/1100859

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2024, 11:31 PM
Eevans updated the task description. (Show Details)Dec 5 2024, 11:41 PM
Eevans updated the task description. (Show Details)Dec 5 2024, 11:44 PM
gerritbot added a comment.Dec 9 2024, 6:55 PM

Change #1101576 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs1010: canary Cassandra 4.1.7

https://gerrit.wikimedia.org/r/1101576

gerritbot added a project: Patch-For-Review.Dec 9 2024, 6:55 PM
gerritbot added a comment.Dec 9 2024, 7:54 PM

Change #1101576 merged by Eevans:

[operations/puppet@production] aqs1010: canary Cassandra 4.1.7

https://gerrit.wikimedia.org/r/1101576

Stashbot added a comment.Dec 9 2024, 7:58 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-09T19:58:49Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs1010.eqiad.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 9 2024, 8:07 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-09T20:07:27Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs1010.eqiad.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Dec 9 2024, 8:31 PM
gerritbot added a comment.Dec 11 2024, 7:37 PM

Change #1102377 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: Upgrade Cassandra to 4.1.7

https://gerrit.wikimedia.org/r/1102377

gerritbot added a project: Patch-For-Review.Dec 11 2024, 7:37 PM
ops-monitoring-bot added a comment.Dec 11 2024, 8:42 PM

Icinga downtime and Alertmanager silence (ID=1e594cac-6111-4c7c-8e83-1c9bdf32ac87) set by eevans@cumin1002 for 7 days, 0:00:00 on 1 host(s) and their services with reason: Hardware replacement

aqs1014.eqiad.wmnet
gerritbot added a comment.Dec 12 2024, 2:53 PM

Change #1102377 merged by Eevans:

[operations/puppet@production] aqs: Upgrade Cassandra to 4.1.7

https://gerrit.wikimedia.org/r/1102377

Stashbot added a comment.Dec 12 2024, 2:55 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T14:55:37Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 12 2024, 3:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T15:03:28Z] <eevans@cumin1002> END (ERROR) - Cookbook sre.cassandra.roll-restart (exit_code=97) for nodes matching A:aqs-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 12 2024, 3:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T15:09:25Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2024, 3:31 PM
Stashbot added a comment.Dec 12 2024, 4:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T16:37:44Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 12 2024, 4:52 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T16:52:47Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 12 2024, 6:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-12T18:22:28Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Eevans updated the task description. (Show Details)Dec 12 2024, 7:56 PM
Eevans added a subscriber: klausman.Dec 16 2024, 2:33 PM

@klausman is ml-cache in use? How would you like to proceed here? Do you want me to upgrade?

gerritbot added a comment.Dec 16 2024, 6:57 PM

Change #1104716 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase1028: canary Cassandra 4.1.7

https://gerrit.wikimedia.org/r/1104716

gerritbot added a project: Patch-For-Review.Dec 16 2024, 6:57 PM
gerritbot added a comment.Dec 16 2024, 7:01 PM

Change #1104716 merged by Eevans:

[operations/puppet@production] restbase1028: canary Cassandra 4.1.7

https://gerrit.wikimedia.org/r/1104716

Stashbot added a comment.Dec 16 2024, 7:07 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-16T19:07:12Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching restbase1028.eqiad.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 16 2024, 7:16 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-16T19:16:34Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase1028.eqiad.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Dec 16 2024, 7:30 PM
gerritbot added a comment.Dec 17 2024, 3:13 PM

Change #1105011 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade Cassandra to 4.1.7

https://gerrit.wikimedia.org/r/1105011

gerritbot added a project: Patch-For-Review.Dec 17 2024, 3:13 PM
gerritbot added a comment.Dec 17 2024, 3:22 PM

Change #1105011 merged by Eevans:

[operations/puppet@production] restbase: Upgrade Cassandra to 4.1.7

https://gerrit.wikimedia.org/r/1105011

Stashbot added a comment.Dec 17 2024, 3:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-17T15:30:42Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Dec 17 2024, 3:30 PM
Stashbot added a comment.Dec 17 2024, 5:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-17T17:46:05Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 17 2024, 6:11 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-17T18:11:25Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 17 2024, 8:23 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-17T20:23:41Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Eevans updated the task description. (Show Details)Dec 18 2024, 1:19 AM
gerritbot added a comment.Dec 18 2024, 4:58 PM

Change #1105386 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: Upgrade Cassandra to v4.1.7

https://gerrit.wikimedia.org/r/1105386

gerritbot added a project: Patch-For-Review.Dec 18 2024, 4:58 PM
gerritbot added a comment.Dec 18 2024, 5:28 PM

Change #1105386 merged by Eevans:

[operations/puppet@production] sessionstore: Upgrade Cassandra to v4.1.7

https://gerrit.wikimedia.org/r/1105386

Maintenance_bot removed a project: Patch-For-Review.Dec 18 2024, 5:31 PM
Stashbot added a comment.Dec 18 2024, 5:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-18T17:32:33Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore2*.codfw.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 18 2024, 5:50 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-18T17:50:20Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore2*.codfw.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 18 2024, 6:00 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-18T18:00:17Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore1*.eqiad.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Dec 18 2024, 6:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-18T18:18:09Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore1*.eqiad.wmnet: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Eevans changed the task status from Open to Stalled.Dec 18 2024, 7:33 PM
Eevans updated the task description. (Show Details)

With the completion of sessionstore, the only remaining cluster is ml-cache, which I think will have to wait until after the holidays.

Eevans moved this task from Backlog to In-Progress on the Cassandra board.Dec 19 2024, 8:43 PM
Eevans added a comment.Jan 6 2025, 8:32 PM

@klausman ping! Would you like me to upgrade ml-cache to 4.1.7?

klausman added a comment.Jan 9 2025, 10:33 AM
In T380420#10434102, @Eevans wrote:

@klausman ping! Would you like me to upgrade ml-cache to 4.1.7?

Yes, please go ahead! The machines are currently not in use.

gerritbot added a comment.Jan 10 2025, 5:12 PM

Change #1109750 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] ml-cache: upgrade Cassandra to 4.1.7

https://gerrit.wikimedia.org/r/1109750

gerritbot added a project: Patch-For-Review.Jan 10 2025, 5:12 PM
gerritbot added a comment.Jan 10 2025, 7:08 PM

Change #1109750 merged by Eevans:

[operations/puppet@production] ml-cache: upgrade Cassandra to 4.1.7

https://gerrit.wikimedia.org/r/1109750

Stashbot added a comment.Jan 10 2025, 7:23 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-10T19:23:33Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Jan 10 2025, 7:30 PM
Stashbot added a comment.Jan 10 2025, 7:41 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-10T19:41:15Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Jan 10 2025, 7:43 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-10T19:43:29Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Eevans updated the task description. (Show Details)Jan 10 2025, 7:53 PM
Stashbot added a comment.Jan 10 2025, 8:01 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-10T20:01:13Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Eevans updated the task description. (Show Details)Jan 10 2025, 8:04 PM
gerritbot added a comment.Jan 10 2025, 8:22 PM

Change #1109767 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: rotate target_version 'dev' to '4.x'

https://gerrit.wikimedia.org/r/1109767

gerritbot added a project: Patch-For-Review.Jan 10 2025, 8:22 PM
gerritbot added a comment.Jan 10 2025, 8:43 PM

Change #1109768 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] cassandra: set target_dev to 4.x (no-op)

https://gerrit.wikimedia.org/r/1109768

gerritbot added a comment.Jan 14 2025, 12:50 AM

Change #1109767 merged by Eevans:

[operations/puppet@production] cassandra: rotate target_version 'dev' to '4.x'

https://gerrit.wikimedia.org/r/1109767

gerritbot added a comment.Jan 15 2025, 4:54 PM

Change #1109768 merged by Eevans:

[operations/puppet@production] cassandra: set target_dev to 4.x (no-op)

https://gerrit.wikimedia.org/r/1109768

Stashbot added a comment.Jan 15 2025, 4:58 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-15T16:58:33Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:cassandra-dev: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Stashbot added a comment.Jan 15 2025, 5:18 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-15T17:18:44Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:cassandra-dev: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2025, 5:31 PM
Eevans updated the task description. (Show Details)Jan 15 2025, 5:37 PM
Eevans closed this task as Resolved.Jan 15 2025, 5:39 PM
Eevans claimed this task.

Done!!

Stashbot added a comment.Feb 20 2025, 5:29 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-20T17:29:51Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Eevans added a comment.Feb 20 2025, 5:40 PM
In T380420#10568591, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2025-02-20T17:29:51Z] <eevans@cumin1002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

FYI, this was (apparently) a PEBKAC of some sort while on the cumin host (a command from my shell history re-ran?). The cluster nodes were restarted, but not to apply 4.1.7 (that has already happened). Apologies for the noise.

Stashbot added a comment.Feb 20 2025, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-02-20T17:47:33Z] <eevans@cumin1002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Upgrading to Cassandra 4.1.7 — T380420 - eevans@cumin1002

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits