Page MenuHomePhabricator
Create Task
Maniphest T339299

Upgrade aqs cluster to Cassandra 4.1.1
Closed, ResolvedPublic
Actions

Description

Upgrade aqs to Cassandra 4.1.1

Upgrade steps
  1. Override cassandra:settings in host-specific hiera files (hieradata/hosts/aqs*.yaml) and set:
    1. internode_encryption: all
    2. server_encryption_optional: true
    3. legacy_ssl_storage_port_enabled: true
    4. target_version: '4.x'
  2. c-foreach-nt snapshot --tag 3_11_14
  3. sudo rm /etc/cassandra-[a-z]/service-enabled
  4. Merge Puppet changeset
  5. sudo run-puppet-agent
  6. Restart each instance (id=X; sudo touch /etc/cassandra-$id/service-enabled && sudo systemctl restart cassandra-$id)
  7. check-aqs
NOTE: The aqs service logging seems to be broken; Prior to upgrade, locally hack /etc/aqs/config.yaml to add a logging type of stdout (restart the service), and use journalctl -f _UID=498 to ensure there are no errors.
codfw
  • a_c
    • aqs2001
    • aqs2002
    • aqs2003
    • aqs2004
  • b_e
    • aqs2005
    • aqs2006
    • aqs2007
    • aqs2008
  • c_f
    • aqs2009
    • aqs2010
    • aqs2011
    • aqs2012
eqiad
  • rack1
    • aqs1010
    • aqs1013
    • aqs1016
    • aqs1019
  • rack2
    • aqs1011
    • aqs1014
    • aqs1017
    • aqs1020
  • rack3
    • aqs1012
    • aqs1015
    • aqs1018
    • aqs1021
Post-upgrade steps
  • Move per-host hiera settings back to role (no-op)
  • Set profile::cassandra::monitor_tls_port: 7000 ¹
  • Set legacy_ssl_storage_port_enabled: false (remove assignment)
  • Set server_encryption_optional: false (remove assignment)
  • Clear snapshots
¹ Push out the change, run puppet on cluster nodes, then alert[1-2]001.wikimedia.org, and verify before continuing. Failure to do so will result in alerts.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
aqs: set legacy ssl port & optional encryption to falseoperations/puppetproduction+0 -2
aqs: move per-host settings back to roleoperations/puppetproduction+8 -1 K
aqs: upgrade codfw/c_f nodes to Cassandra 4.1.1operations/puppetproduction+176 -0
aqs: upgrade codfw/b_c nodes to Cassandra 4.1.1operations/puppetproduction+176 -0
aqs: upgrade codfw/a_c nodes to Cassandra 4.1.1operations/puppetproduction+132 -0
aqs: upgrade rack3 nodes to Cassandra 4.1.1operations/puppetproduction+176 -0
aqs: upgrade rack2 nodes to Cassandra 4.1.1operations/puppetproduction+176 -0
aqs: upgrade rack1 nodes to Cassandra 4.1.1operations/puppetproduction+132 -0
aqs: upgrade aqs1010 to Cassandra 4.1.1operations/puppetproduction+44 -0
aqs: upgrade aqs2001 to Cassandra 4.1.1operations/puppetproduction+44 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedEevansT313814 Upgrade to Cassandra 4.x
 ResolvedEevansT339299 Upgrade aqs cluster to Cassandra 4.1.1

Event Timeline

Eevans created this task.Jun 15 2023, 8:48 PM
Eevans triaged this task as Medium priority.
LSobanski subscribed.Jun 20 2023, 9:52 AM
Eevans added a comment.Aug 2 2023, 8:54 PM

I spent a bit of time trying to run the test suite against Cassandra 4.1.1, but failed. The tests rely on having the database wiped between iterations —the keyspaces are dropped— and the code that (re)creates schema does not work on 4.1.1 (it (ab)uses system tables that have since changed).

Eevans added a comment.Aug 2 2023, 9:34 PM
In T339299#9064591, @Eevans wrote:

I spent a bit of time trying to run the test suite against Cassandra 4.1.1, but failed. The tests rely on having the database wiped between iterations —the keyspaces are dropped— and the code that (re)creates schema does not work on 4.1.1 (it (ab)uses system tables that have since changed).

After a bit more finagling, I was able to get some clean test runs (all 281 tests pass).

For this to work you have to have Cassandra running, and cqlsh in your PATH.

You have to have Docker running, and the aqs-test-local image (run make docker).

You have to patch test/utils/cleandb.sh to change the system keyspace/table name (see: cleandb.sh.patch below).

Finally:

bash test/utils/cleandb.sh \
    && cqlsh -f aqs_test_schema.cql \
    && docker run --rm -it --network host -v $(pwd):/work -w /work aqs-test-local node_modules/mocha/bin/mocha

By creating the expected schema ahead of time (ala aqs_test_schema.cql), the code path broken by the newer Cassandra version is never invoked, and the tests proceed.

Obviously nothing here constitutes a "fix" (either for properly running the test suite, or schema creation), but it at least seems to demonstrate that the driver is compatible with Cassandra 4.1.

cleandb.sh.patch572 BDownload

aqs_test_schema.cql22 KBDownload

Eevans updated the task description. (Show Details)Aug 17 2023, 8:01 PM
Eevans added a subscriber: Milimetric.Aug 18 2023, 3:45 PM

If there are no objections, I'd like to begin an upgrade on Monday (2023-08-21).

/cc @Milimetric @BTullis

gerritbot added a comment.Aug 21 2023, 2:15 PM

Change 951127 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade aqs2001 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951127

gerritbot added a project: Patch-For-Review.Aug 21 2023, 2:15 PM
Eevans updated the task description. (Show Details)Aug 21 2023, 2:19 PM
Eevans updated the task description. (Show Details)Aug 21 2023, 2:24 PM
gerritbot added a comment.Aug 21 2023, 2:29 PM

Change 951127 merged by Eevans:

[operations/puppet@production] aqs: upgrade aqs2001 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951127

Maintenance_bot removed a project: Patch-For-Review.Aug 21 2023, 2:29 PM
Stashbot added a comment.Aug 21 2023, 2:32 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-21T14:32:33Z] <urandom> Upgrading aq2001/cassandra-a (canary) to Cassandra 4.1.1 — T339299

Stashbot added a comment.Aug 21 2023, 2:39 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-21T14:39:24Z] <urandom> Upgrading aq2001/cassandra-b (canary) to Cassandra 4.1.1 — T339299

Eevans updated the task description. (Show Details)Aug 21 2023, 2:46 PM
gerritbot added a comment.Aug 21 2023, 3:32 PM

Change 951145 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade aqs1010 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951145

gerritbot added a project: Patch-For-Review.Aug 21 2023, 3:32 PM
gerritbot added a comment.Aug 21 2023, 6:54 PM

Change 951145 merged by Eevans:

[operations/puppet@production] aqs: upgrade aqs1010 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951145

Stashbot added a comment.Aug 21 2023, 6:56 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-21T18:56:34Z] <urandom> Upgrading aq1010/cassandra-{a,b} (canary) to Cassandra 4.1.1 — T339299

Eevans updated the task description. (Show Details)Aug 21 2023, 7:02 PM
Eevans updated the task description. (Show Details)Aug 21 2023, 7:05 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 21 2023, 7:10 PM
gerritbot added a comment.Aug 22 2023, 1:47 PM

Change 951476 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade rack1 nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951476

gerritbot added a project: Patch-For-Review.Aug 22 2023, 1:47 PM
gerritbot added a comment.Aug 22 2023, 1:50 PM

Change 951478 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade rack2 nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951478

gerritbot added a comment.Aug 22 2023, 1:50 PM

Change 951479 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade rack3 nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951479

gerritbot added a comment.Aug 22 2023, 2:13 PM

Change 951476 merged by Eevans:

[operations/puppet@production] aqs: upgrade rack1 nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951476

Eevans updated the task description. (Show Details)Aug 22 2023, 2:30 PM
Stashbot added a comment.Aug 22 2023, 2:37 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T14:37:01Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs101[6,9].eqiad.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 22 2023, 2:44 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T14:44:56Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs101[6,9].eqiad.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Eevans updated the task description. (Show Details)Aug 22 2023, 2:45 PM
Eevans updated the task description. (Show Details)
gerritbot added a comment.Aug 22 2023, 2:46 PM

Change 951478 merged by Eevans:

[operations/puppet@production] aqs: upgrade rack2 nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951478

Stashbot added a comment.Aug 22 2023, 2:50 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T14:50:47Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs10[11,14,17,20].eqiad.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 22 2023, 3:07 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T15:07:23Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs10[11,14,17,20].eqiad.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

gerritbot added a comment.Aug 22 2023, 3:09 PM

Change 951479 merged by Eevans:

[operations/puppet@production] aqs: upgrade rack3 nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951479

Maintenance_bot removed a project: Patch-For-Review.Aug 22 2023, 3:11 PM
Stashbot added a comment.Aug 22 2023, 3:12 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T15:12:28Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs10[12,15,18,21].eqiad.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 22 2023, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T15:29:17Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs10[12,15,18,21].eqiad.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Eevans updated the task description. (Show Details)Aug 22 2023, 3:43 PM
gerritbot added a comment.Aug 22 2023, 4:01 PM

Change 951535 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade codfw/a_c nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951535

gerritbot added a project: Patch-For-Review.Aug 22 2023, 4:01 PM

Change 951536 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade codfw/b_c nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951536

gerritbot added a comment.Aug 22 2023, 4:01 PM

Change 951537 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: upgrade codfw/c_f nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951537

gerritbot added a comment.Aug 22 2023, 4:55 PM

Change 951535 merged by Eevans:

[operations/puppet@production] aqs: upgrade codfw/a_c nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951535

Stashbot added a comment.Aug 22 2023, 4:58 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T16:58:44Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs200[2-4].codfw.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 22 2023, 5:13 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T17:13:38Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs200[2-4].codfw.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

gerritbot added a comment.Aug 22 2023, 5:26 PM

Change 951536 merged by Eevans:

[operations/puppet@production] aqs: upgrade codfw/b_c nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951536

Eevans updated the task description. (Show Details)Aug 22 2023, 5:28 PM
Stashbot added a comment.Aug 22 2023, 5:30 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T17:30:00Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs200[5-8].codfw.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 22 2023, 5:48 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T17:48:44Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs200[5-8].codfw.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Eevans updated the task description. (Show Details)Aug 22 2023, 8:34 PM
gerritbot added a comment.Aug 22 2023, 8:40 PM

Change 951537 merged by Eevans:

[operations/puppet@production] aqs: upgrade codfw/c_f nodes to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/951537

Stashbot added a comment.Aug 22 2023, 8:44 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T20:44:17Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[09-12].codfw.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 22 2023, 9:02 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T21:02:27Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs20[09-12].codfw.wmnet: Upgrade Cassandra to 4.1.1 — T339299 - eevans@cumin1001

Eevans updated the task description. (Show Details)Aug 22 2023, 9:04 PM
Eevans updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Aug 22 2023, 9:12 PM
gerritbot added a comment.Aug 22 2023, 9:47 PM

Change 951585 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: move per-host settings back to role

https://gerrit.wikimedia.org/r/951585

gerritbot added a project: Patch-For-Review.Aug 22 2023, 9:47 PM
gerritbot added a comment.Aug 22 2023, 9:51 PM

Change 951585 merged by Eevans:

[operations/puppet@production] aqs: move per-host settings back to role

https://gerrit.wikimedia.org/r/951585

Eevans updated the task description. (Show Details)Aug 22 2023, 9:59 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 22 2023, 10:10 PM
gerritbot added a comment.Aug 22 2023, 10:52 PM

Change 951589 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] aqs: set legacy ssl port & optional encryption to false

https://gerrit.wikimedia.org/r/951589

gerritbot added a project: Patch-For-Review.Aug 22 2023, 10:52 PM
gerritbot added a comment.Aug 22 2023, 10:55 PM

Change 951589 merged by Eevans:

[operations/puppet@production] aqs: set legacy ssl port & optional encryption to false

https://gerrit.wikimedia.org/r/951589

Stashbot added a comment.Aug 22 2023, 10:59 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T22:59:43Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Disable legacy SSL port — T339299 - eevans@cumin1001

Maintenance_bot removed a project: Patch-For-Review.Aug 22 2023, 11:10 PM
Stashbot added a comment.Aug 22 2023, 11:45 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T23:45:04Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-codfw: Disable legacy SSL port — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 22 2023, 11:49 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T23:49:52Z] <eevans@cumin1001> START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Disable legacy SSL port — T339299 - eevans@cumin1001

Stashbot added a comment.Aug 23 2023, 12:35 AM

Mentioned in SAL (#wikimedia-operations) [2023-08-23T00:35:58Z] <eevans@cumin1001> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Disable legacy SSL port — T339299 - eevans@cumin1001

Eevans updated the task description. (Show Details)Aug 23 2023, 2:03 AM
Eevans added a comment.Aug 23 2023, 2:13 AM

This upgrade is for all intents and purposes, complete; The only remaining item is to cleanup the snapshots.

I can't see us having to rollback at this stage —but out of an abundance of caution— I will leave the snapshots in place until Monday (Aug. 28), and close this ticket after they've been cleaned up.

Stashbot added a comment.Aug 28 2023, 8:27 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-28T20:27:48Z] <urandom> clear pre-upgrade aqs snapshots — T339299

Eevans updated the task description. (Show Details)Aug 28 2023, 8:29 PM

macro-deployed

Eevans closed this task as Resolved.Aug 28 2023, 8:29 PM
Eevans claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits