Page MenuHomePhabricator
Create Task
Maniphest T339298

Upgrade restbase cluster to Cassandra 4.1.1
Closed, ResolvedPublic
Actions

Description

Upgrade restbase to Cassandra 4.1.1

Upgrade steps
  1. Override cassandra:settings in host-specific hiera files (hieradata/hosts/restbase*.yaml) and set:
    1. internode_encryption: all
    2. server_encryption_optional: true
    3. legacy_ssl_storage_port_enabled: true
    4. target_version: '4.x'
  2. sudo rm /etc/cassandra-[a-z]/service-enabled
  3. Merge Puppet changeset
  4. sudo run-puppet-agent
  5. Restart each instance (id=X; sudo touch /etc/cassandra-$id/service-enabled && sudo systemctl restart cassandra-$id)
  6. check-restbase
codfw
  • row b
    • restbase2013.codfw.wmnet
    • restbase2014.codfw.wmnet
    • restbase2019.codfw.wmnet
    • restbase2021.codfw.wmnet
    • restbase2024.codfw.wmnet
  • row c
    • restbase2015.codfw.wmnet
    • restbase2016.codfw.wmnet
    • restbase2020.codfw.wmnet
    • restbase2022.codfw.wmnet
    • restbase2025.codfw.wmnet
  • row d
    • restbase2012.codfw.wmnet
    • restbase2017.codfw.wmnet
    • restbase2018.codfw.wmnet
    • restbase2023.codfw.wmnet
    • restbase2026.codfw.wmnet
    • restbase2027.codfw.wmnet
eqiad
  • row a
    • restbase1016.eqiad.wmnet
    • restbase1019.eqiad.wmnet
    • restbase1020.eqiad.wmnet
    • restbase1021.eqiad.wmnet
    • restbase1028.eqiad.wmnet
    • restbase1031.eqiad.wmnet
  • row b
    • restbase1017.eqiad.wmnet
    • restbase1022.eqiad.wmnet
    • restbase1023.eqiad.wmnet
    • restbase1024.eqiad.wmnet
    • restbase1029.eqiad.wmnet
    • restbase1032.eqiad.wmnet
  • row d
    • restbase1018.eqiad.wmnet
    • restbase1025.eqiad.wmnet
    • restbase1026.eqiad.wmnet
    • restbase1027.eqiad.wmnet
    • restbase1030.eqiad.wmnet
    • restbase1033.eqiad.wmnet
Post-upgrade steps
  • Move per-host hiera settings back to role
  • Set profile::cassandra::monitor_tls_port: 7000
  • Set legacy_ssl_storage_port_enabled: false (remove assignment)
  • Set server_encryption_optional: false (remove assignment)

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+3 -0restbase: Use port 7000 for ssl monitoring checks
operations/puppetproduction+0 -2restbase: set legacy ssl port & optional encryption to false
operations/puppetproduction+4 -785restbase: move (temporary) per-host settings back to role
operations/puppetproduction+138 -0restbase: upgrade to Cassandra 4.1.1, eqiad/row D (6 hosts)
operations/puppetproduction+138 -0restbase: upgrade to Cassandra 4.1.1, eqiad/row B (6 hosts)
operations/puppetproduction+23 -0restbase: Upgrade restbase1031 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase1028 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase1021 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase1020 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase1019 to Cassandra 4.1.1
operations/puppetproduction+138 -0restbase: upgrade to Cassandra 4.1.1, codfw/row D (6 hosts)
operations/puppetproduction+115 -0restbase: upgrade to Cassandra 4.1.1, codfw/row C (5 hosts)
operations/puppetproduction+23 -0restbase: Upgrade restbase2024 to Cassandra 4.1.1
operations/puppetproduction+24 -0restbase: Upgrade restbase2021 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase2019 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase2014 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase1016 to Cassandra 4.1.1
operations/puppetproduction+23 -0restbase: Upgrade restbase2013 to Cassandra 4.1.1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Eevans triaged this task as Medium priority.Jun 15 2023, 8:48 PM
Eevans created this task.
LSobanski added a subscriber: LSobanski.Jun 20 2023, 9:52 AM
Eevans added a comment.Jul 19 2023, 8:51 PM

The restbase test suite passes with Cassandra 4.1.1 (see: pull/1327), and deployment-restbase04 has been upgraded as well (afaict everything there is working as expected). If there are no objections, I will plan to begin upgrading the production cluster, perhaps as early as the week of the 31st.

Eevans updated the task description. (Show Details)Aug 2 2023, 4:32 PM
Eevans updated the task description. (Show Details)
gerritbot added a comment.Aug 2 2023, 4:48 PM

Change 944959 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase2013 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944959

gerritbot added a project: Patch-For-Review.Aug 2 2023, 4:48 PM

Change 944960 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase2014 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944960

gerritbot added a comment.Aug 2 2023, 4:48 PM

Change 944961 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase2019 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944961

gerritbot added a comment.Aug 2 2023, 4:48 PM

Change 944962 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase2021 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944962

gerritbot added a comment.Aug 2 2023, 4:48 PM

Change 944963 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase2024 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944963

Eevans added a comment.Aug 2 2023, 5:06 PM

@hnowlan I've posted gerrits for the first 5 machines (codfw, row B).

My thinking was to start with one node (say restbase2013) as a canary, and wait a healthy spell before moving forward (24 hours?) If everything goes well, we could gradually pick up the pace and do the remaining nodes over the next 2-3 days. Does this sounds OK to you? If so, I suggest targeting Monday Aug 7 to begin.

gerritbot added a comment.Aug 14 2023, 1:55 PM

Change 944959 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase2013 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944959

Stashbot added a comment.Aug 14 2023, 2:05 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T14:04:59Z] <urandom> upgrading Cassandra to 4.1.1, restbase2013-{a,b,c} — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 2:19 PM
gerritbot added a comment.Aug 14 2023, 2:30 PM

Change 948574 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase1016 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948574

gerritbot added a comment.Aug 14 2023, 2:30 PM

Change 948575 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase1019 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948575

gerritbot added a comment.Aug 14 2023, 2:30 PM

Change 948576 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase1020 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948576

gerritbot added a comment.Aug 14 2023, 2:30 PM

Change 948577 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase1021 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948577

gerritbot added a comment.Aug 14 2023, 2:30 PM

Change 948578 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase1028 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948578

gerritbot added a comment.Aug 14 2023, 2:30 PM

Change 948579 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Upgrade restbase1031 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948579

gerritbot added a comment.Aug 14 2023, 3:32 PM

Change 948574 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase1016 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948574

Eevans updated the task description. (Show Details)Aug 14 2023, 3:33 PM
Eevans updated the task description. (Show Details)
Stashbot added a comment.Aug 14 2023, 3:36 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T15:36:23Z] <urandom> upgrading Cassandra to 4.1.1, restbase1016-{a,b,c} — T339298

Eevans claimed this task.Aug 14 2023, 3:42 PM
Eevans updated the task description. (Show Details)
gerritbot added a comment.Aug 14 2023, 7:08 PM

Change 944960 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase2014 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944960

Stashbot added a comment.Aug 14 2023, 7:11 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T19:11:11Z] <urandom> upgrading Cassandra to 4.1.1, restbase2014-{a,b,c} — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 7:14 PM
Eevans updated the task description. (Show Details)Aug 14 2023, 7:19 PM
gerritbot added a comment.Aug 14 2023, 7:21 PM

Change 944961 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase2019 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944961

Stashbot added a comment.Aug 14 2023, 7:24 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T19:24:23Z] <urandom> upgrading Cassandra to 4.1.1, restbase2019-{a,b,c} — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 7:31 PM
gerritbot added a comment.Aug 14 2023, 7:31 PM

Change 944962 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase2021 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944962

Stashbot added a comment.Aug 14 2023, 7:34 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T19:34:35Z] <urandom> upgrading Cassandra to 4.1.1, restbase2021-{a,b,c} — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 7:41 PM
gerritbot added a comment.Aug 14 2023, 7:42 PM

Change 944963 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase2024 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/944963

Stashbot added a comment.Aug 14 2023, 7:43 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T19:43:07Z] <urandom> upgrading Cassandra to 4.1.1, restbase2024-{a,b,c} — T339298

gerritbot added a comment.Aug 14 2023, 7:49 PM

Change 948654 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, codfw/row C (5 hosts)

https://gerrit.wikimedia.org/r/948654

Eevans updated the task description. (Show Details)Aug 14 2023, 7:50 PM
gerritbot added a comment.Aug 14 2023, 7:53 PM

Change 948654 merged by Eevans:

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, codfw/row C (5 hosts)

https://gerrit.wikimedia.org/r/948654

Stashbot added a comment.Aug 14 2023, 7:57 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T19:57:02Z] <urandom> upgrading Cassandra to 4.1.1, restbase20[15,16,20,22,25]-{a,b,c} (codfw/row C) — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 8:04 PM
gerritbot added a comment.Aug 14 2023, 8:07 PM

Change 948657 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, codfw/row D (6 hosts)

https://gerrit.wikimedia.org/r/948657

gerritbot added a comment.Aug 14 2023, 8:13 PM

Change 948657 merged by Eevans:

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, codfw/row D (6 hosts)

https://gerrit.wikimedia.org/r/948657

Stashbot added a comment.Aug 14 2023, 8:17 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T20:17:10Z] <urandom> upgrading Cassandra to 4.1.1, restbase20[12,17-18,23,26-27]-{a,b,c} (codfw/row C) — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 8:24 PM
gerritbot added a comment.Aug 14 2023, 8:27 PM

Change 948575 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase1019 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948575

gerritbot added a comment.Aug 14 2023, 8:28 PM

Change 948576 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase1020 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948576

gerritbot added a comment.Aug 14 2023, 8:28 PM

Change 948577 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase1021 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948577

gerritbot added a comment.Aug 14 2023, 8:29 PM

Change 948578 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase1028 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948578

gerritbot added a comment.Aug 14 2023, 8:29 PM

Change 948579 merged by Eevans:

[operations/puppet@production] restbase: Upgrade restbase1031 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/948579

Stashbot added a comment.Aug 14 2023, 8:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T20:33:46Z] <urandom> upgrading Cassandra to 4.1.1, restbase10[19-21,28,31]-{a,b,c} (eqiad/row A) — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 8:43 PM
gerritbot added a comment.Aug 14 2023, 8:45 PM

Change 948661 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, eqiad/row B (6 hosts)

https://gerrit.wikimedia.org/r/948661

gerritbot added a comment.Aug 14 2023, 9:23 PM

Change 948661 merged by Eevans:

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, eqiad/row B (6 hosts)

https://gerrit.wikimedia.org/r/948661

Stashbot added a comment.Aug 14 2023, 9:27 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T21:27:43Z] <urandom> upgrading Cassandra to 4.1.1, restbase10[17,22-24,29,32]-{a,b,c} (eqiad/row B) — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 9:37 PM
gerritbot added a comment.Aug 14 2023, 9:40 PM

Change 948666 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, eqiad/row D (6 hosts)

https://gerrit.wikimedia.org/r/948666

gerritbot added a comment.Aug 14 2023, 9:42 PM

Change 948666 merged by Eevans:

[operations/puppet@production] restbase: upgrade to Cassandra 4.1.1, eqiad/row D (6 hosts)

https://gerrit.wikimedia.org/r/948666

Stashbot added a comment.Aug 14 2023, 9:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-14T21:46:23Z] <urandom> upgrading Cassandra to 4.1.1, restbase10[18,25-27,30,33]-{a,b,c} (eqiad/row D) — T339298

Eevans updated the task description. (Show Details)Aug 14 2023, 10:10 PM
ops-monitoring-bot added a comment.Aug 14 2023, 11:32 PM

Host rebooted by eevans@cumin1001 with reason: None

gerritbot added a comment.Aug 16 2023, 4:44 PM

Change 949556 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: move (temporary) per-host settings back to role

https://gerrit.wikimedia.org/r/949556

gerritbot added a comment.Aug 16 2023, 4:49 PM

Change 949556 merged by Eevans:

[operations/puppet@production] restbase: move (temporary) per-host settings back to role

https://gerrit.wikimedia.org/r/949556

Eevans updated the task description. (Show Details)Aug 16 2023, 4:50 PM
gerritbot added a comment.Aug 16 2023, 6:16 PM

Change 949587 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: set legacy ssl port & optional encryption to false

https://gerrit.wikimedia.org/r/949587

gerritbot added a comment.Aug 17 2023, 6:47 PM

Change 949587 merged by Eevans:

[operations/puppet@production] restbase: set legacy ssl port & optional encryption to false

https://gerrit.wikimedia.org/r/949587

Stashbot added a comment.Aug 17 2023, 6:53 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T18:53:10Z] <urandom> Rolling Cassandra restart codfw/b — T339298

gerritbot added a comment.Aug 17 2023, 7:07 PM

Change 950028 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] restbase: Use port 7000 for ssl monitoring checks

https://gerrit.wikimedia.org/r/950028

Eevans updated the task description. (Show Details)Aug 17 2023, 7:08 PM
gerritbot added a comment.Aug 17 2023, 7:09 PM

Change 950028 merged by Eevans:

[operations/puppet@production] restbase: Use port 7000 for ssl monitoring checks

https://gerrit.wikimedia.org/r/950028

Stashbot added a comment.Aug 17 2023, 8:03 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T20:03:55Z] <urandom> Rolling Cassandra restart codfw/c (RESTBase cluster) — T339298

Stashbot added a comment.Aug 17 2023, 8:20 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T20:20:00Z] <urandom> Rolling Cassandra restart codfw/d (RESTBase cluster) — T339298

Stashbot added a comment.Aug 17 2023, 8:30 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T20:30:53Z] <urandom> Rolling Cassandra restart eqiad/a (RESTBase cluster) — T339298

Stashbot added a comment.Aug 17 2023, 8:49 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T20:49:16Z] <urandom> Rolling Cassandra restart eqiad/b (RESTBase cluster) — T339298

Stashbot added a comment.Aug 17 2023, 8:56 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-17T20:56:06Z] <urandom> Rolling Cassandra restart eqiad/d (RESTBase cluster) — T339298

Eevans closed this task as Resolved.Aug 17 2023, 9:05 PM

macro-deployed

Eevans closed subtask T344210: restbase1030: Cassandra crashing (signal 11) as Resolved.Tue, Sep 19, 3:56 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL