Page MenuHomePhabricator
Create Task
Maniphest T337426

Upgrade sessionstore cluster to Cassandra 4.1.1
Closed, ResolvedPublic
Actions

Assigned To
Eevans
Authored By
Eevans
May 24 2023, 7:07 PM
Tags
Referenced Files
F37097221: sessionss latency distribution (posts).png
Jun 7 2023, 8:39 PM
F37097219: sessions latency distribution.png
Jun 7 2023, 8:39 PM
Restricted File
Jun 7 2023, 6:59 PM
F37097130: image.png
Jun 7 2023, 6:54 PM
F37097128: image.png
Jun 7 2023, 6:54 PM
Subscribers
Aklapper
Eevans
elukey
hnowlan
KOfori

Description

codfw (2023-06-07 (tentative))

  • Depool codfw data-center
  • Create keyspace snapshots
sudo cumin 'sessionstore2* ' 'c-foreach-nt snapshot -t 4x_upgrade-`date +%Y%m%d%H%I` -- sessions system system_auth system_distributed system_schema system_traces'
  • Upgrade sessionstore2001.codfw.wmnet (merge r926588)
    • Kask Ok (logs)?
    • Logging works (local/remote)?
    • Logged errors/warnings?
    • Prometheus metrics?
    • cqlsh works?
    • nodetool works?
    • Handles (generated) traffic Ok?
      • Load
      • Latency
      • GC
  • Upgrade sessionstore2002.codfw.wmnet (merge r926589)
  • Upgrade sessionstore2003.codfw.wmnet (merge r926590)
  • Generate traffic/load
ssh deploy2002.codfw.wmnet -- siege -f /home/eevans/T327954/urls.txt -i -c 64 -t 3H -d 0.1
    • Cassandra memory/GC Ok?
    • Load
    • Latency
    • Logged errors?
  • Repool codfw data-center (+3h (tentative))
    • Logged errors/warnings?
    • Handles traffic Ok?

eqiad (2023-06-08 (tentative))

  • Depool eqiad data-center
  • Create keyspace snapshots
sudo cumin 'sessionstore1*' 'c-foreach-nt snapshot -t 4x_upgrade-`date +%Y%m%d%H%I` -- sessions system system_auth system_distributed system_schema system_traces'
  • Upgrade sessionstore1001.eqiad.wmnet (merge r928569)
  • Upgrade sessionstore1002.eqiad.wmnet (merge r928570)
  • Upgrade sessionstore1003.eqiad.wmnet (merge r928571)

Post-upgrade

  • Per-host hiera settings moved back to role (r928572)
  • Set legacy_ssl_storage_port_enabled: false (remove assignment, r928573)
  • Set server_encryption_optional: false (remove assignment, r928573)
  • Clear snapshots
c-foreach-nt clearsnapshot -t 4x_upgrade-202306081604

Details

SubjectRepoBranchLines +/-
sessionstore: remove transitional settingsoperations/puppetproduction+0 -2
sessionstore: move per-host settings back to roleoperations/puppetproduction+4 -152
sessionstore: upgrade sessionstore1003 to Cassandra 4.1.1operations/puppetproduction+25 -0
sessionstore: upgrade sessionstore1002 to Cassandra 4.1.1operations/puppetproduction+25 -0
sessionstore: upgrade sessionstore1001 to Cassandra 4.1.1operations/puppetproduction+25 -0
sessionstore: upgrade sessionstore2003 to Cassandra 4.1.1operations/puppetproduction+25 -0
sessionstore: upgrade sessionstore2002 to Cassandra 4.1.1operations/puppetproduction+25 -0
sessionstore: upgrade sessionstore2001 to Cassandra 4.1.1operations/puppetproduction+25 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedEevansT313814 Upgrade to Cassandra 4.x
 ResolvedEevansT337426 Upgrade sessionstore cluster to Cassandra 4.1.1

Event Timeline

Eevans created this task.May 24 2023, 7:07 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMay 24 2023, 7:07 PM
Eevans updated the task description. (Show Details)May 24 2023, 7:24 PM
Eevans updated the task description. (Show Details)May 24 2023, 7:27 PM
Eevans triaged this task as Medium priority.May 24 2023, 8:23 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)May 26 2023, 6:12 PM
Eevans updated the task description. (Show Details)Jun 1 2023, 9:44 PM
Eevans updated the task description. (Show Details)Jun 2 2023, 4:04 PM
Eevans updated the task description. (Show Details)Jun 2 2023, 5:04 PM
gerritbot added a comment.Jun 2 2023, 5:18 PM

Change 926588 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: upgrade sessionstore2001 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/926588

gerritbot added a project: Patch-For-Review.Jun 2 2023, 5:18 PM

Change 926589 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: upgrade sessionstore2002 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/926589

gerritbot added a comment.Jun 2 2023, 5:18 PM

Change 926590 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: upgrade sessionstore2003 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/926590

Eevans updated the task description. (Show Details)Jun 2 2023, 5:19 PM
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)Jun 5 2023, 2:55 PM
Stashbot added a comment.Jun 7 2023, 3:02 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T15:02:33Z] <urandom> de-pooling sessionstore/codfw — T337426

Eevans updated the task description. (Show Details)Jun 7 2023, 3:08 PM
Eevans updated the task description. (Show Details)
gerritbot added a comment.Jun 7 2023, 3:10 PM

Change 926588 merged by Eevans:

[operations/puppet@production] sessionstore: upgrade sessionstore2001 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/926588

Stashbot added a comment.Jun 7 2023, 3:14 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T15:14:45Z] <urandom> Upgrading Cassandra to 4.1.1, sessionstore2001 — T337426

Eevans updated the task description. (Show Details)Jun 7 2023, 3:23 PM
Eevans updated the task description. (Show Details)Jun 7 2023, 3:34 PM
Eevans updated the task description. (Show Details)Jun 7 2023, 3:41 PM
gerritbot added a comment.Jun 7 2023, 3:42 PM

Change 926589 merged by Eevans:

[operations/puppet@production] sessionstore: upgrade sessionstore2002 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/926589

Stashbot added a comment.Jun 7 2023, 3:44 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T15:44:43Z] <urandom> Upgrading Cassandra to 4.1.1, sessionstore2002 — T337426

Eevans updated the task description. (Show Details)Jun 7 2023, 3:49 PM
gerritbot added a comment.Jun 7 2023, 3:50 PM

Change 926590 merged by Eevans:

[operations/puppet@production] sessionstore: upgrade sessionstore2003 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/926590

Stashbot added a comment.Jun 7 2023, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T15:52:07Z] <urandom> Upgrading Cassandra to 4.1.1, sessionstore2003 — T337426

Eevans updated the task description. (Show Details)Jun 7 2023, 3:54 PM
Stashbot added a comment.Jun 7 2023, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T15:56:51Z] <urandom> Beginning (3 hour) generated traffic testing of sessionstore.svc.codfw.wmnet — T337426

Eevans added a comment.EditedJun 7 2023, 6:54 PM

Latency at the service-level seems to be a bit higher than before the upgrade...

image.png (928×1 px, 182 KB)

image.png (792×1 px, 225 KB)

...but this isn't reflected in Cassandra latency, so I'm not sure what (if anything) to make of it, particularly since this is fake/generated (read: not production) traffic.

{F37097133}

Stashbot added a comment.Jun 7 2023, 7:11 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-07T19:11:33Z] <urandom> (Re)pooling codfw sessionstore — T337426

Eevans updated the task description. (Show Details)Jun 7 2023, 7:12 PM
Eevans updated the task description. (Show Details)Jun 7 2023, 8:34 PM
Eevans added a comment.EditedJun 7 2023, 8:39 PM

Here are some latency metrics after repooling the datacenter.

sessions latency distribution.png (859×1 px, 82 KB)
Kask latency distribution (GET)
sessionss latency distribution (posts).png (859×1 px, 84 KB)
Kask latency distribution (POST)
gerritbot added a comment.Jun 8 2023, 2:42 PM

Change 928569 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: upgrade sessionstore1001 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/928569

gerritbot added a comment.Jun 8 2023, 2:42 PM

Change 928570 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: upgrade sessionstore1002 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/928570

gerritbot added a comment.Jun 8 2023, 2:42 PM

Change 928571 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: upgrade sessionstore1003 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/928571

gerritbot added a comment.Jun 8 2023, 2:42 PM

Change 928572 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: move per-host settings back to role

https://gerrit.wikimedia.org/r/928572

gerritbot added a comment.Jun 8 2023, 2:42 PM

Change 928573 had a related patch set uploaded (by Eevans; author: Eevans):

[operations/puppet@production] sessionstore: remove transitional settings

https://gerrit.wikimedia.org/r/928573

Eevans updated the task description. (Show Details)Jun 8 2023, 2:44 PM
Stashbot added a comment.Jun 8 2023, 4:06 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-08T16:06:01Z] <urandom> depooling eqiad sessionstore for Cassandra upgrade — T337426

Eevans updated the task description. (Show Details)Jun 8 2023, 4:08 PM
Stashbot added a comment.Jun 8 2023, 4:22 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-08T16:22:33Z] <urandom> creating pre-upgrade Cassandra snapshots, sessionstore/eqiad — T337426

Eevans updated the task description. (Show Details)Jun 8 2023, 4:23 PM
gerritbot added a comment.Jun 8 2023, 4:24 PM

Change 928569 merged by Eevans:

[operations/puppet@production] sessionstore: upgrade sessionstore1001 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/928569

Stashbot added a comment.Jun 8 2023, 4:26 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-08T16:26:47Z] <urandom> Upgrading Cassandra to 4.1.1, sessionstore1001 — T337426

Eevans updated the task description. (Show Details)Jun 8 2023, 4:32 PM
gerritbot added a comment.Jun 8 2023, 4:32 PM

Change 928570 merged by Eevans:

[operations/puppet@production] sessionstore: upgrade sessionstore1002 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/928570

Stashbot added a comment.Jun 8 2023, 4:35 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-08T16:35:11Z] <urandom> Upgrading Cassandra to 4.1.1, sessionstore1002 — T337426

Eevans updated the task description. (Show Details)Jun 8 2023, 4:38 PM
gerritbot added a comment.Jun 8 2023, 4:39 PM

Change 928571 merged by Eevans:

[operations/puppet@production] sessionstore: upgrade sessionstore1003 to Cassandra 4.1.1

https://gerrit.wikimedia.org/r/928571

Stashbot added a comment.Jun 8 2023, 4:41 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-08T16:40:59Z] <urandom> Upgrading Cassandra to 4.1.1, sessionstore1003 — T337426

Eevans updated the task description. (Show Details)Jun 8 2023, 4:43 PM
Stashbot added a comment.Jun 8 2023, 4:46 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-08T16:46:43Z] <urandom> Starting traffic test against sessionstore.svc.eqiad.wmnet — T337426

gerritbot added a comment.Jun 8 2023, 5:42 PM

Change 928572 merged by Eevans:

[operations/puppet@production] sessionstore: move per-host settings back to role

https://gerrit.wikimedia.org/r/928572

Eevans updated the task description. (Show Details)Jun 8 2023, 5:45 PM
gerritbot added a comment.Jun 8 2023, 6:08 PM

Change 928573 merged by Eevans:

[operations/puppet@production] sessionstore: remove transitional settings

https://gerrit.wikimedia.org/r/928573

Stashbot added a comment.Jun 8 2023, 6:18 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-08T18:18:48Z] <urandom> (Re)pooling sessionstore/eqiad — T337426

Eevans updated the task description. (Show Details)Jun 8 2023, 6:24 PM

macro-deployed

Eevans closed this task as Resolved.Jun 15 2023, 8:51 PM
Eevans claimed this task.
Eevans updated the task description. (Show Details)
Eevans updated the task description. (Show Details)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL