Page MenuHomePhabricator
Create Task
Maniphest T178905

Upgrade RESTBase cluster to Cassandra release: 3.11.2
Closed, ResolvedPublic
Actions

Description

3.11.1 has been released upstream, and contains a number of fixes that we previously patched our internal build of 3.11.0 for. It also contains many fixes for things that we were not patching into our build, and so should be tested prior to deployment.

3.11.2, when released, will contain a fix necessary to use OpenJDK versions >= 1.8.0_162

CompleteHostInstance
restbase1007.eqiad.wmneta
restbase1007.eqiad.wmnetb
restbase1007.eqiad.wmnetc
restbase1010.eqiad.wmneta
restbase1010.eqiad.wmnetb
restbase1010.eqiad.wmnetc
restbase1011.eqiad.wmneta
restbase1011.eqiad.wmnetb
restbase1011.eqiad.wmnetc
restbase1016.eqiad.wmneta
restbase1016.eqiad.wmnetb
restbase1016.eqiad.wmnetc
restbase1008.eqiad.wmneta
restbase1008.eqiad.wmnetb
restbase1008.eqiad.wmnetc
restbase1012.eqiad.wmneta
restbase1012.eqiad.wmnetb
restbase1012.eqiad.wmnetc
restbase1017.eqiad.wmneta
restbase1017.eqiad.wmnetb
restbase1017.eqiad.wmnetc
restbase1013.eqiad.wmneta
restbase1013.eqiad.wmnetb
restbase1013.eqiad.wmnetc
restbase1009.eqiad.wmneta
restbase1009.eqiad.wmnetb
restbase1009.eqiad.wmnetc
restbase1014.eqiad.wmneta
restbase1014.eqiad.wmnetb
restbase1014.eqiad.wmnetc
restbase1015.eqiad.wmneta
restbase1015.eqiad.wmnetb
restbase1015.eqiad.wmnetc
restbase1018.eqiad.wmneta
restbase1018.eqiad.wmnetb
restbase1018.eqiad.wmnetc
restbase2003.codfw.wmneta
restbase2003.codfw.wmnetb
restbase2003.codfw.wmnetc
restbase2004.codfw.wmneta
restbase2004.codfw.wmnetb
restbase2004.codfw.wmnetc
restbase2008.codfw.wmneta
restbase2008.codfw.wmnetb
restbase2008.codfw.wmnetc
restbase2011.codfw.wmneta
restbase2011.codfw.wmnetb
restbase2011.codfw.wmnetc
restbase2001.codfw.wmneta
restbase2001.codfw.wmnetb
restbase2001.codfw.wmnetc
restbase2002.codfw.wmneta
restbase2002.codfw.wmnetb
restbase2002.codfw.wmnetc
restbase2007.codfw.wmneta
restbase2007.codfw.wmnetb
restbase2007.codfw.wmnetc
restbase2010.codfw.wmneta
restbase2010.codfw.wmnetb
restbase2010.codfw.wmnetc
restbase2005.codfw.wmneta
restbase2005.codfw.wmnetb
restbase2005.codfw.wmnetc
restbase2006.codfw.wmneta
restbase2006.codfw.wmnetb
restbase2006.codfw.wmnetc
restbase2009.codfw.wmneta
restbase2009.codfw.wmnetb
restbase2009.codfw.wmnetc
restbase2012.codfw.wmneta
restbase2012.codfw.wmnetb
restbase2012.codfw.wmnetc

Details

SubjectRepoBranchLines +/-
cassandra: upgrade 3.x version to 3.11.2operations/puppetproduction+2 -2
cassandra: update configuration for 3.11.2operations/puppetproduction+24 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Eevans created this task.Oct 24 2017, 2:45 PM
• mobrovac edited projects, added Services (next); removed Services (watching).Oct 24 2017, 2:47 PM
Eevans added a project: User-Eevans.Oct 25 2017, 6:19 PM
Eevans moved this task from Backlog to Next on the User-Eevans board.Dec 5 2017, 3:39 PM
Eevans renamed this task from Evaluate new upstream Cassandra release: 3.11.1 to Evaluate new upstream Cassandra release: 3.11.2.Feb 2 2018, 5:37 PM
Eevans updated the task description. (Show Details)
Eevans mentioned this in T186360: Cassandra 3.11.0 incompatible with OpenJDK >= 1.8.0_162.Feb 2 2018, 9:04 PM
Ricordisamoa subscribed.Feb 2 2018, 10:39 PM
Eevans added a comment.Mar 19 2018, 3:56 PM

Times up, 8u162-b12-1~bpo8+1 has been made available.

MoritzMuehlenhoff subscribed.Mar 19 2018, 6:06 PM
Stashbot subscribed.Mar 28 2018, 6:12 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:12:14Z] <urandom> upgrading restbase-dev1004-a to cassandra 3.11.2 (canary) -- T178905

Stashbot added a comment.Mar 28 2018, 6:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:17:17Z] <urandom> upgrading restbase-dev1004-b to cassandra 3.11.2 (canary) -- T178905

Stashbot added a comment.Mar 28 2018, 6:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:52:33Z] <urandom> upgrading restbase-dev1005-{a,b} to cassandra 3.11.2 -- T178905

Eevans added a comment.Mar 29 2018, 9:24 PM

Opened CASSANDRA-14355 to track a memory leak encountered in the dev environment (reproduced on 3.11.0 and 3.11.2).

Eevans renamed this task from Evaluate new upstream Cassandra release: 3.11.2 to Upgrade RESTBase cluster to Cassandra release: 3.11.2.Apr 13 2018, 1:29 PM
Eevans raised the priority of this task from Low to Medium.
Eevans moved this task from Next to In-Progress on the User-Eevans board.Jun 4 2018, 4:39 PM
Eevans edited projects, added Services (doing); removed Services (next).
gerritbot subscribed.Jun 4 2018, 7:05 PM

Change 437298 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: update configuration for 3.11.2

https://gerrit.wikimedia.org/r/437298

gerritbot added a project: Patch-For-Review.Jun 4 2018, 7:05 PM
• mobrovac added a parent task: T190689: FY17/18 Q4 Program 7 Services Goal: Language variants support.Jun 5 2018, 9:16 AM
gerritbot added a comment.Jun 5 2018, 2:28 PM

Change 437298 merged by Elukey:
[operations/puppet@production] cassandra: update configuration for 3.11.2

https://gerrit.wikimedia.org/r/437298

Stashbot added a comment.Jun 5 2018, 2:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T14:52:50Z] <urandom> restarting restbase-dev1004 - T178905

Stashbot added a comment.Jun 5 2018, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T15:58:22Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1004 - T178905

Stashbot added a comment.Jun 5 2018, 4:56 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T16:56:15Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1005 - T178905

Stashbot added a comment.Jun 5 2018, 5:00 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T17:00:56Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1006 - T178905

Stashbot added a comment.Jun 5 2018, 5:45 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T17:45:08Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001 (canary) - T178905

Eevans updated the task description. (Show Details)Jun 5 2018, 5:52 PM
Stashbot added a comment.Jun 5 2018, 7:35 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T19:35:21Z] <urandom> upgrade Cassandra to 3.11.2, restbase1007-a (canary) - T178905

Eevans updated the task description. (Show Details)Jun 5 2018, 10:30 PM
Stashbot added a comment.Jun 6 2018, 3:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:06:17Z] <urandom> upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 3:18 PM
Stashbot added a comment.Jun 6 2018, 3:27 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:27:23Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001-{b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 3:51 PM
Stashbot added a comment.Jun 6 2018, 3:54 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:54:25Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 5:00 PM
Stashbot added a comment.Jun 6 2018, 5:01 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T17:01:20Z] <urandom> upgrade Cassandra to 3.11.2, restbase2007-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 5:42 PM
Stashbot added a comment.Jun 6 2018, 5:59 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T17:59:22Z] <urandom> upgrade Cassandra to 3.11.2, restbase2010-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 6:16 PM
Stashbot added a comment.Jun 6 2018, 6:19 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T18:19:00Z] <urandom> upgrade Cassandra to 3.11.2, restbase2003-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 6:53 PM
Stashbot added a comment.Jun 6 2018, 6:54 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T18:54:50Z] <urandom> upgrade Cassandra to 3.11.2, restbase2004-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 7:36 PM
Stashbot added a comment.Jun 6 2018, 7:38 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T19:38:08Z] <urandom> upgrade Cassandra to 3.11.2, restbase2008-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 8:15 PM
Stashbot added a comment.Jun 6 2018, 8:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T20:17:27Z] <urandom> upgrade Cassandra to 3.11.2, restbase2011-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 8:41 PM
Stashbot added a comment.Jun 6 2018, 8:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T20:42:19Z] <urandom> upgrade Cassandra to 3.11.2, restbase2005-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 10:02 PM
Eevans updated the task description. (Show Details)Jun 6 2018, 11:23 PM
Stashbot added a comment.Jun 6 2018, 11:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T23:23:46Z] <urandom> upgrade Cassandra to 3.11.2, restbase2009-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 11:50 PM
Stashbot added a comment.Jun 6 2018, 11:51 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T23:51:48Z] <urandom> upgrade Cassandra to 3.11.2, restbase2012-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 7 2018, 12:22 AM
Stashbot added a comment.Jun 7 2018, 3:38 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T15:38:53Z] <urandom> upgrade Cassandra to 3.11.2, restbase1010-{a,b,c} - T178905

gerritbot added a comment.Jun 7 2018, 3:55 PM

Change 438035 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: upgrade 3.x version to 3.11.2

https://gerrit.wikimedia.org/r/438035

Eevans updated the task description. (Show Details)Jun 7 2018, 3:58 PM
Stashbot added a comment.Jun 7 2018, 4:21 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T16:21:54Z] <urandom> rolling Cassandra restart, restbase1007 - T178905

Stashbot added a comment.Jun 7 2018, 4:45 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T16:45:45Z] <urandom> rolling Cassandra restart, restbase2001, restbase2002, restbase2007 - T178905

Stashbot added a comment.Jun 7 2018, 6:20 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T18:20:40Z] <urandom> restarting Cassandra, restbase2003-c -- T178905

Stashbot added a comment.Jun 7 2018, 6:28 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T18:28:20Z] <urandom> rolling Cassandra restart, restbase2004, restbase2008, restbase2011 -- T178905

Stashbot added a comment.Jun 7 2018, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T19:47:23Z] <urandom> rolling Cassandra restart, restbase2005, restbase2006, restbase2012 -- T178905

gerritbot added a comment.Jun 8 2018, 7:27 AM

Change 438035 merged by Dzahn:
[operations/puppet@production] cassandra: upgrade 3.x version to 3.11.2

https://gerrit.wikimedia.org/r/438035

Stashbot added a comment.Jun 8 2018, 2:17 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T14:17:21Z] <urandom> upgrade Cassandra to 3.11.2, restbase1011 & restbase1016 - T178905

Eevans changed the status of subtask T196745: Add Cassandra 3.11.2 package to internal APT repository from Open to Stalled.Jun 8 2018, 2:57 PM
Eevans updated the task description. (Show Details)
Stashbot added a comment.Jun 8 2018, 3:00 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T15:00:09Z] <urandom> upgrade Cassandra to 3.11.2, restbase1008 - T178905

Stashbot added a comment.Jun 8 2018, 3:35 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T15:35:26Z] <urandom> upgrade Cassandra to 3.11.2, restbase1012 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 5:58 PM
Stashbot added a comment.Jun 8 2018, 6:02 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T18:02:33Z] <urandom> upgrade Cassandra to 3.11.2, restbase1017 & restbase1013 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 6:44 PM
Stashbot added a comment.Jun 8 2018, 6:49 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T18:49:16Z] <urandom> upgrade Cassandra to 3.11.2, restbase1009 & restbase1014 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 7:32 PM
Stashbot added a comment.Jun 8 2018, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T19:36:00Z] <urandom> upgrade Cassandra to 3.11.2, restbase1015 & restbase1018 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 8:21 PM
Eevans added a comment.Jun 8 2018, 8:33 PM

The upgrade is complete:

P7229 Masterwork From Distant Lands
1$ cdsh -- c-foreach-nt version
2restbase1007.eqiad.wmnet: a: ReleaseVersion: 3.11.2
3restbase1007.eqiad.wmnet: b: ReleaseVersion: 3.11.2
4restbase1007.eqiad.wmnet: c: ReleaseVersion: 3.11.2
5restbase1010.eqiad.wmnet: a: ReleaseVersion: 3.11.2
6restbase1010.eqiad.wmnet: b: ReleaseVersion: 3.11.2
7restbase1010.eqiad.wmnet: c: ReleaseVersion: 3.11.2
8restbase1011.eqiad.wmnet: a: ReleaseVersion: 3.11.2
9restbase1011.eqiad.wmnet: b: ReleaseVersion: 3.11.2
10restbase1011.eqiad.wmnet: c: ReleaseVersion: 3.11.2
11restbase1016.eqiad.wmnet: a: ReleaseVersion: 3.11.2
12restbase1016.eqiad.wmnet: b: ReleaseVersion: 3.11.2
13restbase1016.eqiad.wmnet: c: ReleaseVersion: 3.11.2
14restbase1008.eqiad.wmnet: a: ReleaseVersion: 3.11.2
15restbase1008.eqiad.wmnet: b: ReleaseVersion: 3.11.2
16restbase1008.eqiad.wmnet: c: ReleaseVersion: 3.11.2
17restbase1012.eqiad.wmnet: a: ReleaseVersion: 3.11.2
18restbase1012.eqiad.wmnet: b: ReleaseVersion: 3.11.2
19restbase1012.eqiad.wmnet: c: ReleaseVersion: 3.11.2
20restbase1017.eqiad.wmnet: a: ReleaseVersion: 3.11.2
21restbase1017.eqiad.wmnet: b: ReleaseVersion: 3.11.2
22restbase1017.eqiad.wmnet: c: ReleaseVersion: 3.11.2
23restbase1013.eqiad.wmnet: a: ReleaseVersion: 3.11.2
24restbase1013.eqiad.wmnet: b: ReleaseVersion: 3.11.2
25restbase1013.eqiad.wmnet: c: ReleaseVersion: 3.11.2
26restbase1009.eqiad.wmnet: a: ReleaseVersion: 3.11.2
27restbase1009.eqiad.wmnet: b: ReleaseVersion: 3.11.2
28restbase1009.eqiad.wmnet: c: ReleaseVersion: 3.11.2
29restbase1014.eqiad.wmnet: a: ReleaseVersion: 3.11.2
30restbase1014.eqiad.wmnet: b: ReleaseVersion: 3.11.2
31restbase1014.eqiad.wmnet: c: ReleaseVersion: 3.11.2
32restbase1015.eqiad.wmnet: a: ReleaseVersion: 3.11.2
33restbase1015.eqiad.wmnet: b: ReleaseVersion: 3.11.2
34restbase1015.eqiad.wmnet: c: ReleaseVersion: 3.11.2
35restbase1018.eqiad.wmnet: a: ReleaseVersion: 3.11.2
36restbase1018.eqiad.wmnet: b: ReleaseVersion: 3.11.2
37restbase1018.eqiad.wmnet: c: ReleaseVersion: 3.11.2
38restbase2003.codfw.wmnet: a: ReleaseVersion: 3.11.2
39restbase2003.codfw.wmnet: b: ReleaseVersion: 3.11.2
40restbase2003.codfw.wmnet: c: ReleaseVersion: 3.11.2
41restbase2004.codfw.wmnet: a: ReleaseVersion: 3.11.2
42restbase2004.codfw.wmnet: b: ReleaseVersion: 3.11.2
43restbase2004.codfw.wmnet: c: ReleaseVersion: 3.11.2
44restbase2008.codfw.wmnet: a: ReleaseVersion: 3.11.2
45restbase2008.codfw.wmnet: b: ReleaseVersion: 3.11.2
46restbase2008.codfw.wmnet: c: ReleaseVersion: 3.11.2
47restbase2011.codfw.wmnet: a: ReleaseVersion: 3.11.2
48restbase2011.codfw.wmnet: b: ReleaseVersion: 3.11.2
49restbase2011.codfw.wmnet: c: ReleaseVersion: 3.11.2
50restbase2001.codfw.wmnet: a: ReleaseVersion: 3.11.2
51restbase2001.codfw.wmnet: b: ReleaseVersion: 3.11.2
52restbase2001.codfw.wmnet: c: ReleaseVersion: 3.11.2
53restbase2002.codfw.wmnet: a: ReleaseVersion: 3.11.2
54restbase2002.codfw.wmnet: b: ReleaseVersion: 3.11.2
55restbase2002.codfw.wmnet: c: ReleaseVersion: 3.11.2
56restbase2007.codfw.wmnet: a: ReleaseVersion: 3.11.2
57restbase2007.codfw.wmnet: b: ReleaseVersion: 3.11.2
58restbase2007.codfw.wmnet: c: ReleaseVersion: 3.11.2
59restbase2010.codfw.wmnet: a: ReleaseVersion: 3.11.2
60restbase2010.codfw.wmnet: b: ReleaseVersion: 3.11.2
61restbase2010.codfw.wmnet: c: ReleaseVersion: 3.11.2
62restbase2005.codfw.wmnet: a: ReleaseVersion: 3.11.2
63restbase2005.codfw.wmnet: b: ReleaseVersion: 3.11.2
64restbase2005.codfw.wmnet: c: ReleaseVersion: 3.11.2
65restbase2006.codfw.wmnet: a: ReleaseVersion: 3.11.2
66restbase2006.codfw.wmnet: b: ReleaseVersion: 3.11.2
67restbase2006.codfw.wmnet: c: ReleaseVersion: 3.11.2
68restbase2009.codfw.wmnet: a: ReleaseVersion: 3.11.2
69restbase2009.codfw.wmnet: b: ReleaseVersion: 3.11.2
70restbase2009.codfw.wmnet: c: ReleaseVersion: 3.11.2
71restbase2012.codfw.wmnet: a: ReleaseVersion: 3.11.2
72restbase2012.codfw.wmnet: b: ReleaseVersion: 3.11.2
73restbase2012.codfw.wmnet: c: ReleaseVersion: 3.11.2
74$

I see no issues, but we can leave this until Monday and bump T196745: Add Cassandra 3.11.2 package to internal APT repository once we're sure a rollback will not be necessary.

MoritzMuehlenhoff added a comment.Jun 11 2018, 7:01 AM

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

Eevans changed the status of subtask T196745: Add Cassandra 3.11.2 package to internal APT repository from Stalled to Open.Jun 11 2018, 2:53 PM
Eevans added a comment.Jun 11 2018, 7:40 PM
In T178905#4270966, @MoritzMuehlenhoff wrote:

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

Sorry, just noticed this; I have Puppet re-enabled everywhere now (I'd failed to notice that r/437298 had been merged, or I'd have done it sooner).

MoritzMuehlenhoff closed subtask T196745: Add Cassandra 3.11.2 package to internal APT repository as Resolved.Jun 12 2018, 11:49 AM
Eevans closed this task as Resolved.Jun 12 2018, 2:21 PM
• mobrovac edited projects, added Services (done); removed Patch-For-Review, Services (doing).Jun 13 2018, 9:37 AM
• Vvjjkkii reopened subtask T196745: Add Cassandra 3.11.2 package to internal APT repository as Open.Jul 1 2018, 1:05 AM
• mobrovac closed subtask T196745: Add Cassandra 3.11.2 package to internal APT repository as Resolved.Jul 1 2018, 10:17 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL