Page MenuHomePhabricator

Upgrade RESTBase cluster to Cassandra release: 3.11.2
Closed, ResolvedPublic

Description

3.11.1 has been released upstream, and contains a number of fixes that we previously patched our internal build of 3.11.0 for. It also contains many fixes for things that we were not patching into our build, and so should be tested prior to deployment.

3.11.2, when released, will contain a fix necessary to use OpenJDK versions >= 1.8.0_162


CompleteHostInstance
restbase1007.eqiad.wmneta
restbase1007.eqiad.wmnetb
restbase1007.eqiad.wmnetc
restbase1010.eqiad.wmneta
restbase1010.eqiad.wmnetb
restbase1010.eqiad.wmnetc
restbase1011.eqiad.wmneta
restbase1011.eqiad.wmnetb
restbase1011.eqiad.wmnetc
restbase1016.eqiad.wmneta
restbase1016.eqiad.wmnetb
restbase1016.eqiad.wmnetc
restbase1008.eqiad.wmneta
restbase1008.eqiad.wmnetb
restbase1008.eqiad.wmnetc
restbase1012.eqiad.wmneta
restbase1012.eqiad.wmnetb
restbase1012.eqiad.wmnetc
restbase1017.eqiad.wmneta
restbase1017.eqiad.wmnetb
restbase1017.eqiad.wmnetc
restbase1013.eqiad.wmneta
restbase1013.eqiad.wmnetb
restbase1013.eqiad.wmnetc
restbase1009.eqiad.wmneta
restbase1009.eqiad.wmnetb
restbase1009.eqiad.wmnetc
restbase1014.eqiad.wmneta
restbase1014.eqiad.wmnetb
restbase1014.eqiad.wmnetc
restbase1015.eqiad.wmneta
restbase1015.eqiad.wmnetb
restbase1015.eqiad.wmnetc
restbase1018.eqiad.wmneta
restbase1018.eqiad.wmnetb
restbase1018.eqiad.wmnetc
restbase2003.codfw.wmneta
restbase2003.codfw.wmnetb
restbase2003.codfw.wmnetc
restbase2004.codfw.wmneta
restbase2004.codfw.wmnetb
restbase2004.codfw.wmnetc
restbase2008.codfw.wmneta
restbase2008.codfw.wmnetb
restbase2008.codfw.wmnetc
restbase2011.codfw.wmneta
restbase2011.codfw.wmnetb
restbase2011.codfw.wmnetc
restbase2001.codfw.wmneta
restbase2001.codfw.wmnetb
restbase2001.codfw.wmnetc
restbase2002.codfw.wmneta
restbase2002.codfw.wmnetb
restbase2002.codfw.wmnetc
restbase2007.codfw.wmneta
restbase2007.codfw.wmnetb
restbase2007.codfw.wmnetc
restbase2010.codfw.wmneta
restbase2010.codfw.wmnetb
restbase2010.codfw.wmnetc
restbase2005.codfw.wmneta
restbase2005.codfw.wmnetb
restbase2005.codfw.wmnetc
restbase2006.codfw.wmneta
restbase2006.codfw.wmnetb
restbase2006.codfw.wmnetc
restbase2009.codfw.wmneta
restbase2009.codfw.wmnetb
restbase2009.codfw.wmnetc
restbase2012.codfw.wmneta
restbase2012.codfw.wmnetb
restbase2012.codfw.wmnetc

Event Timeline

Eevans created this task.Oct 24 2017, 2:45 PM
Eevans moved this task from Backlog to Next on the User-Eevans board.Dec 5 2017, 3:39 PM
Eevans renamed this task from Evaluate new upstream Cassandra release: 3.11.1 to Evaluate new upstream Cassandra release: 3.11.2.Feb 2 2018, 5:37 PM
Eevans updated the task description. (Show Details)

Times up, 8u162-b12-1~bpo8+1 has been made available.

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:12:14Z] <urandom> upgrading restbase-dev1004-a to cassandra 3.11.2 (canary) -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:17:17Z] <urandom> upgrading restbase-dev1004-b to cassandra 3.11.2 (canary) -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:52:33Z] <urandom> upgrading restbase-dev1005-{a,b} to cassandra 3.11.2 -- T178905

Opened CASSANDRA-14355 to track a memory leak encountered in the dev environment (reproduced on 3.11.0 and 3.11.2).

Eevans renamed this task from Evaluate new upstream Cassandra release: 3.11.2 to Upgrade RESTBase cluster to Cassandra release: 3.11.2.Apr 13 2018, 1:29 PM
Eevans raised the priority of this task from Low to Normal.
Eevans moved this task from Next to In-Progress on the User-Eevans board.Jun 4 2018, 4:39 PM
Eevans edited projects, added Services (doing); removed Services (next).

Change 437298 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: update configuration for 3.11.2

https://gerrit.wikimedia.org/r/437298

Change 437298 merged by Elukey:
[operations/puppet@production] cassandra: update configuration for 3.11.2

https://gerrit.wikimedia.org/r/437298

Mentioned in SAL (#wikimedia-operations) [2018-06-05T14:52:50Z] <urandom> restarting restbase-dev1004 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T15:58:22Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1004 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T16:56:15Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1005 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T17:00:56Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1006 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T17:45:08Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001 (canary) - T178905

Eevans updated the task description. (Show Details)Jun 5 2018, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T19:35:21Z] <urandom> upgrade Cassandra to 3.11.2, restbase1007-a (canary) - T178905

Eevans updated the task description. (Show Details)Jun 5 2018, 10:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:06:17Z] <urandom> upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:27:23Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001-{b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:54:25Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 5:00 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T17:01:20Z] <urandom> upgrade Cassandra to 3.11.2, restbase2007-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 5:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T17:59:22Z] <urandom> upgrade Cassandra to 3.11.2, restbase2010-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 6:16 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T18:19:00Z] <urandom> upgrade Cassandra to 3.11.2, restbase2003-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 6:53 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T18:54:50Z] <urandom> upgrade Cassandra to 3.11.2, restbase2004-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T19:38:08Z] <urandom> upgrade Cassandra to 3.11.2, restbase2008-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 8:15 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T20:17:27Z] <urandom> upgrade Cassandra to 3.11.2, restbase2011-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 8:41 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T20:42:19Z] <urandom> upgrade Cassandra to 3.11.2, restbase2005-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 10:02 PM
Eevans updated the task description. (Show Details)Jun 6 2018, 11:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T23:23:46Z] <urandom> upgrade Cassandra to 3.11.2, restbase2009-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 11:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T23:51:48Z] <urandom> upgrade Cassandra to 3.11.2, restbase2012-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 7 2018, 12:22 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T15:38:53Z] <urandom> upgrade Cassandra to 3.11.2, restbase1010-{a,b,c} - T178905

Change 438035 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: upgrade 3.x version to 3.11.2

https://gerrit.wikimedia.org/r/438035

Eevans updated the task description. (Show Details)Jun 7 2018, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T16:21:54Z] <urandom> rolling Cassandra restart, restbase1007 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T16:45:45Z] <urandom> rolling Cassandra restart, restbase2001, restbase2002, restbase2007 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T18:20:40Z] <urandom> restarting Cassandra, restbase2003-c -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T18:28:20Z] <urandom> rolling Cassandra restart, restbase2004, restbase2008, restbase2011 -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T19:47:23Z] <urandom> rolling Cassandra restart, restbase2005, restbase2006, restbase2012 -- T178905

Change 438035 merged by Dzahn:
[operations/puppet@production] cassandra: upgrade 3.x version to 3.11.2

https://gerrit.wikimedia.org/r/438035

Mentioned in SAL (#wikimedia-operations) [2018-06-08T14:17:21Z] <urandom> upgrade Cassandra to 3.11.2, restbase1011 & restbase1016 - T178905

Eevans updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2018-06-08T15:00:09Z] <urandom> upgrade Cassandra to 3.11.2, restbase1008 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-08T15:35:26Z] <urandom> upgrade Cassandra to 3.11.2, restbase1012 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 5:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T18:02:33Z] <urandom> upgrade Cassandra to 3.11.2, restbase1017 & restbase1013 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 6:44 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T18:49:16Z] <urandom> upgrade Cassandra to 3.11.2, restbase1009 & restbase1014 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 7:32 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T19:36:00Z] <urandom> upgrade Cassandra to 3.11.2, restbase1015 & restbase1018 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 8:21 PM
Eevans added a comment.Jun 8 2018, 8:33 PM

The upgrade is complete:

1$ cdsh -- c-foreach-nt version
2restbase1007.eqiad.wmnet: a: ReleaseVersion: 3.11.2
3restbase1007.eqiad.wmnet: b: ReleaseVersion: 3.11.2
4restbase1007.eqiad.wmnet: c: ReleaseVersion: 3.11.2
5restbase1010.eqiad.wmnet: a: ReleaseVersion: 3.11.2
6restbase1010.eqiad.wmnet: b: ReleaseVersion: 3.11.2
7restbase1010.eqiad.wmnet: c: ReleaseVersion: 3.11.2
8restbase1011.eqiad.wmnet: a: ReleaseVersion: 3.11.2
9restbase1011.eqiad.wmnet: b: ReleaseVersion: 3.11.2
10restbase1011.eqiad.wmnet: c: ReleaseVersion: 3.11.2
11restbase1016.eqiad.wmnet: a: ReleaseVersion: 3.11.2
12restbase1016.eqiad.wmnet: b: ReleaseVersion: 3.11.2
13restbase1016.eqiad.wmnet: c: ReleaseVersion: 3.11.2
14restbase1008.eqiad.wmnet: a: ReleaseVersion: 3.11.2
15restbase1008.eqiad.wmnet: b: ReleaseVersion: 3.11.2
16restbase1008.eqiad.wmnet: c: ReleaseVersion: 3.11.2
17restbase1012.eqiad.wmnet: a: ReleaseVersion: 3.11.2
18restbase1012.eqiad.wmnet: b: ReleaseVersion: 3.11.2
19restbase1012.eqiad.wmnet: c: ReleaseVersion: 3.11.2
20restbase1017.eqiad.wmnet: a: ReleaseVersion: 3.11.2
21restbase1017.eqiad.wmnet: b: ReleaseVersion: 3.11.2
22restbase1017.eqiad.wmnet: c: ReleaseVersion: 3.11.2
23restbase1013.eqiad.wmnet: a: ReleaseVersion: 3.11.2
24restbase1013.eqiad.wmnet: b: ReleaseVersion: 3.11.2
25restbase1013.eqiad.wmnet: c: ReleaseVersion: 3.11.2
26restbase1009.eqiad.wmnet: a: ReleaseVersion: 3.11.2
27restbase1009.eqiad.wmnet: b: ReleaseVersion: 3.11.2
28restbase1009.eqiad.wmnet: c: ReleaseVersion: 3.11.2
29restbase1014.eqiad.wmnet: a: ReleaseVersion: 3.11.2
30restbase1014.eqiad.wmnet: b: ReleaseVersion: 3.11.2
31restbase1014.eqiad.wmnet: c: ReleaseVersion: 3.11.2
32restbase1015.eqiad.wmnet: a: ReleaseVersion: 3.11.2
33restbase1015.eqiad.wmnet: b: ReleaseVersion: 3.11.2
34restbase1015.eqiad.wmnet: c: ReleaseVersion: 3.11.2
35restbase1018.eqiad.wmnet: a: ReleaseVersion: 3.11.2
36restbase1018.eqiad.wmnet: b: ReleaseVersion: 3.11.2
37restbase1018.eqiad.wmnet: c: ReleaseVersion: 3.11.2
38restbase2003.codfw.wmnet: a: ReleaseVersion: 3.11.2
39restbase2003.codfw.wmnet: b: ReleaseVersion: 3.11.2
40restbase2003.codfw.wmnet: c: ReleaseVersion: 3.11.2
41restbase2004.codfw.wmnet: a: ReleaseVersion: 3.11.2
42restbase2004.codfw.wmnet: b: ReleaseVersion: 3.11.2
43restbase2004.codfw.wmnet: c: ReleaseVersion: 3.11.2
44restbase2008.codfw.wmnet: a: ReleaseVersion: 3.11.2
45restbase2008.codfw.wmnet: b: ReleaseVersion: 3.11.2
46restbase2008.codfw.wmnet: c: ReleaseVersion: 3.11.2
47restbase2011.codfw.wmnet: a: ReleaseVersion: 3.11.2
48restbase2011.codfw.wmnet: b: ReleaseVersion: 3.11.2
49restbase2011.codfw.wmnet: c: ReleaseVersion: 3.11.2
50restbase2001.codfw.wmnet: a: ReleaseVersion: 3.11.2
51restbase2001.codfw.wmnet: b: ReleaseVersion: 3.11.2
52restbase2001.codfw.wmnet: c: ReleaseVersion: 3.11.2
53restbase2002.codfw.wmnet: a: ReleaseVersion: 3.11.2
54restbase2002.codfw.wmnet: b: ReleaseVersion: 3.11.2
55restbase2002.codfw.wmnet: c: ReleaseVersion: 3.11.2
56restbase2007.codfw.wmnet: a: ReleaseVersion: 3.11.2
57restbase2007.codfw.wmnet: b: ReleaseVersion: 3.11.2
58restbase2007.codfw.wmnet: c: ReleaseVersion: 3.11.2
59restbase2010.codfw.wmnet: a: ReleaseVersion: 3.11.2
60restbase2010.codfw.wmnet: b: ReleaseVersion: 3.11.2
61restbase2010.codfw.wmnet: c: ReleaseVersion: 3.11.2
62restbase2005.codfw.wmnet: a: ReleaseVersion: 3.11.2
63restbase2005.codfw.wmnet: b: ReleaseVersion: 3.11.2
64restbase2005.codfw.wmnet: c: ReleaseVersion: 3.11.2
65restbase2006.codfw.wmnet: a: ReleaseVersion: 3.11.2
66restbase2006.codfw.wmnet: b: ReleaseVersion: 3.11.2
67restbase2006.codfw.wmnet: c: ReleaseVersion: 3.11.2
68restbase2009.codfw.wmnet: a: ReleaseVersion: 3.11.2
69restbase2009.codfw.wmnet: b: ReleaseVersion: 3.11.2
70restbase2009.codfw.wmnet: c: ReleaseVersion: 3.11.2
71restbase2012.codfw.wmnet: a: ReleaseVersion: 3.11.2
72restbase2012.codfw.wmnet: b: ReleaseVersion: 3.11.2
73restbase2012.codfw.wmnet: c: ReleaseVersion: 3.11.2
74$

I see no issues, but we can leave this until Monday and bump T196745: Add Cassandra 3.11.2 package to internal APT repository once we're sure a rollback will not be necessary.

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

Sorry, just noticed this; I have Puppet re-enabled everywhere now (I'd failed to notice that r/437298 had been merged, or I'd have done it sooner).

Eevans closed this task as Resolved.Jun 12 2018, 2:21 PM