Upgrade RESTBase cluster to Cassandra release: 3.11.2
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Eevans
	Oct 24 2017, 2:45 PM

Description

3.11.1 has been released upstream, and contains a number of fixes that we previously patched our internal build of 3.11.0 for. It also contains many fixes for things that we were not patching into our build, and so should be tested prior to deployment.

3.11.2, when released, will contain a fix necessary to use OpenJDK versions >= 1.8.0_162

Complete	Host	Instance
	restbase1007.eqiad.wmnet	a
	restbase1007.eqiad.wmnet	b
	restbase1007.eqiad.wmnet	c
	restbase1010.eqiad.wmnet	a
	restbase1010.eqiad.wmnet	b
	restbase1010.eqiad.wmnet	c
	restbase1011.eqiad.wmnet	a
	restbase1011.eqiad.wmnet	b
	restbase1011.eqiad.wmnet	c
	restbase1016.eqiad.wmnet	a
	restbase1016.eqiad.wmnet	b
	restbase1016.eqiad.wmnet	c
	restbase1008.eqiad.wmnet	a
	restbase1008.eqiad.wmnet	b
	restbase1008.eqiad.wmnet	c
	restbase1012.eqiad.wmnet	a
	restbase1012.eqiad.wmnet	b
	restbase1012.eqiad.wmnet	c
	restbase1017.eqiad.wmnet	a
	restbase1017.eqiad.wmnet	b
	restbase1017.eqiad.wmnet	c
	restbase1013.eqiad.wmnet	a
	restbase1013.eqiad.wmnet	b
	restbase1013.eqiad.wmnet	c
	restbase1009.eqiad.wmnet	a
	restbase1009.eqiad.wmnet	b
	restbase1009.eqiad.wmnet	c
	restbase1014.eqiad.wmnet	a
	restbase1014.eqiad.wmnet	b
	restbase1014.eqiad.wmnet	c
	restbase1015.eqiad.wmnet	a
	restbase1015.eqiad.wmnet	b
	restbase1015.eqiad.wmnet	c
	restbase1018.eqiad.wmnet	a
	restbase1018.eqiad.wmnet	b
	restbase1018.eqiad.wmnet	c
	restbase2003.codfw.wmnet	a
	restbase2003.codfw.wmnet	b
	restbase2003.codfw.wmnet	c
	restbase2004.codfw.wmnet	a
	restbase2004.codfw.wmnet	b
	restbase2004.codfw.wmnet	c
	restbase2008.codfw.wmnet	a
	restbase2008.codfw.wmnet	b
	restbase2008.codfw.wmnet	c
	restbase2011.codfw.wmnet	a
	restbase2011.codfw.wmnet	b
	restbase2011.codfw.wmnet	c
	restbase2001.codfw.wmnet	a
	restbase2001.codfw.wmnet	b
	restbase2001.codfw.wmnet	c
	restbase2002.codfw.wmnet	a
	restbase2002.codfw.wmnet	b
	restbase2002.codfw.wmnet	c
	restbase2007.codfw.wmnet	a
	restbase2007.codfw.wmnet	b
	restbase2007.codfw.wmnet	c
	restbase2010.codfw.wmnet	a
	restbase2010.codfw.wmnet	b
	restbase2010.codfw.wmnet	c
	restbase2005.codfw.wmnet	a
	restbase2005.codfw.wmnet	b
	restbase2005.codfw.wmnet	c
	restbase2006.codfw.wmnet	a
	restbase2006.codfw.wmnet	b
	restbase2006.codfw.wmnet	c
	restbase2009.codfw.wmnet	a
	restbase2009.codfw.wmnet	b
	restbase2009.codfw.wmnet	c
	restbase2012.codfw.wmnet	a
	restbase2012.codfw.wmnet	b
	restbase2012.codfw.wmnet	c

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	cassandra: upgrade 3.x version to 3.11.2	operations/puppet	production	+2 -2
	cassandra: update configuration for 3.11.2	operations/puppet	production	+24 -1

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T324931 Clean up open RESTBase related tickets
Open	None	T262315 <CORE TECHNOLOGY> API Migration & RESTBase Sunset
Resolved	DAlangi_WMF	T324678 Migrate proton (chromium-render) away from restbase
Open	None	T167603 Any Chinese Wiki's projects about "Download as PDF" can not auto change to Simplified Chinese or Traditional Chinese
Resolved	ovasileva	T147553 [EPIC] Page previews broken on many projects
Open	ovasileva	T244262 [Epic] Enable page previews and reference previews as a beta feature on all projects
Open	None	T111231 Page previews for Wikidata
Invalid	None	T148854 Use RESTBase for zhwiki
Resolved	• Pchelolo	T188164 Popups don‘t support language variant conversion and {{lang}} template
Resolved	• mobrovac	T190689 FY17/18 Q4 Program 7 Services Goal: Language variants support
Open	None	T177621 Apache Cassandra Tracking
Resolved	Eevans	T178905 Upgrade RESTBase cluster to Cassandra release: 3.11.2
Resolved	Eevans	T196745 Add Cassandra 3.11.2 package to internal APT repository

Event Timeline

Eevans created this task.Oct 24 2017, 2:45 PM

• mobrovac edited projects, added Services (next); removed Services (watching).Oct 24 2017, 2:47 PM

Eevans added a project: User-Eevans.Oct 25 2017, 6:19 PM

Eevans moved this task from Backlog to Next on the User-Eevans board.Dec 5 2017, 3:39 PM

Eevans renamed this task from Evaluate new upstream Cassandra release: 3.11.1 to Evaluate new upstream Cassandra release: 3.11.2.Feb 2 2018, 5:37 PM

Eevans updated the task description. (Show Details)

Eevans mentioned this in T186360: Cassandra 3.11.0 incompatible with OpenJDK >= 1.8.0_162.Feb 2 2018, 9:04 PM

Ricordisamoa subscribed.Feb 2 2018, 10:39 PM

Times up, 8u162-b12-1~bpo8+1 has been made available.

MoritzMuehlenhoff subscribed.Mar 19 2018, 6:06 PM

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:12:14Z] <urandom> upgrading restbase-dev1004-a to cassandra 3.11.2 (canary) -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:17:17Z] <urandom> upgrading restbase-dev1004-b to cassandra 3.11.2 (canary) -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-03-28T18:52:33Z] <urandom> upgrading restbase-dev1005-{a,b} to cassandra 3.11.2 -- T178905

Opened CASSANDRA-14355 to track a memory leak encountered in the dev environment (reproduced on 3.11.0 and 3.11.2).

Eevans renamed this task from Evaluate new upstream Cassandra release: 3.11.2 to Upgrade RESTBase cluster to Cassandra release: 3.11.2.Apr 13 2018, 1:29 PM

Eevans raised the priority of this task from Low to Medium.

Eevans moved this task from Next to In-Progress on the User-Eevans board.Jun 4 2018, 4:39 PM

Eevans edited projects, added Services (doing); removed Services (next).

Change 437298 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: update configuration for 3.11.2

https://gerrit.wikimedia.org/r/437298

gerritbot added a project: Patch-For-Review.Jun 4 2018, 7:05 PM

• mobrovac added a parent task: T190689: FY17/18 Q4 Program 7 Services Goal: Language variants support.Jun 5 2018, 9:16 AM

Change 437298 merged by Elukey:
[operations/puppet@production] cassandra: update configuration for 3.11.2

https://gerrit.wikimedia.org/r/437298

Mentioned in SAL (#wikimedia-operations) [2018-06-05T14:52:50Z] <urandom> restarting restbase-dev1004 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T15:58:22Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1004 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T16:56:15Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1005 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T17:00:56Z] <urandom> upgrade Cassandra to 3.11.2, restbase-dev1006 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-05T17:45:08Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001 (canary) - T178905

Eevans updated the task description. (Show Details)Jun 5 2018, 5:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-05T19:35:21Z] <urandom> upgrade Cassandra to 3.11.2, restbase1007-a (canary) - T178905

Eevans updated the task description. (Show Details)Jun 5 2018, 10:30 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:06:17Z] <urandom> upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:27:23Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001-{b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T15:54:25Z] <urandom> upgrade Cassandra to 3.11.2, restbase2001-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 5:00 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T17:01:20Z] <urandom> upgrade Cassandra to 3.11.2, restbase2007-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 5:42 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T17:59:22Z] <urandom> upgrade Cassandra to 3.11.2, restbase2010-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 6:16 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T18:19:00Z] <urandom> upgrade Cassandra to 3.11.2, restbase2003-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 6:53 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T18:54:50Z] <urandom> upgrade Cassandra to 3.11.2, restbase2004-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T19:38:08Z] <urandom> upgrade Cassandra to 3.11.2, restbase2008-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 8:15 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T20:17:27Z] <urandom> upgrade Cassandra to 3.11.2, restbase2011-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 8:41 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T20:42:19Z] <urandom> upgrade Cassandra to 3.11.2, restbase2005-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 10:02 PM

Eevans updated the task description. (Show Details)Jun 6 2018, 11:23 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T23:23:46Z] <urandom> upgrade Cassandra to 3.11.2, restbase2009-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 6 2018, 11:50 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-06T23:51:48Z] <urandom> upgrade Cassandra to 3.11.2, restbase2012-{a,b,c} - T178905

Eevans updated the task description. (Show Details)Jun 7 2018, 12:22 AM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T15:38:53Z] <urandom> upgrade Cassandra to 3.11.2, restbase1010-{a,b,c} - T178905

Change 438035 had a related patch set uploaded (by Eevans; owner: Eevans):
[operations/puppet@production] cassandra: upgrade 3.x version to 3.11.2

https://gerrit.wikimedia.org/r/438035

Eevans updated the task description. (Show Details)Jun 7 2018, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-07T16:21:54Z] <urandom> rolling Cassandra restart, restbase1007 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T16:45:45Z] <urandom> rolling Cassandra restart, restbase2001, restbase2002, restbase2007 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T18:20:40Z] <urandom> restarting Cassandra, restbase2003-c -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T18:28:20Z] <urandom> rolling Cassandra restart, restbase2004, restbase2008, restbase2011 -- T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-07T19:47:23Z] <urandom> rolling Cassandra restart, restbase2005, restbase2006, restbase2012 -- T178905

Change 438035 merged by Dzahn:
[operations/puppet@production] cassandra: upgrade 3.x version to 3.11.2

https://gerrit.wikimedia.org/r/438035

Mentioned in SAL (#wikimedia-operations) [2018-06-08T14:17:21Z] <urandom> upgrade Cassandra to 3.11.2, restbase1011 & restbase1016 - T178905

Eevans changed the status of subtask T196745: Add Cassandra 3.11.2 package to internal APT repository from Open to Stalled.Jun 8 2018, 2:57 PM

Eevans updated the task description. (Show Details)

Mentioned in SAL (#wikimedia-operations) [2018-06-08T15:00:09Z] <urandom> upgrade Cassandra to 3.11.2, restbase1008 - T178905

Mentioned in SAL (#wikimedia-operations) [2018-06-08T15:35:26Z] <urandom> upgrade Cassandra to 3.11.2, restbase1012 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 5:58 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T18:02:33Z] <urandom> upgrade Cassandra to 3.11.2, restbase1017 & restbase1013 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 6:44 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T18:49:16Z] <urandom> upgrade Cassandra to 3.11.2, restbase1009 & restbase1014 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 7:32 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-08T19:36:00Z] <urandom> upgrade Cassandra to 3.11.2, restbase1015 & restbase1018 - T178905

Eevans updated the task description. (Show Details)Jun 8 2018, 8:21 PM

The upgrade is complete:

P7229 Masterwork From Distant Lands

1	$ cdsh -- c-foreach-nt version
2	restbase1007.eqiad.wmnet: a: ReleaseVersion: 3.11.2
3	restbase1007.eqiad.wmnet: b: ReleaseVersion: 3.11.2
4	restbase1007.eqiad.wmnet: c: ReleaseVersion: 3.11.2
5	restbase1010.eqiad.wmnet: a: ReleaseVersion: 3.11.2
6	restbase1010.eqiad.wmnet: b: ReleaseVersion: 3.11.2
7	restbase1010.eqiad.wmnet: c: ReleaseVersion: 3.11.2
8	restbase1011.eqiad.wmnet: a: ReleaseVersion: 3.11.2
9	restbase1011.eqiad.wmnet: b: ReleaseVersion: 3.11.2
10	restbase1011.eqiad.wmnet: c: ReleaseVersion: 3.11.2
11	restbase1016.eqiad.wmnet: a: ReleaseVersion: 3.11.2
12	restbase1016.eqiad.wmnet: b: ReleaseVersion: 3.11.2
13	restbase1016.eqiad.wmnet: c: ReleaseVersion: 3.11.2
14	restbase1008.eqiad.wmnet: a: ReleaseVersion: 3.11.2
15	restbase1008.eqiad.wmnet: b: ReleaseVersion: 3.11.2
16	restbase1008.eqiad.wmnet: c: ReleaseVersion: 3.11.2
17	restbase1012.eqiad.wmnet: a: ReleaseVersion: 3.11.2
18	restbase1012.eqiad.wmnet: b: ReleaseVersion: 3.11.2
19	restbase1012.eqiad.wmnet: c: ReleaseVersion: 3.11.2
20	restbase1017.eqiad.wmnet: a: ReleaseVersion: 3.11.2
21	restbase1017.eqiad.wmnet: b: ReleaseVersion: 3.11.2
22	restbase1017.eqiad.wmnet: c: ReleaseVersion: 3.11.2
23	restbase1013.eqiad.wmnet: a: ReleaseVersion: 3.11.2
24	restbase1013.eqiad.wmnet: b: ReleaseVersion: 3.11.2
25	restbase1013.eqiad.wmnet: c: ReleaseVersion: 3.11.2
26	restbase1009.eqiad.wmnet: a: ReleaseVersion: 3.11.2
27	restbase1009.eqiad.wmnet: b: ReleaseVersion: 3.11.2
28	restbase1009.eqiad.wmnet: c: ReleaseVersion: 3.11.2
29	restbase1014.eqiad.wmnet: a: ReleaseVersion: 3.11.2
30	restbase1014.eqiad.wmnet: b: ReleaseVersion: 3.11.2
31	restbase1014.eqiad.wmnet: c: ReleaseVersion: 3.11.2
32	restbase1015.eqiad.wmnet: a: ReleaseVersion: 3.11.2
33	restbase1015.eqiad.wmnet: b: ReleaseVersion: 3.11.2
34	restbase1015.eqiad.wmnet: c: ReleaseVersion: 3.11.2
35	restbase1018.eqiad.wmnet: a: ReleaseVersion: 3.11.2
36	restbase1018.eqiad.wmnet: b: ReleaseVersion: 3.11.2
37	restbase1018.eqiad.wmnet: c: ReleaseVersion: 3.11.2
38	restbase2003.codfw.wmnet: a: ReleaseVersion: 3.11.2
39	restbase2003.codfw.wmnet: b: ReleaseVersion: 3.11.2
40	restbase2003.codfw.wmnet: c: ReleaseVersion: 3.11.2
41	restbase2004.codfw.wmnet: a: ReleaseVersion: 3.11.2
42	restbase2004.codfw.wmnet: b: ReleaseVersion: 3.11.2
43	restbase2004.codfw.wmnet: c: ReleaseVersion: 3.11.2
44	restbase2008.codfw.wmnet: a: ReleaseVersion: 3.11.2
45	restbase2008.codfw.wmnet: b: ReleaseVersion: 3.11.2
46	restbase2008.codfw.wmnet: c: ReleaseVersion: 3.11.2
47	restbase2011.codfw.wmnet: a: ReleaseVersion: 3.11.2
48	restbase2011.codfw.wmnet: b: ReleaseVersion: 3.11.2
49	restbase2011.codfw.wmnet: c: ReleaseVersion: 3.11.2
50	restbase2001.codfw.wmnet: a: ReleaseVersion: 3.11.2
51	restbase2001.codfw.wmnet: b: ReleaseVersion: 3.11.2
52	restbase2001.codfw.wmnet: c: ReleaseVersion: 3.11.2
53	restbase2002.codfw.wmnet: a: ReleaseVersion: 3.11.2
54	restbase2002.codfw.wmnet: b: ReleaseVersion: 3.11.2
55	restbase2002.codfw.wmnet: c: ReleaseVersion: 3.11.2
56	restbase2007.codfw.wmnet: a: ReleaseVersion: 3.11.2
57	restbase2007.codfw.wmnet: b: ReleaseVersion: 3.11.2
58	restbase2007.codfw.wmnet: c: ReleaseVersion: 3.11.2
59	restbase2010.codfw.wmnet: a: ReleaseVersion: 3.11.2
60	restbase2010.codfw.wmnet: b: ReleaseVersion: 3.11.2
61	restbase2010.codfw.wmnet: c: ReleaseVersion: 3.11.2
62	restbase2005.codfw.wmnet: a: ReleaseVersion: 3.11.2
63	restbase2005.codfw.wmnet: b: ReleaseVersion: 3.11.2
64	restbase2005.codfw.wmnet: c: ReleaseVersion: 3.11.2
65	restbase2006.codfw.wmnet: a: ReleaseVersion: 3.11.2
66	restbase2006.codfw.wmnet: b: ReleaseVersion: 3.11.2
67	restbase2006.codfw.wmnet: c: ReleaseVersion: 3.11.2
68	restbase2009.codfw.wmnet: a: ReleaseVersion: 3.11.2
69	restbase2009.codfw.wmnet: b: ReleaseVersion: 3.11.2
70	restbase2009.codfw.wmnet: c: ReleaseVersion: 3.11.2
71	restbase2012.codfw.wmnet: a: ReleaseVersion: 3.11.2
72	restbase2012.codfw.wmnet: b: ReleaseVersion: 3.11.2
73	restbase2012.codfw.wmnet: c: ReleaseVersion: 3.11.2
74	$

I see no issues, but we can leave this until Monday and bump T196745: Add Cassandra 3.11.2 package to internal APT repository once we're sure a rollback will not be necessary.

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

Eevans changed the status of subtask T196745: Add Cassandra 3.11.2 package to internal APT repository from Stalled to Open.Jun 11 2018, 2:53 PM

In T178905#4270966, @MoritzMuehlenhoff wrote:

restbase-dev1004 and -1006 still have puppet disabled with a note about upgrading Cassandra, safe to re-enable? Ping me when you're around and I'll upload the new 3.11.2 debs to apt.wikimedia.org

Sorry, just noticed this; I have Puppet re-enabled everywhere now (I'd failed to notice that r/437298 had been merged, or I'd have done it sooner).

MoritzMuehlenhoff closed subtask T196745: Add Cassandra 3.11.2 package to internal APT repository as Resolved.Jun 12 2018, 11:49 AM

Eevans closed this task as Resolved.Jun 12 2018, 2:21 PM

• mobrovac edited projects, added Services (done); removed Patch-For-Review, Services (doing).Jun 13 2018, 9:37 AM

• Vvjjkkii reopened subtask T196745: Add Cassandra 3.11.2 package to internal APT repository as Open.Jul 1 2018, 1:05 AM

• mobrovac closed subtask T196745: Add Cassandra 3.11.2 package to internal APT repository as Resolved.Jul 1 2018, 10:17 AM