Page MenuHomePhabricator

Cassandra 2.2.6
Closed, ResolvedPublic

Description

Cassandra 2.2.5 should be evaluated/tested for use in production.

Given the (presumably) low delta to 2.1.12, (and the maturity/stability of the 2.1.x branch at this point), it might make sense to first consider an upgrade to 2.1.13 while we spend the time evaluating 2.2.5.


2.1.13

Possible issues addressed:

Timeline

Per #2075200, after the bootstrap of 1010 (2016-03-07?).

2.2.5 2.2.6

Interesting changes (non-exhaustive)

  • JSON
  • UDFs
  • UDAs
  • Role-based access control
  • Native protocol v4
  • Resumable bootstraps
  • Direct ByteBuffers for decompression reads
  • Compressed commitlog
  • Message coalescing
  • Async debug logging
  • Improved repair concurrency/paralleism
  • Full repairs mark SSTables as repaired
  • Incremental repairs as default

Upgrading

Possible gotchas
  • Conflict with system installed libjna-java
  • Use of -XX:+PerfDisableSharedMem may break cassandra-metrics-collector auto-discovery
  • Conflict of Jackson dependencies w/ those needed by logstash
  • System keyspace schema migration for RBAC (Role-based access control)
Configuration
  • cassandra-env.sh
  • cassandra.yaml
    • From baseline cassandra.yaml: P2700
    • From WMF production cassandra.yaml: P2703
  • logback.xml
Timeline

TBD (after: T125906: Evaluate Brotli compression for Cassandra)


Current Status (v2.2.5 v2.2.6 upgrade)

There is a work-in-progress Puppet changeset here. I believe it is mostly OK at this point (a few nits remain).

Between 2.1.13 and 2.2.5, metrics in Cassandra were refactored to wrap the Dropwizard mbeans in Cassandra delegator objects. Doing this changed the names (obviously), and broke cassandra-metrics-collector. The fix for that is here, though a little work is still needed to the Puppet patch to (conditionally) setup the systemd unit on machines running the 2.2 series (it's meant to keep working as-is for 2.1 machines).

In addition to the breakage in cassandra-metrics-collector, there is also some breakage that results from the metric type changing (in certain select cases), for example from Gauge to Counter (the JMX attribute changes from value to count, and so the Graphite metric names changes accordingly). I'm still assessing the full extent of this class of breakage. Options will include papering over this in cassandra-metrics-collector, or letting the chips fall as they may, and dealing with the fallout in the dashboards.

In addition to the breakage in cassandra-metrics-collector, there are some differences to the available metrics between the two versions, a detailed analysis follows:

1o.a.c.metrics.ClientRequest.CASRead.Failures.15MinuteRate
2o.a.c.metrics.ClientRequest.CASRead.Failures.1MinuteRate
3o.a.c.metrics.ClientRequest.CASRead.Failures.5MinuteRate
4o.a.c.metrics.ClientRequest.CASRead.Failures.count
5o.a.c.metrics.ClientRequest.CASRead.Failures.meanRate
6o.a.c.metrics.ClientRequest.CASWrite.Failures.15MinuteRate
7o.a.c.metrics.ClientRequest.CASWrite.Failures.1MinuteRate
8o.a.c.metrics.ClientRequest.CASWrite.Failures.5MinuteRate
9o.a.c.metrics.ClientRequest.CASWrite.Failures.count
10o.a.c.metrics.ClientRequest.CASWrite.Failures.meanRate
11o.a.c.metrics.ClientRequest.RangeSlice.Failures.15MinuteRate
12o.a.c.metrics.ClientRequest.RangeSlice.Failures.1MinuteRate
13o.a.c.metrics.ClientRequest.RangeSlice.Failures.5MinuteRate
14o.a.c.metrics.ClientRequest.RangeSlice.Failures.count
15o.a.c.metrics.ClientRequest.RangeSlice.Failures.meanRate
16o.a.c.metrics.ClientRequest.Read.Failures.15MinuteRate
17o.a.c.metrics.ClientRequest.Read.Failures.1MinuteRate
18o.a.c.metrics.ClientRequest.Read.Failures.5MinuteRate
19o.a.c.metrics.ClientRequest.Read.Failures.count
20o.a.c.metrics.ClientRequest.Read.Failures.meanRate
21o.a.c.metrics.ClientRequest.Write.Failures.15MinuteRate
22o.a.c.metrics.ClientRequest.Write.Failures.1MinuteRate
23o.a.c.metrics.ClientRequest.Write.Failures.5MinuteRate
24o.a.c.metrics.ClientRequest.Write.Failures.count
25o.a.c.metrics.ClientRequest.Write.Failures.meanRate
26o.a.c.metrics.Connection.127.0.0.3.GossipMessageCompletedTasks.value
27o.a.c.metrics.Connection.127.0.0.3.GossipMessageDroppedTasks.value
28o.a.c.metrics.Connection.127.0.0.3.GossipMessagePendingTasks.value
29o.a.c.metrics.Connection.127.0.0.3.LargeMessageCompletedTasks.value
30o.a.c.metrics.Connection.127.0.0.3.LargeMessageDroppedTasks.value
31o.a.c.metrics.Connection.127.0.0.3.LargeMessagePendingTasks.value
32o.a.c.metrics.Connection.127.0.0.3.SmallMessageCompletedTasks.value
33o.a.c.metrics.Connection.127.0.0.3.SmallMessageDroppedTasks.value
34o.a.c.metrics.Connection.127.0.0.3.SmallMessagePendingTasks.value

1o.a.c.metrics.ThreadPools.request.CounterMutationStage.CurrentlyBlockedTasks.count
2o.a.c.metrics.ThreadPools.request.CounterMutationStage.TotalBlockedTasks.count
3o.a.c.metrics.ThreadPools.request.MutationStage.CurrentlyBlockedTasks.count
4o.a.c.metrics.ThreadPools.request.MutationStage.TotalBlockedTasks.count
5o.a.c.metrics.ThreadPools.request.ReadStage.CurrentlyBlockedTasks.count
6o.a.c.metrics.ThreadPools.request.ReadStage.TotalBlockedTasks.count
7o.a.c.metrics.ThreadPools.request.RequestResponseStage.CurrentlyBlockedTasks.count
8o.a.c.metrics.ThreadPools.request.RequestResponseStage.TotalBlockedTasks.count
9o.a.c.metrics.ThreadPools.transport.Native-Transport-Requests.CurrentlyBlockedTasks.count
10o.a.c.metrics.ThreadPools.transport.Native-Transport-Requests.TotalBlockedTasks.count

Explanation: I don't even.

1o.a.c.metrics.Connection.{host}.CommandCompletedTasks.value
2o.a.c.metrics.Connection.{host}.CommandDroppedTasks.value
3o.a.c.metrics.Connection.{host}.CommandPendingTasks.value
4o.a.c.metrics.Connection.{host}.ResponseCompletedTasks.value
5o.a.c.metrics.Connection.{host}.ResponsePendingTasks.value
6o.a.c.metrics.Connection.{host}.CommandCompletedTasks.value
7o.a.c.metrics.Connection.{host}.CommandDroppedTasks.value
8o.a.c.metrics.Connection.{host}.CommandPendingTasks.value
9o.a.c.metrics.Connection.{host}.ResponseCompletedTasks.value
10o.a.c.metrics.Connection.{host}.ResponsePendingTasks.value

Explanation: Messages are no longer tracked here by "Command/Response", but by "GossipMessage", and "{Large,Small}Message", (so the old metrics are replaced by the new).

1o.a.c.metrics.DroppedMessage.BINARY.Dropped.15MinuteRate
2o.a.c.metrics.DroppedMessage.BINARY.Dropped.1MinuteRate
3o.a.c.metrics.DroppedMessage.BINARY.Dropped.5MinuteRate
4o.a.c.metrics.DroppedMessage.BINARY.Dropped.count
5o.a.c.metrics.DroppedMessage.BINARY.Dropped.meanRate

Explanation: The BINARY verb was deprecated in 2.1.13

The Good News: None of the above changes will impact us; With the exception of dropped messages for BINARY (which are no longer relevant), none of our existing dashboards will be effected.


Upgrade Process

For each node in a cluster

  • Disable Puppet (sudo puppet agent --disable "Upgrading to 2.2.6 : T126629")
  • Set cassandra::target_version in Puppet to 2.2, and merge
  • Shutdown Cassandra (sudo service cassandra stop)
  • Place a hold on the cassandra package (echo "cassandra hold" | sudo dpkg --set-selections)
  • Place a hold on the cassandra-tools package, (if installed) (echo "cassandra-tools hold" | sudo dpkg --set-selections)
  • Upgrade Cassandra package (sudo dpkg -i cassandra_2.2.6._all.deb)
  • Upgrade Cassandra Tools package (if installed) (sudo dpkg -i cassandra-tools_2.2.6._all.deb)
  • Enable Puppet (sudo puppet agent --enable)
  • Force Puppet run, (sudo puppet agent -tv)

After all nodes are upgraded

  • Drop the legacy system_auth tables

After all clusters are upgraded

  • Update the APT repository with the 2.2.6 package(s)
  • Unhold cassandra{-tools} on all machines

On each 2.2.6 node startup, Cassandra will attempt to migrate the legacy tables. This will fail until enough nodes have been upgraded to satisfy the consistency level. When that succeeds, you'll see the following output in the logs.

1INFO [OptionalTasks:1] 2016-05-02 18:33:13,958 CassandraRoleManager.java:410 - Converting legacy users
2INFO [OptionalTasks:1] 2016-05-02 18:33:14,065 CassandraRoleManager.java:420 - Completed conversion of legacy users
3INFO [OptionalTasks:1] 2016-05-02 18:33:14,066 CassandraRoleManager.java:425 - Migrating legacy credentials data to new system table
4INFO [OptionalTasks:1] 2016-05-02 18:33:14,149 CassandraRoleManager.java:438 - Completed conversion of legacy credentials
5INFO [OptionalTasks:1] 2016-05-02 18:33:14,150 CassandraAuthorizer.java:396 - Converting legacy permissions data
6INFO [OptionalTasks:1] 2016-05-02 18:33:14,295 CassandraAuthorizer.java:435 - Completed conversion of legacy permissions

Once all of the nodes have been upgraded, and preserving the possibility of a rollback is no longer necessary, then the 3 legacy tables should be dropped (see below).

1$ sudo c-commands/c-cqlsh default
2Connected to Test Cluster at 10.68.16.128:9042.
3[cqlsh 5.0.1 | Cassandra 2.2.6 | CQL spec 3.3.1 | Native protocol v4]
4Use HELP for help.
5cassandra@cqlsh> DROP TABLE system_auth.users;
6cassandra@cqlsh> DROP TABLE system_auth.credentials;
7cassandra@cqlsh> DROP TABLE system_auth.permissions;
8cassandra@cqlsh>

Once the legacy tables have been dropped, Cassandra will begin using the new tables. The log output post-DROP looks like the following:

1INFO [SharedPool-Worker-1] 2016-05-02 18:37:04,543 MigrationManager.java:401 - Drop table 'system_auth/users'
2INFO [SharedPool-Worker-1] 2016-05-02 18:37:19,372 MigrationManager.java:401 - Drop table 'system_auth/credentials'
3INFO [SharedPool-Worker-1] 2016-05-02 18:37:36,073 MigrationManager.java:401 - Drop table 'system_auth/permissions'


NOTE: 2016-04-25T13:18-05:00 : A release vote is underway for 2.2.6; We should plan to upgrade to 2.2.6, instead.
NOTE: 2016-04-26T15:57+0000: Cassandra 2.2.6 has been released.
NOTE: 2016-05-02T18:41+0000: deployment-prep upgrade to 2.2.6 is complete (including migration of system_auth tables)

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL [2016-07-06T16:45:22Z] <urandom> Restarting Cassandra for restbase1011-b.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T16:50:04Z] <urandom> Restarting Cassandra for restbase1011-c.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T16:53:59Z] <urandom> Upgrade of restbase1011.eqiad.wmnet instances to Cassandra 2.2.6 complete : T126629

Mentioned in SAL [2016-07-06T17:14:39Z] <urandom> Disabling Puppet on restbase{1008,1012,1013}.eqiad.wmnet in preparation for rack 'b' Cassandra upgrade : T126629

Change 297619 had a related patch set uploaded (by Eevans):
Upgrade rack 'b' Cassandra nodes to 2.2.6

https://gerrit.wikimedia.org/r/297619

Change 297619 merged by Elukey:
Upgrade rack 'b' Cassandra nodes to 2.2.6

https://gerrit.wikimedia.org/r/297619

Mentioned in SAL [2016-07-06T18:40:38Z] <urandom> Upgrading Cassandra package to 2.2.6-wmf1 on restbase1008 : T126629

Mentioned in SAL [2016-07-06T18:42:55Z] <urandom> Restarting Cassandra instance restbase1008-a : T126629

Mentioned in SAL [2016-07-06T18:47:46Z] <urandom> Restarting Cassandra instance restbase1008-b : T126629

Mentioned in SAL [2016-07-06T18:52:46Z] <urandom> Restarting Cassandra instance restbase1008-c : T126629

Mentioned in SAL [2016-07-06T18:55:34Z] <urandom> Cassandra upgrade of restbase1008.eqiad.wmnet instances complete : T126629

Mentioned in SAL [2016-07-06T19:07:12Z] <urandom> Upgrading Cassandra package to 2.2.6-wmf1 on restbase1012.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T19:09:00Z] <urandom> Restarting Cassandra instance restbase1012-a.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T19:18:12Z] <urandom> Restarting Cassandra instance restbase1012-b.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T19:24:24Z] <urandom> Restarting Cassandra instance restbase1012-c.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T19:30:42Z] <urandom> Upgrade of restbase1012.eqiad.wmnet instances complete : T126629

Mentioned in SAL [2016-07-06T19:57:56Z] <urandom> Upgrading Cassandra package to 2.2.6-wmf1 on restbase1013.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T20:00:19Z] <urandom> Restarting Cassandra instance restbase1013-a.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T20:06:25Z] <urandom> Restarting Cassandra instance restbase1013-b.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T20:13:53Z] <urandom> Upgrade of restbase1013.eqiad.wmnet instances complete : T126629

Change 297645 had a related patch set uploaded (by Eevans):
Upgrade restbase1009 to Cassandra 2.2.6

https://gerrit.wikimedia.org/r/297645

Mentioned in SAL [2016-07-06T20:23:47Z] <urandom> Disable Puppet on restbase1009.eqiad.wmnet : T126629

Change 297645 merged by Ottomata:
Upgrade restbase1009 to Cassandra 2.2.6

https://gerrit.wikimedia.org/r/297645

Mentioned in SAL [2016-07-06T20:37:58Z] <urandom> Restarting Cassandra instance restbase1008-a.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-06T20:44:07Z] <urandom> Restarting Cassandra instance restbase1008-b.eqiad.wmnet : T126629

Change 297809 had a related patch set uploaded (by Eevans):
Upgrade remaining rack 'd' Cassandra nodes to 2.2.6

https://gerrit.wikimedia.org/r/297809

Mentioned in SAL [2016-07-07T16:29:36Z] <urandom> Disabling Puppet on restbase101[4-5].eqiad.wmnet : T126629

Change 297809 merged by Elukey:
Upgrade remaining rack 'd' Cassandra nodes to 2.2.6

https://gerrit.wikimedia.org/r/297809

Mentioned in SAL [2016-07-07T16:38:58Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase1014.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-07T16:39:22Z] <urandom> Restarting Cassandra instance restbase1014-a.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-07T16:48:34Z] <urandom> Restarting Cassandra instance restbase1014-b.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-07T16:55:40Z] <urandom> Restarting Cassandra instance restbase1014-c.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-07T18:07:02Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase1015.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-07T18:08:43Z] <urandom> Restarting Cassandra instance restbase1015-a.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-07T18:11:54Z] <urandom> Restarting Cassandra instance restbase1015-b.eqiad.wmnet : T126629

Mentioned in SAL [2016-07-07T18:15:23Z] <urandom> Cassandra 2.2.6 upgrade of restbase1015.eqiad.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T18:27:28Z] <urandom> Disabling Puppet on RESTBase codfw nodes : T126629

Change 297838 had a related patch set uploaded (by Eevans):
Upgrade codfw nodes to Cassandra 2.2

https://gerrit.wikimedia.org/r/297838

Change 297838 merged by Dzahn:
Upgrade codfw nodes to Cassandra 2.2

https://gerrit.wikimedia.org/r/297838

Mentioned in SAL [2016-07-07T18:53:55Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2001.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T18:55:59Z] <urandom> Restarting Cassandra instance restbase2001-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:02:28Z] <urandom> Restarting Cassandra instance restbase2001-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:05:41Z] <urandom> Restarting Cassandra instance restbase2001-c.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:08:18Z] <urandom> Cassandra upgrade of restbase2001.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T19:09:19Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2002.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:11:35Z] <urandom> Restarting Cassandra instance restbase2002-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:13:45Z] <urandom> Restarting Cassandra instance restbase2002-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:16:43Z] <urandom> Restarting Cassandra instance restbase2002-c.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:19:34Z] <urandom> Upgrade of restbase2002.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T19:22:29Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2007.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:24:38Z] <urandom> Restarting Cassandra instance restbase2007-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:27:10Z] <urandom> Restarting Cassandra instance restbase2007-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:29:57Z] <urandom> Restarting Cassandra instance restbase2007-c.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:32:56Z] <urandom> Upgrade of restbase2007.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T19:45:13Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2003.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:46:50Z] <urandom> Restarting Cassandra instance restbase2003-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:50:36Z] <urandom> Restarting Cassandra instance restbase2003-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:53:42Z] <urandom> Cassandra upgrade of restbase2003.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T19:55:11Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2004.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T19:55:50Z] <urandom> Restarting Cassandra instance restbase2004-a.codfw.wmnet : T126629

Reedy renamed this task from Cassandra 2.1.13 and/or 2.2.6 to Cassandra 2.2.6.Jul 7 2016, 7:57 PM

Mentioned in SAL [2016-07-07T19:59:11Z] <urandom> Restarting Cassandra instance restbase2004-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:01:00Z] <urandom> Cassandra upgrade of restbase2004.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T20:01:41Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2008.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:02:53Z] <urandom> Restarting Cassandra instance restbase2008-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:04:45Z] <urandom> Restarting Cassandra instance restbase2008-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:07:00Z] <urandom> Cassandra upgrade of restbase2008.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T20:26:00Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2005.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:27:31Z] <urandom> Restarting Cassandra instance restbase2005-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:30:39Z] <urandom> Restarting Cassandra instance restbase2005-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:33:51Z] <urandom> Cassandra upgrade of restbase2005.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T20:36:32Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2006.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:38:16Z] <urandom> Restarting Cassandra instance restbase2006-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:41:31Z] <urandom> Restarting Cassandra instance restbase2006-b.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:48:18Z] <urandom> Cassandra upgrade of restbase2006.codfw.wmnet instances complete : T126629

Mentioned in SAL [2016-07-07T20:48:59Z] <urandom> Upgrading Cassandra to 2.2.6-wmf1 on restbase2009.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:50:28Z] <urandom> Restarting Cassandra instance restbase2009-a.codfw.wmnet : T126629

Mentioned in SAL [2016-07-07T20:52:53Z] <urandom> Restarting Cassandra instance restbase2009-b.codfw.wmnet : T126629

All nodes are now running 2.2.6:

1eevans@agenor:~$ cdsh -c restbase -- sudo c-commands/c-foreach-nt version
2restbase1007.eqiad.wmnet: a: ReleaseVersion: 2.2.6
3restbase1007.eqiad.wmnet: b: ReleaseVersion: 2.2.6
4restbase1007.eqiad.wmnet: c: ReleaseVersion: 2.2.6
5restbase1010.eqiad.wmnet: a: ReleaseVersion: 2.2.6
6restbase1010.eqiad.wmnet: b: ReleaseVersion: 2.2.6
7restbase1010.eqiad.wmnet: c: ReleaseVersion: 2.2.6
8restbase1011.eqiad.wmnet: a: ReleaseVersion: 2.2.6
9restbase1011.eqiad.wmnet: b: ReleaseVersion: 2.2.6
10restbase1011.eqiad.wmnet: c: ReleaseVersion: 2.2.6
11restbase1008.eqiad.wmnet: a: ReleaseVersion: 2.2.6
12restbase1008.eqiad.wmnet: b: ReleaseVersion: 2.2.6
13restbase1008.eqiad.wmnet: c: ReleaseVersion: 2.2.6
14restbase1012.eqiad.wmnet: a: ReleaseVersion: 2.2.6
15restbase1012.eqiad.wmnet: b: ReleaseVersion: 2.2.6
16restbase1012.eqiad.wmnet: c: ReleaseVersion: 2.2.6
17restbase1013.eqiad.wmnet: a: ReleaseVersion: 2.2.6
18restbase1013.eqiad.wmnet: b: ReleaseVersion: 2.2.6
19restbase1009.eqiad.wmnet: a: ReleaseVersion: 2.2.6
20restbase1009.eqiad.wmnet: b: ReleaseVersion: 2.2.6
21restbase1014.eqiad.wmnet: a: ReleaseVersion: 2.2.6
22restbase1014.eqiad.wmnet: b: ReleaseVersion: 2.2.6
23restbase1014.eqiad.wmnet: c: ReleaseVersion: 2.2.6
24restbase1015.eqiad.wmnet: a: ReleaseVersion: 2.2.6
25restbase1015.eqiad.wmnet: b: ReleaseVersion: 2.2.6
26restbase2003.codfw.wmnet: a: ReleaseVersion: 2.2.6
27restbase2003.codfw.wmnet: b: ReleaseVersion: 2.2.6
28restbase2004.codfw.wmnet: a: ReleaseVersion: 2.2.6
29restbase2004.codfw.wmnet: b: ReleaseVersion: 2.2.6
30restbase2008.codfw.wmnet: a: ReleaseVersion: 2.2.6
31restbase2008.codfw.wmnet: b: ReleaseVersion: 2.2.6
32restbase2001.codfw.wmnet: a: ReleaseVersion: 2.2.6
33restbase2001.codfw.wmnet: b: ReleaseVersion: 2.2.6
34restbase2001.codfw.wmnet: c: ReleaseVersion: 2.2.6
35restbase2002.codfw.wmnet: a: ReleaseVersion: 2.2.6
36restbase2002.codfw.wmnet: b: ReleaseVersion: 2.2.6
37restbase2002.codfw.wmnet: c: ReleaseVersion: 2.2.6
38restbase2007.codfw.wmnet: a: ReleaseVersion: 2.2.6
39restbase2007.codfw.wmnet: b: ReleaseVersion: 2.2.6
40restbase2007.codfw.wmnet: c: ReleaseVersion: 2.2.6
41restbase2005.codfw.wmnet: a: ReleaseVersion: 2.2.6
42restbase2005.codfw.wmnet: b: ReleaseVersion: 2.2.6
43restbase2006.codfw.wmnet: a: ReleaseVersion: 2.2.6
44restbase2006.codfw.wmnet: b: ReleaseVersion: 2.2.6
45restbase2009.codfw.wmnet: a: ReleaseVersion: 2.2.6
46restbase2009.codfw.wmnet: b: ReleaseVersion: 2.2.6

Mentioned in SAL [2016-07-13T16:05:21Z] <urandom> Installing Cassandra 2.2.6-wmf1 on deployment-restbase0[1-2].deployment-prep.eqiad.wmflabs : T126629

@akosiaris Are there any plans to upgrade the maps cluster? Do you need any help from me?

@Eevans, I think that's mostly a question for @Gehel these days. FWIW, I am ok with it and I can help but I 'd rather not be the point person driving the upgrade

Note: the cassandra 2.2.6-wmf1 packages are available at https://people.wikimedia.org/~eevans/debian/

This was left open pending Discovery's upgrade of the Maps cluster. Since they are evaluating other options, I will close this ticket now.