The short term solution here is to add more rack d instances (to 1009 and 1015), and get 1014 cleaned up. I see two options:

Option #1: Plow through the 2.2.6 upgrade, then bootstrap 1009-c and/or 1015-c, and cleanup 1014-{a,b,c}.

Option #2: Bootstrap 1009-c and/or 1015-c, and cleanup 1014-{a,b,c}.

You are never, ever, ever supposed to perform range movements in a mixed version cluster (at this time 1007-{a,b,c} are running 2.2.6, the rest of the cluster is running 2.1.13). Option #1 keeps us from violating this cardinal rule. However, given the problems we have already experienced with the upgrade, rushing through the remaining nodes is unwise. Additionally, cleaning up 1014 at 95% utilization is already dubious, postponing it further will only make matters worse.

Option #2 might fail. The likely consequences of a failure are that the bootstrap would not succeed, but as with any unsupported operation, YMMV. This still feels the least risky to me, and so if there are no objections, I will start a bootstrap of 1009-c (RSN).

Change 297417 had a related patch set uploaded (by Eevans):
Bootstrap Cassandra instances restbase1009-c

https://gerrit.wikimedia.org/r/297417

Given the urgency of the matter, I'm voting for trying option #2 as well. Waiting the upgrade to be complete does not sound feasible given the current constraints. Even though making range movements in a mixed env just shouldn't be done, I don't see how the cluster could go down for this bootstrap. From what I can tell, the worst-case scenario here is that the two new bootstraps will fail but the cluster will remain intact otherwise.

Change 297417 merged by Filippo Giunchedi:
Bootstrap Cassandra instance restbase1009-c

https://gerrit.wikimedia.org/r/297417

Mentioned in SAL [2016-07-05T16:09:05Z] <urandom> Bootstrapping restbase1009-c.eqiad.wmnet : T139362

A bootstrap of restbase1009-c.eqiad.wmnet is running, but there were some errors logged. The IPs in question are for restbase1007-{a,b,c} (the 2.2.6 instances), so not entirely unexpected. With any luck this does not result in ~12 hours of streaming, only to fail for these (early) errors. Stay tuned...

In T139362#2430337, @Eevans wrote:

With any luck this does not result in ~12 hours of streaming, only to fail for these (early) errors. Stay tuned...

Actually, after thinking about this more, I'm suspect this is exactly what will happen. The exceptions encountered during stream planning will prevent the bootstrap from completing (i.e. that it did not fail fast, but will fail).

and indeed the bootstrap failed just now as reported below

At least the cluster is still functional. So now we can either revert 1007 back to 2.1.13 and bootstrap these or expedite the conversion to 2.2.6. I'd vote for the latter, but I guess that is highly dependant on the time frame for it. Thoughts?

Alternatively, would decommissioning rb1014-c help?

In T139362#2432838, @mobrovac wrote:

At least the cluster is still functional. So now we can either revert 1007 back to 2.1.13 and bootstrap these or expedite the conversion to 2.2.6. I'd vote for the latter, but I guess that is highly dependant on the time frame for it. Thoughts?

Reverting 1007 would be destructive to 3 instances. The worst-case scenario for 1014 is a removenode of one instance, also destructive, but to much less data.

In T139362#2432948, @mobrovac wrote:

Alternatively, would decommissioning rb1014-c help?

Unfortunately, I don't think it would. Some of that data would stream to 1014-{a,b} as well, and we'd have to wait for that to complete before starting a cleanup. We are very tight on space, I'm not sure a cleanup is possible with the current storage load, let alone after adding more.

I think our best bet is now Option 1 from above. Only, I want to amend that to omit the use of 'plow'; Now is not the time for haste.

I think it is unlikely that we will be able to get out of this without some replica loss on 1014. Even if we were able to start a cleanup now, it would be rough making progress with so little working space. I recommend that we proceed with the 2.2.6 upgrade. If we can get the entire cluster on 2.2.6, we can attempt a bootstrap of 1009-c and/or 1015-c, and attempt a cleanup at that time. If we cannot complete the upgrade before running out of space on 1014, we'll simply have to removenode one instance. Likewise, if we do complete the upgrade, and don't have the working space to make progress on the cleanup, a removenode will be necessary.

I don't think decommissioning would help, data would move off to other instances in rack d including 1014 to make space even tighter, until cleanups get a chance to run, assuming there's enough space

Change 297598 had a related patch set uploaded (by Eevans):
Revert "Bootstrap Cassandra instance restbase1009-c"

https://gerrit.wikimedia.org/r/297598

Change 297598 merged by Filippo Giunchedi:
Revert "Bootstrap Cassandra instance restbase1009-c"

https://gerrit.wikimedia.org/r/297598

Change 297905 had a related patch set uploaded (by Eevans):
Enable instance 1009-c

https://gerrit.wikimedia.org/r/297905

Change 297905 merged by Dzahn:
Enable instance 1009-c

https://gerrit.wikimedia.org/r/297905

Mentioned in SAL [2016-07-07T21:32:27Z] <urandom> Bootstrapping restbase1009-c : T139362

looks like the latest bootstrap failed too

and the "memory was freed" + sockettimeout seem to point to bugs like https://issues.apache.org/jira/browse/CASSANDRA-11345 and related https://issues.apache.org/jira/browse/CASSANDRA-8343

In T139362#2441203, @fgiunchedi wrote:

looks like the latest bootstrap failed too

[ ... ]

and the "memory was freed" + sockettimeout seem to point to bugs like https://issues.apache.org/jira/browse/CASSANDRA-11345 and related https://issues.apache.org/jira/browse/CASSANDRA-8343

Yes, CASSANDRA-8343 does seem applicable here, in particular that a sockettimeout would be generated for a file that took longer than streaming_socket_timeout_in_ms to complete.

Background: This bootstrap was wreaking havoc on the participating machines, 1009 in particular (high cpu utilization, iowait, etc). This was creating high columnfamily and client request latency, which in turn was driving up RESTBase latency. I throttled the streams way back as a result (to 15mbit). I imagine this must have pushed the largest transfers to more than 1 hour in length (the default timeout), causing these failures.

I've resumed the bootstrap (you can resume now!), but suspect it will just fail once more.

1014 is now at 96% utilization (204G free). Assuming we could increase streaming_socket_timeout_in_ms, and or increase the stream throughput so that the bootstrap succeeds, it's likely to be quite late in the afternoon (early Saturday morning EU) before it completes, which would require us to nurse the cleanups throughout the weekend (a process that I believe will likely fail anyway).

Perhaps it's time to call it, and employ the nuclear option (aka nodetool removenode).

@mobrovac, @fgiunchedi ?

indeed sounds like 1014 won't be able to recover the disk space anyway on its own, I'm ok to go removenode

Change 297993 had a related patch set uploaded (by Eevans):
Deinit 1009-c and 1014-c

https://gerrit.wikimedia.org/r/297993

In T139362#2441603, @fgiunchedi wrote:

indeed sounds like 1014 won't be able to recover the disk space anyway on its own, I'm ok to go removenode

Precisely. +1 from me too

Change 297993 merged by Filippo Giunchedi:
Deinit 1009-c and 1014-c

https://gerrit.wikimedia.org/r/297993

Mentioned in SAL [2016-07-08T18:39:11Z] <urandom> Stopping restbase1014-c.eqiad.wmnet : T139362

Mentioned in SAL [2016-07-08T18:48:38Z] <urandom> "This is going to hurt me more than it does you."; `nodetool removenode' of restbase1014-c.eqiad.wmnet : T139362

Mentioned in SAL [2016-07-08T19:56:17Z] <urandom> Throttle RESTBase Cassandra outgoing streams to 3mbit cluster-wide : T139362

Status Update:

The Good News: A nodetool removenode for restbase1014-c.eqiad.wmnet has been initiated, and the space consumed by this instance has been reclaimed.

The Bad News: Until/unless a force is issued (i.e. nodetool removenode force), Cassandra will attempt to stream data from intact replicas to the newly calculated locations. In this case, the stream destinations are obviously the 6 remaining replicas in rack 'd' (restbase1009-{a,b}, and restbase101[4-5]-{a,b}). However, unlike a bootstrap, the stream sources here are taken from all over the cluster. As of this time, I am seeing as many as 23 incoming streams to individual instances in rack 'd'. This has driven utilization quite high and, unsurprisingly, latency as well. I am currently in the process of trying to throttle stream throughput low enough that latency is kept within acceptable limits. If unsuccessful I will force. Stay tuned...

Mentioned in SAL [2016-07-08T20:51:32Z] <urandom> Throttle RESTBase Cassandra outgoing streams to 1mbit cluster-wide : T139362 (actually happened at 21:26)

Mentioned in SAL [2016-07-08T20:59:50Z] <urandom> Forcing node removal (restbase1014-c.eqiad.wmnet) : T139362

Mentioned in SAL [2016-07-08T21:04:28Z] <urandom> Restarting restbase1009-a.eqiad.wmnet to cancel running streams : T139362

Mentioned in SAL [2016-07-08T21:10:59Z] <urandom> Restarting restbase1014-a.eqiad.wmnet to cancel running streams : T139362

Mentioned in SAL [2016-07-08T21:17:18Z] <urandom> Restarting restbase1015-a.eqiad.wmnet to cancel running streams : T139362

Mentioned in SAL [2016-07-08T21:48:05Z] <urandom> Restarting restbase1009-b.eqiad.wmnet to cancel running streams : T139362

Mentioned in SAL [2016-07-08T22:13:42Z] <urandom> Restarting restbase1014-b.eqiad.wmnet to cancel running streams : T139362

Mentioned in SAL [2016-07-08T22:59:35Z] <urandom> Restarting restbase1015-b.eqiad.wmnet to cancel running streams : T139362