OK, I've started mining some random SSTables in the wikipedia_T_parsoid__ng_html keyspace, and I've found some examples that definitely seem to support the prevailing theory.

The decommission of restbase1010-c was discontinued after other instances in the rack began to fail (1016-{a,b,c}, 1010-b & 1007-a). The failures in question all seem to be the result of a JVM out of memory (as opposed to an application OutOfMemory exception).

For the machines affected, executing curl against the exporter URL just hangs indefinitely. I attempted to restart 1011-a to no avail. I then live-hacked cassandra-env.sh to roll back the exporter jar to the 0.8 version we used before, and it is now working. More investigation is needed.

In T189822#4136684, @Eevans wrote:

In T189822#4136670, @Cmjohnson wrote:

@mobrovac The 5 ssds arrived for restbase1010. Do you need to schedule down time to replace?

We'll need to decommission the 3 instances running there first. After the disks are swapped, we'll need someone to do the re-image before we can bootstrap. I'm guessing that will be @fgiunchedi, so we should probably confirm his availability before we begin.

In T189822#4136670, @Cmjohnson wrote:

@mobrovac The 5 ssds arrived for restbase1010. Do you need to schedule down time to replace?

At this point I'm fairly certain that this isn't a memory leak in the conventional sense. A bug in change-propagation had prevented sampling from working, and the dev cluster (with < 20% the capacity of production), was seeing throughput levels in excess of 5x what have in production. The excessive heap utilization would seem to be the result of additional per-thread state associated with this higher throughput. This is still worth pursuing upstream, since this is not how an application should degrade in the face of high load, but since it effects 3.11.0 as well, I think we can remove this as a blocker to a 3.11.2 upgrade

In T192112#4128317, @Eevans wrote:

Let's start with a couple of canaries in restbase1010 and restbase2003 respectively.

Let's start with a couple of canaries in restbase1010 and restbase2003 respectively.

In T186751#4124167, @Dzahn wrote:

fyi: T189050 / T189050#4124163 means you should not have worry anymore about scheduling downtimes for these services when on the dev environment.

Status update:

The cluster is now running 3.11.2 (release).

FWIW: The RESTBase cluster has been disabled for some time. I just disabled the RESTBase Dev cluster as well.

Opened CASSANDRA-14355 to track a memory leak encountered in the dev environment (reproduced on 3.11.0 and 3.11.2).

I'm not sure what else we can do to troubleshoot this; Hopefully this is a one-time freak occurrence. If this recurs, we can reopen this ticket, and try again.

In T189822#4085294, @RobH wrote:

The SSDs we get these days from Intels line are: https://www.intel.com/content/www/us/en/solid-state-drives/ssd-dc-s4500-s4600-brief.html

In T189822#4085294, @RobH wrote:

I'm a bit confused on which system will get the replacement SSDs installed? I'm guessing its restbase2007 or restbase2008, as those each have 5 Samsung 850 EVO SSDs.

I'll generate a sub-task with pricing, since it cannot be in a public task. On that sub-task, I'll request a quote from Dasher for the Intel SSDs, since the restbase200[78] systems are in warranty until 2019-04-22.

In T186567#4083805, @elukey wrote:

In T186567#3993452, @Eevans wrote:

I propose that we first start with the AQS cluster by installing the Prometheus exporter and cassandra-metrics-collector side-by-side, and see where we stand with the dashboards. If it turns out to be straightforward to make the dashboards work interchangeably, then we can consider either a) doing the same with Maps, or b) upgrading Maps to Cassandra 2.2.

AQS is now running the jmx agent and its metrics are available in the prometheus analytics instance (not the services one). @Eevans I'd love to keep one set of dashboard shared between restbase and aqs/maps to avoid duplication of efforts, whenever you have time let's chat about the differences between 2.2 and 3.x.

In T189529#4072704, @elukey wrote:

Thanks! When I try to push the .changes I get that I am missing orig.tar.gz:

root@install1002:/srv/wikimedia# reprepro -C component/cassandra22 --ignore=wrongdistribution include jessie-wikimedia ~elukey/cassandra/cassandra_2.2.6-wmf3_amd64.changes
.changes put in a distribution not listed within it!
Ignoring as --ignore=wrongdistribution given.
Unable to find pool/component/cassandra22/c/cassandra/cassandra_2.2.6.orig.tar.gz needed by cassandra_2.2.6-wmf3.dsc!
Perhaps you forgot to give dpkg-buildpackage the -sa option,
 or you could try --ignore=missingfile to guess possible files to use.
Deleting files just added to the pool but not used.
(to avoid use --keepunusednewfiles next time)
There have been errors!

This is probably my ignorance about debian build process, but are we missing the file or should I try the option to skip it?

A new package that actually applies the patch within has been uploaded; Sorry for the wasted cycles!

In T189889#4062503, @mobrovac wrote:

In T189889#4061081, @Eevans wrote:

Without spelunking too deeply into the NodeJS driver code, I'd say it is to a) intelligently route requests to a coordinator that has the data (and in doing so, eliminate a hop of routing stretch), b) to load-balance among the subset of nodes selected in (a), and c) to route around failures in doing so (these are the common conventions, anyway). This would all require the full compliment of nodes to do correctly.

Right, I meant I don't know why all of these connections are established from the get-go. A better approach IMHO would be to keep the amount of connections minimal at start-up and then open new ones as needed. This would mean that the first queries to hit certain nodes would be slower, but I think that's a good balance. One could argue that in doing so all of the connections would be established eventually, but given the amount of idle connections, I don't think this would happen in practice in our case.

Times up, 8u162-b12-1~bpo8+1 has been made available.

In T189529#4060241, @elukey wrote:

Tried to (manually via dpkg -i) install cassandra 2.2.6-wmf3 on aqs1004:

elukey@aqs1004:~$ dpkg -l | grep cassandra
ii  cassandra                                2.2.6-wmf3                       all          distributed storage system for structured data
ii  cassandra-tools                          2.2.6-wmf3                       all          distributed storage system for structured data
ii  cassandra-tools-wmf                      1.0.1-1                          all          add-ons to make Wikimedia Cassandra operations easier

But the patch for /usr/sbin/cassandra seems not applied, the following snippet is still uncommeted:

# see CASSANDRA-7254
"$JAVA" -cp "$CLASSPATH" $JVM_OPTS 2>&1 | grep -q 'Error: Exception thrown by the agent : java.lang.NullPointerException'
if [ $? -ne "1" ]; then
    echo Unable to bind JMX, is Cassandra already running?
    exit 1;
fi

In T189889#4057226, @mobrovac wrote:

Agreed that we have to upgrade the driver, but just hiding the metrics does not sound like a solution to me here. Given the amount of connections kept open, I think we should explore two possibilities:

• increase the heart-beat interval (perhaps something like 120s would be enough)

Another example:

Some preliminary information from query trace examination:

At this point, I think we've established that more reasonable performance is possible by configuring a JBOD in HBA mode, instead of as a collection of single-disk RAID0s. Many of the existing HP nodes in the RESTBase cluster have already been configured this way, and all that remains is to re-image the 9 nodes stood up prior to this conclusion. That work is being tracked in T186562: Reimage JBO-RAID0 configured RESTBase HP machines, so I'll close this issue. If anyone objects (for example, they're convinced that more can/should be done), then feel free to re-open.

In T185494#4029109, @Eevans wrote:

I removed sdc1, sdc2, and sdc3 from md0, md1, and md2 respectively, and rebooted believing that might be the easiest way to correct the device ordering (the new drive showed as sde). Instead, the machine didn't come back up (and I don't have console access).

I removed sdc1, sdc2, and sdc3 from md0, md1, and md2 respectively, and rebooted believing that might be the easiest way to correct the device ordering (the new drive showed as sde). Instead, the machine didn't come back up (and I don't have console access).

In T92471#4015525, @mobrovac wrote:

@Eevans @fgiunchedi is there something left to be done here?