Page MenuHomePhabricator

cassandra slow streaming during (de)commission
Closed, ResolvedPublic

Description

during the recent (de)commissioning of restbase1007 we've observed slow (~5MB/s) streams to/from other machines, there's a nodetool option to supposedly set streaming bandwidth but no observable effect

restbase1002:~$ nodetool getstreamthroughput
Current stream throughput: 500 Mb/s

Event Timeline

fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added a subscriber: fgiunchedi.

during restbase1007-b bootstrap data is streaming from restbase1007-a (i.e. localhost) though the observed speeds are the same, in the order of 4.5MB/s

Eevans triaged this task as High priority.Feb 15 2016, 5:10 PM
Eevans added a subscriber: Eevans.

We have a number of cluster changes coming down the pipe (T119935, T125842, and T95253) that would benefit from higher throughput; Bumping priority to high

fgiunchedi lowered the priority of this task from High to Medium.Dec 1 2016, 11:38 PM

Increasing the number of instances has alleviated the problem by increasing the number of streams, the issue still stands though.

GWicke moved this task from Backlog to later on the Services board.
GWicke edited projects, added Services (later); removed Services.

The limitation here is that there is no per-session concurrency for streaming in Cassandra. The throughput observed is the result of what a single thread is capable of decompressing on the receiving end. This has been reported upstream as: https://issues.apache.org/jira/browse/CASSANDRA-14097.

Some additional information: CASSANDRA-4663 adds concurrency at the keyspace level (Cassandra 4.0+)