Page MenuHomePhabricator
Create Task
Maniphest T126619

cassandra slow streaming during (de)commission
Closed, ResolvedPublic
Actions

Description

during the recent (de)commissioning of restbase1007 we've observed slow (~5MB/s) streams to/from other machines, there's a nodetool option to supposedly set streaming bandwidth but no observable effect

restbase1002:~$ nodetool getstreamthroughput
Current stream throughput: 500 Mb/s

Related Objects

Event Timeline

fgiunchedi created this task.Feb 11 2016, 4:31 PM
fgiunchedi raised the priority of this task from to Needs Triage.
fgiunchedi updated the task description. (Show Details)
fgiunchedi added projects: SRE, RESTBase-Cassandra.
fgiunchedi subscribed.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptFeb 11 2016, 4:31 PM
fgiunchedi added a comment.Feb 15 2016, 3:49 PM

during restbase1007-b bootstrap data is streaming from restbase1007-a (i.e. localhost) though the observed speeds are the same, in the order of 4.5MB/s

Eevans triaged this task as High priority.Feb 15 2016, 5:10 PM
Eevans subscribed.

We have a number of cluster changes coming down the pipe (T119935, T125842, and T95253) that would benefit from higher throughput; Bumping priority to high

Eevans mentioned this in T126221: Evaluate efficacy of DateTieredCompactionStrategy.Feb 15 2016, 7:20 PM
Eevans mentioned this in T95253: Finish conversion to multiple Cassandra instances per hardware node.Feb 16 2016, 2:29 PM
• GWicke added a project: Services.Oct 12 2016, 11:21 PM
fgiunchedi lowered the priority of this task from High to Medium.Dec 1 2016, 11:38 PM

Increasing the number of instances has alleviated the problem by increasing the number of streams, the issue still stands though.

• GWicke assigned this task to Eevans.Jul 11 2017, 7:56 PM
• GWicke moved this task from Backlog to later on the Services board.
• GWicke edited projects, added Services (later); removed Services.
Eevans closed this task as Resolved.Dec 5 2017, 8:24 PM

The limitation here is that there is no per-session concurrency for streaming in Cassandra. The throughput observed is the result of what a single thread is capable of decompressing on the receiving end. This has been reported upstream as: https://issues.apache.org/jira/browse/CASSANDRA-14097.

Eevans added a comment.Dec 6 2017, 3:49 PM

Some additional information: CASSANDRA-4663 adds concurrency at the keyspace level (Cassandra 4.0+)

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits