Investigate recent Cassandra cluster performance issues
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Eevans
	Oct 27 2015, 10:38 PM

Description

There are a number of concerns regarding Cassandra cluster performance.

Since provisioned, eqiad nodes 1007, 1008, and 1009 have been outliers on a number of key performance metrics.
For most of this month, connection timeouts have trended in a negative direction.
More recently, 1007 has been exhibiting higher than usual client read latency
...and earlier this morning, 1007 died with an OOM exception.

Note: It is ~~possible~~ likely that one or more of these are unrelated, so for now, this issue is a catch-all for the general investigation; Additional tickets will be opened as needed.

Related Objects
Search...

Status	Assigned	Task
Resolved	Eevans	T116859 Investigate recent Cassandra cluster performance issues
Resolved	• GWicke	T116861 Investigate OOM and elevated read latencies on 1007
Resolved	Pchelolo	T116933 Review mobile apps storage strategy, and look into retention policy select scaling
Resolved	• GWicke	T117115 Use Date-Tiered Compaction Strategy for time series and revisioned data
Resolved	Joe	T116739 restbase endpoints health checks timing out
Resolved	Joe	T116770 service_checker reports success even on endpoints timing out
Resolved	• mobrovac	T116911 restbase endpoint reporting incorrect content-encoding: gzip

Event Timeline

Eevans created this task.Oct 27 2015, 10:38 PM

Eevans claimed this task.

Eevans raised the priority of this task from to High.

Eevans updated the task description. (Show Details)

Eevans added a project: RESTBase-Cassandra.

Eevans added a subscriber: Eevans.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2015, 10:38 PM

T116861 is tracking the recent issues on 1007 in particular.

Eevans updated the task description. (Show Details)Oct 27 2015, 11:18 PM

Eevans set Security to None.

Eevans added a subtask: T116861: Investigate OOM and elevated read latencies on 1007.

• GWicke closed subtask T116861: Investigate OOM and elevated read latencies on 1007 as Resolved.Nov 11 2015, 6:27 AM

That 1007, 1008, and 1009 are outliers in the general sense is due to the lower hardware specs on these machines (for example, disk read throughput is higher because there is less memory for page cache).

I believe the remaining issues cited were all addressed by T116861: Investigate OOM and elevated read latencies on 1007.

Closing.

Investigate recent Cassandra cluster performance issuesClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Investigate recent Cassandra cluster performance issues
Closed, ResolvedPublic
Actions

Related Objects
Search...