Page MenuHomePhabricator

Perf test RAID vs JBOD with new hardware and kafka versions
Closed, ResolvedPublic5 Estimated Story Points

Event Timeline

Nuria added a comment.Jun 26 2017, 3:56 PM

We have been using for kafka single disks, as kafka knows where to put topic partititions. If a disk fails the broker needs to be shut down. We want to measure what is the impact of having of having, say, RAID10. We will install RAID in 3 nodes and measure versus non raid 3 nodes.

Nuria moved this task from Incoming to Dashiki on the Analytics board.Jun 26 2017, 3:56 PM
elukey claimed this task.Jul 18 2017, 12:33 PM
elukey added a project: User-Elukey.
elukey moved this task from Backlog to Analytics Backlog on the User-Elukey board.Jul 19 2017, 8:50 AM
elukey edited projects, added Analytics-Kanban; removed Analytics.Jul 28 2017, 2:57 PM
elukey moved this task from Next Up to In Progress on the Analytics-Kanban board.

I had an interesting chat with the Ops team about this task and I believe that we don't need to spend ton of time working on this now:

  1. A kafka broker works appending data to the end of a file on disk, and usually consumers trigger sequential reads at the end of it. This should ensure, on paper, that disk cache is heavily used and disk is not hit that often (except when flush is forced to sync disk with new data).
  1. The kafka brokers seems not to have any paging or swap activity (pidstat confirmed it). Disk usage is on average 5%, IOPs are really low.
  1. Consumers activity is not concentrated in brief time windows (like consuming ton of data once every hour).
  1. RAID10 is a feature that will ease a lot the maintenance of our Jumbo Brokers, and I don't see any big concern that out-weight its benefits (among all: broker keeps working after a disk failure, no more constrained data directories/partitions that can be filled easily causing alerts).
elukey moved this task from In Progress to Done on the Analytics-Kanban board.Aug 1 2017, 12:39 PM
elukey set the point value for this task to 8.Aug 1 2017, 3:36 PM
Nuria closed this task as Resolved.Aug 2 2017, 12:32 AM
Nuria changed the point value for this task from 8 to 5.