Page MenuHomePhabricator

Load test parallel eventlogging-processor {stag} [5 pts]
Closed, ResolvedPublic

Description

load test eventlogging-processor again, on analytics1004 for similarity with previous test

  • replicate previous load test but reading from kafka with a single processor worker first
  • new load test - read in a group from kafka

This will give us the number of instances to use in puppetization T104228

Event Timeline

kevinator raised the priority of this task from to Normal.
kevinator updated the task description. (Show Details)
kevinator moved this task from Incoming to Prioritized on the Analytics-Backlog board.
kevinator added subscribers: Aklapper, kevinator.
Milimetric updated the task description. (Show Details)Jul 6 2015, 4:22 PM
Milimetric set Security to None.
Milimetric renamed this task from Load test parallel eventlogging-processor {stag} to Load test parallel eventlogging-processor {stag} [3 pts].Jul 6 2015, 4:26 PM
Milimetric moved this task from Prioritized to Tasked on the Analytics-Backlog board.
Milimetric moved this task from Tasked to Blocked on the Analytics-Backlog board.Jul 27 2015, 4:09 PM
madhuvishy edited projects, added Analytics-Kanban; removed Analytics-Backlog.
kevinator renamed this task from Load test parallel eventlogging-processor {stag} [3 pts] to Load test parallel eventlogging-processor {stag} [5 pts].Aug 17 2015, 5:39 PM

Using a the 12 partition eventlogging-client-side raw events topic, I consumed from the beginning of a stream and benchmarked how many messages could be consumed in 60 seconds with different numbers of processors.

Using 1 processors, processed 74003 events in 60 seconds: 1233.3833333333 events/sec
Using 2 processors, processed 138269 events in 60 seconds: 2304.4833333333 events/sec
Using 3 processors, processed 207107 events in 60 seconds: 3451.7833333333 events/sec
Using 4 processors, processed 262497 events in 60 seconds: 4374.9500000000 events/sec
Using 5 processors, processed 313429 events in 61 seconds: 5138.1803278688 events/sec
Using 6 processors, processed 367372 events in 60 seconds: 6122.8666666666 events/sec
Using 7 processors, processed 420120 events in 60 seconds: 7002.0000000000 events/sec
Using 8 processors, processed 459191 events in 60 seconds: 7653.1833333333 events/sec
Using 9 processors, processed 512946 events in 60 seconds: 8549.1000000000 events/sec
Using 10 processors, processed 535246 events in 60 seconds: 8920.7666666666 events/sec
Using 11 processors, processed 571141 events in 60 seconds: 9519.0166666666 events/sec
Using 12 processors, processed 611524 events in 60 seconds: 10192.0666666666 events/sec

The code I ran is here:
https://gist.github.com/ottomata/978e252d09ece47cd714

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Sep 1 2015, 3:36 PM
kevinator closed this task as Resolved.Sep 8 2015, 10:35 PM