load test eventlogging-processor again, on analytics1004 for similarity with previous test
This will give us the number of instances to use in puppetization T104228
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | • kevinator | T102225 {stag} EventLogging on Kafka | |||
| Resolved | Ottomata | T104229 Load test parallel eventlogging-processor {stag} [5 pts] |
Using a the 12 partition eventlogging-client-side raw events topic, I consumed from the beginning of a stream and benchmarked how many messages could be consumed in 60 seconds with different numbers of processors.
Using 1 processors, processed 74003 events in 60 seconds: 1233.3833333333 events/sec Using 2 processors, processed 138269 events in 60 seconds: 2304.4833333333 events/sec Using 3 processors, processed 207107 events in 60 seconds: 3451.7833333333 events/sec Using 4 processors, processed 262497 events in 60 seconds: 4374.9500000000 events/sec Using 5 processors, processed 313429 events in 61 seconds: 5138.1803278688 events/sec Using 6 processors, processed 367372 events in 60 seconds: 6122.8666666666 events/sec Using 7 processors, processed 420120 events in 60 seconds: 7002.0000000000 events/sec Using 8 processors, processed 459191 events in 60 seconds: 7653.1833333333 events/sec Using 9 processors, processed 512946 events in 60 seconds: 8549.1000000000 events/sec Using 10 processors, processed 535246 events in 60 seconds: 8920.7666666666 events/sec Using 11 processors, processed 571141 events in 60 seconds: 9519.0166666666 events/sec Using 12 processors, processed 611524 events in 60 seconds: 10192.0666666666 events/sec
The code I ran is here:
https://gist.github.com/ottomata/978e252d09ece47cd714