Page MenuHomePhabricator

Load Test Event Logging {oryx} [8 pts]
Closed, ResolvedPublic

Description

Make a thorough Load Test of the entire Event Logging pipeline. Document at what point each piece starts to drop events, along with what kind of resources were available at the time of running (CPU, memory, etc):

  • Forwarder
  • Processor
  • Consumer writing to MySQL

Event Timeline

Milimetric claimed this task.
Milimetric raised the priority of this task from to Needs Triage.
Milimetric updated the task description. (Show Details)
Milimetric added a project: Analytics-Kanban.
Milimetric added a subscriber: Milimetric.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 28 2015, 4:00 PM
kevinator renamed this task from Load Test Event Logging to Load Test Event Logging {oryx}.May 28 2015, 4:02 PM
kevinator set Security to None.
Milimetric renamed this task from Load Test Event Logging {oryx} to Load Test Event Logging [8 pts] {oryx}.May 29 2015, 1:58 PM
Milimetric moved this task from Next Up to Tasked_Hidden on the Analytics-Kanban board.
Milimetric moved this task from Tasked_Hidden to In Progress on the Analytics-Kanban board.

Pausing this task as we need a separate box to continue testing. I ran tests on beta labs and it seems clear that the virtual instance is our bottleneck. Two tests were run and documented here: https://meta.wikimedia.org/wiki/User_talk:Milimetric_(WMF)/Work_log/2015-06-02

kevinator triaged this task as Normal priority.Jun 3 2015, 8:36 PM
Milimetric renamed this task from Load Test Event Logging [8 pts] {oryx} to Load Test Event Logging {oryx} [8 pts].Jun 3 2015, 8:55 PM

Conclusion: Processor is slower than the Consumer, by quite a bit, but both can theoretically handle over 1k events per second if given enough resources. The Forwarder is of course the fastest.

Processor on a beefy Cisco box: more than 1700 events per second
Consumer on a beefy Cisco box: more than 2600 events per second with a local mysql server

More details:

https://meta.wikimedia.org/wiki/User_talk:Milimetric_(WMF)/Work_log/2015-06-26#Event_Logging_load_test

Milimetric moved this task from In Code Review to Done on the Analytics-Kanban board.
Milimetric moved this task from In Progress to Done on the Analytics-Kanban board.Jun 30 2015, 2:57 AM
kevinator closed this task as Resolved.Jun 30 2015, 3:40 PM