Page MenuHomePhabricator

Load test production cluster
Closed, ResolvedPublic

Description

The production cluster needs to be load tested to ensure that

  • the new hardware is solid, and
  • there are no stability issues with Debian Jessie (testing so far was with the older Ubuntu Trusty).

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.

Event Timeline

GWicke claimed this task.
GWicke raised the priority of this task from to Medium.
GWicke updated the task description. (Show Details)
GWicke added a project: RESTBase.
GWicke added subscribers: mobrovac, GWicke.

Things have been fairly smooth on the testing front.

  • we encountered no issues from the trusty -> jessie migration
  • restbase1006 turned out to have a faulty disk controller: T89639
  • peak performance we measured is around 10k HTML revision req/s with a mean latency of ~80ms
  • we processed ~7k req/s for a week without issues
  • deliberate cassandra node restarts did not impact availability
  • creating 12k tables for 800 wikis did not go so well; this was addressed in T91188
  • we are currently loading current revisions for all 280 wikipedias into storage, again without issues so far

Resolving. Further testing is happening on the test cluster.