Use the Cirrus server side logs to get some elasticsearch time took metrics
Closed, DeclinedPublic

Description

Some of these metrics are available at the Elasticsearch level but this are only useful for forcasting load - they don't have any bearing on what users see. The server side Elasticsearch took times are only slightly better, but they are slightly better. Here is how I get them:

manybubbles@fluorine:/a/mw-log/archive$ time zcat CirrusSearch-all.log-20150424.gz | grep ": prefix search for '" | sed -e 's/.* took \([0-9]\+\) millis and \([0-9]\+\).*/\1 \2/' | wc -l
69621933

real	32m43.118s
user	35m52.355s
sys	0m41.515s

You can replace the word prefix with full_text and get the full text numbers.
Here are the results:

manybubbles@fluorine:/a/mw-log/archive$ zcat CirrusSearch-all.log-20150424.gz | grep ": prefix search for '" | sed -e 's/.* took \([0-9]\+\) millis and \([0-9]\+\).*/\1 \2/' | head
4 3
54 52
48 47
26 24
28 24
5 2
33 31
42 40
65 63
7 5

The first number is the actual time the request took and the second is the time Elasticsearch thinks it took. The delta is somewhat interesting but not very. The first number is probably the more important bit.

Manybubbles updated the task description. (Show Details)
Manybubbles raised the priority of this task from to Normal.
Manybubbles assigned this task to Jdouglas.
Manybubbles added subscribers: ksmith, Deskana, Aklapper and 2 others.

This task is related to this other task which also uses the server side logs: T101774

Jdouglas removed Jdouglas as the assignee of this task.Jun 11 2015, 5:16 PM
Jdouglas set Security to None.
Ironholds closed this task as Declined.Jul 21 2015, 9:51 PM
Ironholds moved this task from Backlog to Done on the Discovery-Analysis (Current work) board.
Ironholds moved this task from Done to Backlog on the Discovery-Analysis (Current work) board.
Ironholds reopened this task as Open.
Deskana closed this task as Declined.Jul 28 2015, 7:52 PM
Deskana claimed this task.

I'm marking this as declined. It keeps getting kicked around and it seems like we're not really sure what we're doing with it, which is a sign that we either don't need the output of this task, or actually have no idea how to fulfil this task.

If anyone objects, speak up and we can reopen it. :-)