Page MenuHomePhabricator

Create a benchmarking script for tracking Parsoid perf improvements / regressions
Closed, ResolvedPublic

Description

So far, we have been loosely using time or ab -n for detecting how performance changes. But, we need a script that does this a bit more systematically.

Loose set of TODOs:

  1. Identify a set of titles that correspond to p50, mean, p95, p99 parse times on the production cluster
  2. I/O from m/w api requests (especially on non-production cluster, like our dev machines) can introduce noisiness into this. So, the script should do an initial run with the --record option of the parse.js script. The benchmarking runs should then use --replay option to read m/w api results from the recorded responses.
  3. In order to eliminate noise from startup costs, the benchmarking script should spawn Parsoid as a server, and do a couple warmup requests to ensure JIT effects are eliminated, and gather perf data from N runs.
  4. In order to support 3, we may need to add a request header to the api requests to only return perf data (time trace + total parse time / html2wt with components) as a JSON blob. Maybe only when parsoidConfig.devAPI is enabled.

Event Timeline

  1. Run this script continuously and display the results somewhere we can passively monitor.
ssastry triaged this task as Medium priority.Oct 26 2017, 4:58 PM

Change 410613 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] Simple wt2html benchmarking script

https://gerrit.wikimedia.org/r/410613

Change 410613 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Simple wt2html benchmarking script

https://gerrit.wikimedia.org/r/410613

ssastry claimed this task.

For the Parsoid/PHP port, we built a benchmarking script which can potentially be repurposed for this as necessary. Can reopen / create a new ticket if we ever find a need for doing this and our current solution isn't good enough.