During T290652 we discovered that searching with synonyms turned on sometimes takes much longer than expected, and can even cause searching to time out
Here's a comparison of labeled queries ordered by response times
The slowest running query (>10s, which means a timeout) is for "wikimedia commons" which only takes ~1s without synonyms
It's not clear what's going on. Here's what I've found so far:
- the query that gets generated for "wikimedia commons" includes a search for 10 synonyms, and running it directly on production elasticsearch causes a timeout
- the query that gets generated for "dog" includes a search for 7 synonyms, and running it directly on production elasticsearch takes around 1.5s
- running the "wikimedia commons" search with all synonyms except one removed takes ~5s
- running the "wikimedia commons" search with all synonyms removed takes ~1s
Just for comparison, here are the "dog" query and the "wikimedia commons" query, simplified for clarity, and with some synonym subqueries removed from the "wikimedia commons" query so that both have the same number of synonym subqueries
I'm running these queries by setting up a tunnel to production elasticsearch like this
ssh -n -L127.0.0.1:9243:<production elasticsearch server>:9243 mwdebug1002.eqiad.wmnet "sleep 36000"
and then executing the search like this
curl -k https://localhost:9243/commonswiki_file/_search?request_cache=false -H "Content-Type: application/json" -d @</path/to/json/file>
Warning: tunneling to production is dangerous. Be really careful NOT to do vagrant provision with the tunnel open (cos that might delete search indices on production), or tunnel to cloudelastic1001.wikimedia.org instead
The "dog" query takes ~1.5s, the "wikimedia commons" query takes ~8.5s. Even if I take out all synonym subqueries except 1, the "wikimedia commons" query takes ~3.5s