The suggester index for hewiki was empty and caused a very bad user experience.
Reading logs from /var/log/mediawiki/cirrus-suggest/hewiki.eqiad.log-20160216.gz on terbium I can see:
Fetching Elasticsearch version...1.7.1...ok Infering index identifier...hewiki_titlesuggest_1455026477 Recycling index hewiki_titlesuggest Indexing 185537 documents (185537 in the index) with batchId: 1455584084 2% done... 4% done... 6% done... 8% done... [...] 96% done... 98% done... 100% done... Indexing done. Purging deleted docs...ok. Deleting remaining docs from previous batch (184407). 0% done... 2% done... 4% done... [...] 98% done... 100% done... Done. Optimizing index...ok.
This is concerning... it means that the recycle process deleted the docs it just created...
I was not able to reproduce the problem by running the process manually multiple times on codfw.
- One possible (but improbable) cause is that this process was run twice at the same time, both processes could have deleted each other docs because of a batchId mismatch?
- Another possibility is that the optimize to expunge deleted docs did not run synchronously and the reader was refreshed on the previous state?
- The reader was not refreshed on the proper state and caused the query to extract the ids to delete to return all docs?