This API generally responds in under 50 ms, but there is a "slow case" that is consistently hit every single minute for dozens of queries causing timeout fatals and other errors in production.
- 50th percentile (per minute): ~ 50ms
- 75th percentile (per minute): ~ 70ms
- 95th percentile (per minute): ~ 700ms (!)
- 99th percentile (per minute): ~ 1.2 minute (!)
I looked at Logstash fatals with ApiOpenSearch in their trace, and collected a few debug profiles in XHGui.
It seems that the main distinguishing factor between a fast response and a 10-20s+ slow response is whether TextExtracts is doing work.
Example query: https://commons.wikimedia.org//w/api.php?format=json&action=opensearch&search=Ice
XHGui profile: https://performance.wikimedia.org/xhgui/run/view?id=5df3ac043f3dfa6e273f3ba3
This took 19.9 seconds to respond. With 19.7 seconds spent in TextExtracts\ApiQueryExtracts::execute.
This method in turn spends most of its time synchronously parsing wikitext and executing Lua modules for dozens of pages in a row.
The slowest endpoints we have in terms of perf budget are the POST requests for saving edits, which do this once. Doing this dozens of times in a single request is far more than ever meant to happen during a single web request. Especially a GET request to a high-traffic endpoint like the OpenSearch API.
It is also unclear to me why this doesn't use ParserOutput objects from the ParserCache. Instead, it appears to be fetching raw wikitext and feeding it uncached to the Parser, and then caching the result in a custom Memcached key (no warmup in place, no WANCache protections in place).