We have a list of terms to try against the template search dialog as we make changes. We would like an automated method of evaluating performance and progress.

Acceptance criteria:

• Reads a tsv file of (wiki_id, search term, expected results as json list)
• Generates API queries simulating our search. For example, for the string abcdef we might have https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&prop=info|pageprops&action=query&list=search&ppprop=disambiguation&redirects=true&srsearch=abcdef*&srnamespace=10&srlimit=10&srprop=redirecttitle
• The exact query generated should track the current and planned implementation of the feature in VisualEditor and TemplateWizard.
• Consider making this pluggable.
• Supports running against multiple wikis in a run, according to the input column.
• Runs the queries in parallel, at a given maximum concurrency. Initially 4 (see also https://www.mediawiki.org/wiki/API:Etiquette).
• Scores the results by counting how many of the expected results appear (in any order).
• Outputs a tsv file of (wiki_id, search_term, score, expected results, actual results as json).
• README
• at least one basic integration test.

Code: https://gitlab.com/wmde/technical-wishes/search-checker

Relevant patches: VE exact match: 693379 and 693157

Output: https://docs.google.com/spreadsheets/d/1c_1cz6unjhkP57gPJS-Kmn72I6pQUSNwDJftMluyeLk/edit#gid=0