Page MenuHomePhabricator

Write script to evaluate template search strategies
Closed, ResolvedPublic2 Estimated Story Points

Description

We have a list of terms to try against the template search dialog as we make changes. We would like an automated method of evaluating performance and progress.

Acceptance criteria:

  • Reads a tsv file of (wiki_id, search term, expected results as json list)
  • Generates API queries simulating our search. For example, for the string abcdef we might have https://en.wikipedia.org/w/api.php?action=query&format=json&formatversion=2&prop=info|pageprops&action=query&list=search&ppprop=disambiguation&redirects=true&srsearch=abcdef*&srnamespace=10&srlimit=10&srprop=redirecttitle
  • The exact query generated should track the current and planned implementation of the feature in VisualEditor and TemplateWizard.
  • Consider making this pluggable.
  • Supports running against multiple wikis in a run, according to the input column.
  • Runs the queries in parallel, at a given maximum concurrency. Initially 4 (see also https://www.mediawiki.org/wiki/API:Etiquette).
  • Scores the results by counting how many of the expected results appear (in any order).
  • Outputs a tsv file of (wiki_id, search_term, score, expected results, actual results as json).
  • README
  • at least one basic integration test.

Code: https://gitlab.com/wmde/technical-wishes/search-checker

Relevant patches: VE exact match: 693379 and 693157

Output: https://docs.google.com/spreadsheets/d/1c_1cz6unjhkP57gPJS-Kmn72I6pQUSNwDJftMluyeLk/edit#gid=0

Event Timeline

Lena_WMDE set the point value for this task to 5.Jun 3 2021, 1:35 PM

I brought this into the sprint because the sidebar tasks are temporarily blocked on me, and because we seem to keep coming back to the need for this script's results.

WMDE-Fisch changed the point value for this task from 5 to 3.
WMDE-Fisch changed the point value for this task from 3 to 2.Jul 7 2021, 7:43 AM
WMDE-Fisch subscribed.

Slightly updated the results with some fixes. Note that, old results with an empty cell or only one entry in the cell are accurate. Sometimes there would have been no result or just one for a specific search.