As a search engineer I want the search features/profiles to be properly tested after being deployed so that I don't deploy non-functional feature.

Search features affecting the scores are not trivial to test as they might not break immediately after being deployed but they can, if not properly written, not do what they are supposed to do.

While working on T307869 we stumbled upon several problems that should have been detected a lot sooner:

• typo in a config var https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/801792
• some tuning params completely ignored like wikidata items boost: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/809209

These two examples show that when we deployed these features we did not properly test their expected outcome. The problem was not immediately visible as search was still "functional".

This is particularly problematic as we assume these features to be working properly and makes further changes a lot harder to deploy/evaluate.

There might be multiple ways to avoid this and the purpose of this ticket is to discuss this and agree on a testing strategy for future features we push to production (esp. manually tuned ranking features which are very prone to this kind of problems).

Possible solution:

• build on top of what is done when deploying mjolnir model by having a set of obvious & well-known queries with expected outcomes which would be verified after the deploy, every new ranking features/profiles would have to provide a set of such queries.
• clear and detailed AC of the deployed feature with a test scenario to verify (could even use cirrusDebug apis to make sure that the expected query is being built)

AC:

• discuss the subject and agree on a strategy to limit future mistakes.