Page MenuHomePhabricator

Import recent MLR models built by MjoLniR in production and test them
Open, MediumPublic8 Estimated Story Points

Description

As of October 2024 the model used in production were trained on april 2022 (models suffixed with 20220421-20180215-query_explorer).

It could be interesting to upload a new set of models for the projects where LTR is enabled:

  • learn if the search behaviors changed in 2 years
  • exercice our ability to do A/B tests
  • assess our A/B infrastructure
  • assess our ability to run analysis

While some part of this work might be relatively straightforward:

  • verify that models are properly exported weekly to the proper elasticsearch clusters
  • setup an interleaving A/B test in mw-config
  • verify that the A/B test data is flowing in

The analysis part might be more challenging. Can we re-use the automatic report generator or do we have to rebuild it?

AC:

  • Setup an A/B test comparing the production models and recent models on the projects where LTR is enabled
  • Verify that the interleaving A/B test infrastructure is collecting the data we expect
  • Determine how to run an analysis on the A/B test data (possibly create a separate task if we can't re-use the automatic report generator)
  • Promote the new models to production if proven better or try to understand why they perform worse if not

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Gehel triaged this task as Medium priority.Oct 14 2024, 2:41 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel set the point value for this task to 8.Nov 4 2024, 4:27 PM

Change #1099773 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Configure MLR buckets

https://gerrit.wikimedia.org/r/1099773

Verified that models are still regularly flowing, and that the training metrics look reasonable. Poked at the historical data we keep, across all wikis that we train models for the training loss (ndcg@10) varies a bit but is mostly consistent since we started using this feature set.

Attached patch configures the test but does not turn it on. This configures three buckets: historical model, new model, and interleaved between the two. Once the config patch is deployed during a backport window we can verify that the results of manually providing the triggers works as expected. We can then deploy a second patch to turn-on automatic enrollment to the test buckets.

Change #1100158 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Enable mlr-2024 for select wikis

https://gerrit.wikimedia.org/r/1100158

Planning to run this as a 7 day test to capture both weekday and weekend traffic. Expecting to deploy the configuration today, and then activate the test either Wed or Thurs depending on my availability overlap with the deployment windows.

Change #1099773 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Configure MLR buckets

https://gerrit.wikimedia.org/r/1099773

Mentioned in SAL (#wikimedia-operations) [2024-12-03T21:34:37Z] <ebernhardson@deploy2002> Started scap sync-world: Backport for [[gerrit:1099773|cirrus: Configure MLR buckets (T377128)]]

Mentioned in SAL (#wikimedia-operations) [2024-12-03T21:40:31Z] <ebernhardson@deploy2002> ebernhardson: Backport for [[gerrit:1099773|cirrus: Configure MLR buckets (T377128)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-12-03T21:52:25Z] <ebernhardson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1099773|cirrus: Configure MLR buckets (T377128)]] (duration: 17m 47s)

Change #1100158 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Enable mlr-2024 for select wikis

https://gerrit.wikimedia.org/r/1100158

Mentioned in SAL (#wikimedia-operations) [2024-12-09T21:22:56Z] <cjming@deploy2002> Started scap sync-world: Backport for [[gerrit:1100158|cirrus: Enable mlr-2024 for select wikis (T377128)]]

Mentioned in SAL (#wikimedia-operations) [2024-12-09T21:27:15Z] <cjming@deploy2002> cjming, ebernhardson: Backport for [[gerrit:1100158|cirrus: Enable mlr-2024 for select wikis (T377128)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-12-09T21:33:25Z] <cjming@deploy2002> Finished scap sync-world: Backport for [[gerrit:1100158|cirrus: Enable mlr-2024 for select wikis (T377128)]] (duration: 10m 28s)

Change #1105065 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] Revert "cirrus: Enable mlr-2024 for select wikis"

https://gerrit.wikimedia.org/r/1105065

Change #1105065 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "cirrus: Enable mlr-2024 for select wikis"

https://gerrit.wikimedia.org/r/1105065

Mentioned in SAL (#wikimedia-operations) [2024-12-17T21:04:15Z] <urbanecm@deploy2002> Started scap sync-world: Backport for [[gerrit:1104741|Reader Survey: Partially undeploy (T378660)]], [[gerrit:1104992|Enable AutoModerator on azwiki (T382286)]], [[gerrit:1105065|Revert "cirrus: Enable mlr-2024 for select wikis" (T377128)]]

Mentioned in SAL (#wikimedia-operations) [2024-12-17T21:10:50Z] <urbanecm@deploy2002> urbanecm, ebernhardson, dani, jsn: Backport for [[gerrit:1104741|Reader Survey: Partially undeploy (T378660)]], [[gerrit:1104992|Enable AutoModerator on azwiki (T382286)]], [[gerrit:1105065|Revert "cirrus: Enable mlr-2024 for select wikis" (T377128)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Change #1105079 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Revert^2 "cirrus: Enable mlr-2024 for select wikis"

https://gerrit.wikimedia.org/r/1105079

Change #1105079 merged by Urbanecm:

[operations/mediawiki-config@master] Revert^2 "cirrus: Enable mlr-2024 for select wikis"

https://gerrit.wikimedia.org/r/1105079