Page MenuHomePhabricator
Create Task
Maniphest T306644

re-run wbsearchentities optimization process
Closed, ResolvedPublic
Actions

Description

To support elasticsearch 7 the scoring equation for wbsearchentities needs some small shape changes. The weights we use in this search came from relforge_wbsearchentities. The process was last used on elasticserach 5.5, likely some changes will be necessary to get it up and running against 6.8. These reports can be run against the current equation and not the updated one, the goal of having tuning reports is to know that the full process is working and runnable again.

AC: Tuning reports, including weights to deploy to prod, for all languages that have custom weights already deployed

Details

SubjectRepoBranchLines +/-
Revert "cirrus: Turn on AB test of wbsearchentities profiles"operations/mediawiki-configmaster+1 -1
cirrus: Turn on AB test of wbsearchentities profilesoperations/mediawiki-configmaster+1 -1
Add wbsearchentities profiles for testingoperations/mediawiki-configmaster+170 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

EBernhardson created this task.Apr 21 2022, 4:18 PM
EBernhardson moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.Apr 25 2022, 3:48 PM
EBernhardson added a comment.Apr 26 2022, 4:26 PM

Reports generated and published: https://people.wikimedia.org/~ebernhardson/wbsearchentities_202203

EBernhardson claimed this task.Apr 26 2022, 4:35 PM

Few ideas for future exploration:

  • Lots of the weights in the tuning report claim to have minimal influence on the final output, look into why. Do we need to collect more negative samples in the training set? Are the features useless?
  • Could be interesting to generate the sensitivity portion of the report against current production deployed values.
  • The improvement levels are surprisingly similar to before, perhaps suspisously so. Would also be interesting to re-run the optimization process after deploying the new values. If training with the optimized values as the comparison we should see little if any improvement. If it still shows significant improvements there could be errors in the reporting.
gerritbot added a comment.Apr 26 2022, 4:36 PM

Change 786347 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] Add wbsearchentities profiles for testing

https://gerrit.wikimedia.org/r/786347

gerritbot added a project: Patch-For-Review.Apr 26 2022, 4:36 PM
gerritbot added a comment.Apr 26 2022, 8:06 PM

Change 786347 merged by jenkins-bot:

[operations/mediawiki-config@master] Add wbsearchentities profiles for testing

https://gerrit.wikimedia.org/r/786347

Stashbot added a comment.Apr 26 2022, 8:11 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-26T20:11:27Z] <urbanecm@deploy1002> Synchronized wmf-config/: 9805e61f7006edf45199a3e22494945bffaaeb4d: Add wbsearchentities profiles for testing (T306644) (duration: 00m 53s)

Maintenance_bot removed a project: Patch-For-Review.Apr 26 2022, 8:30 PM
EBernhardson added a comment.Apr 26 2022, 8:59 PM

Profiles are deployed, they can be enabled for testing in a single page with a magic query string like wikidataCompletionSearchClicksBucket=T306644-fr. Next steps would be to turn the test on, and set the turn-off date. Previously we did two weeks, I don't remember what went into that decision but running this for two weeks seems plausible as well.

Should we inform anyone at wikidata that we will be turning on the test? Who?

• MPhamWMF added a parent task: T263142: [EPIC] Upgrade Elasticsearch to version 7.10.Apr 26 2022, 9:19 PM
gerritbot added a comment.Apr 27 2022, 8:54 PM

Change 787069 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Turn on AB test of wbsearchentities profiles

https://gerrit.wikimedia.org/r/787069

gerritbot added a project: Patch-For-Review.Apr 27 2022, 8:54 PM
gerritbot added a comment.Apr 27 2022, 8:56 PM

Change 787069 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Turn on AB test of wbsearchentities profiles

https://gerrit.wikimedia.org/r/787069

Stashbot added a comment.Apr 27 2022, 9:01 PM

Mentioned in SAL (#wikimedia-operations) [2022-04-27T21:01:01Z] <ebernhardson@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:787069|cirrus: Turn on AB test of wbsearchentities profiles (T306644)]] (duration: 00m 53s)

Maintenance_bot removed a project: Patch-For-Review.Apr 27 2022, 9:30 PM
EBernhardson added a comment.EditedApr 29 2022, 8:04 PM

Ran the previous AB testing report to get a preliminary look at the data and ensure it's collecting as expected. Everything seems reasonable, the new tuning isn't clearly better but not clearly worse either and we only have a few hundred events for most languages. As stated previously intending to run for two weeks, ending data collection on May 11.

EBernhardson moved this task from In Progress to Waiting on the Discovery-Search (Current work) board.Apr 29 2022, 8:14 PM
gerritbot added a comment.May 16 2022, 6:23 PM

Change 792141 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] Revert "cirrus: Turn on AB test of wbsearchentities profiles"

https://gerrit.wikimedia.org/r/792141

gerritbot added a project: Patch-For-Review.May 16 2022, 6:23 PM
gerritbot added a comment.May 16 2022, 8:38 PM

Change 792141 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "cirrus: Turn on AB test of wbsearchentities profiles"

https://gerrit.wikimedia.org/r/792141

Stashbot added a comment.May 16 2022, 8:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-05-16T20:41:31Z] <catrope@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:792141|Revert "cirrus: Turn on AB test of wbsearchentities profiles" (T306644)]] (duration: 00m 51s)

Maintenance_bot removed a project: Patch-For-Review.May 16 2022, 9:30 PM
EBernhardson moved this task from Waiting to Needs review on the Discovery-Search (Current work) board.May 17 2022, 10:02 PM

Reports found in https://people.wikimedia.org/~ebernhardson/T306644/

Summary is that the tuning is either the same or slightly worse almost everywhere. Unclear currently where things went wrong. It's not significantly worse so the process is still coming up with reasonable values, but those reasonable values aren't resulting in better ranking than the tuning from a few years ago.

Gehel moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.May 23 2022, 3:13 PM
Gehel subscribed.

The situation does not degrade significantly, this allows us to upgrade to ES7, let's move forward.

EBernhardson moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Jun 9 2022, 2:44 PM
Gehel closed this task as Resolved.Jul 25 2022, 2:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits