Page MenuHomePhabricator
Create Task
Maniphest T161632

[Epic] Improve search by researching and deploying machine learning to re-rank search results
Closed, ResolvedPublic
Actions

Description

https://www.mediawiki.org/wiki/Wikimedia_Engineering/2016-17_Q4_Goals#Discovery

This needs breaking down in to subtasks. :-)

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidNoneT174064 [FY 2017-18 Objective] Implement advanced search methodologies
 ResolvedEBernhardsonT161632 [Epic] Improve search by researching and deploying machine learning to re-rank search results
 ResolvedEBernhardsonT162053 backend data engineering and plumbing for LTRank
 ResolvedEBernhardsonT162054 Build oozie job to collect click data for learn to rank
 ResolvedEBernhardsonT162056 Generate DBN relevance labels from click data
 ResolvedEBernhardsonT162059 Kafka consumer to take learn to rank queries from a queue and run them against elasticsearch to generate relevance labels.
 ResolvedEBernhardsonT162072 Push LTRank feature generation queries into elasticsearch, and read back all the feature logs
 DuplicateEBernhardsonT162075 Oozie job for merging click data with DBN relevance scores
 ResolvedEBernhardsonT162311 Setup test/train/validation splits for our training data.
 ResolvedEBernhardsonT163407 Collect feature vectors from elasticsearch in mjolnir
 ResolvedEBernhardsonT166585 Calculate NDCG of click data that feeds the MLR pipeline
 ResolveddcausseT168813 Collect features from the sltr query
 ResolvedEBernhardsonT172589 Evaluate ensemble pruning algorithms
 ResolvedEBernhardsonT162061 Evaluate libraries for ranking with LambdaRank
 ResolveddcausseT162062 Specialize elasticsearch learning to rank plugin for our use case
 ResolvedEBernhardsonT162279 Collect ideas for feature engineering of LTRank
 ResolvedEBernhardsonT187148 Evaluate features provided by `query_explorer` functionality of ltr plugin
 ResolvedEBernhardsonT188015 Increase ltr.cache.max_size in Cirrus elasticsearch clusters
 ResolvedEBernhardsonT162369 Evaluate rescore windows for learning to rank
 ResolvedEBernhardsonT150032 Add support for interleaved results in 2-way A/B test
 ResolveddebtT171212 Interleaved results A/B test: turn on
 ResolvedEBernhardsonT171213 Interleaved results A/B test: check that data is flowing the way we expect
 ResolveddebtT171214 Interleaved results A/B test: turn off test
 ResolvedmpopovT171215 Interleaved results A/B test: analysis of data
 DeclinedEBernhardsonT171984 Turn on test of LTR with standard AB buckets and an interleaved bucket.
 ResolvedEBernhardsonT168664 Evaluate training set sizes for LTR models
 ResolvedEBernhardsonT169002 Load test the ltr-query plugin
 ResolvedTJonesT169103 Quick RelForge analysis of impact of initial LTR model
 ResolvedEBernhardsonT170009 Evaluate training speed and accuracy for 1M and 30M sample training sets with different worker counts
 ResolvedEBernhardsonT175771 Deploy AB test for all wikipedias with > 1% of search traffic
 ResolveddebtT177490 Stop A/B test on 18 wikis with > 1% of search traffic
 Resolved• chelsyxT176493 Analysis of testing on 18 wikis with > 1% of search traffic
 ResolvedEBernhardsonT175772 Deploy MLR as default content search to enwiki
 ResolveddcausseT175951 Search backend error during full_text search for 'QUERY_SRTING' after 39: i_o_exception: Can't read unknown type [50]
 DeclinedNoneT177520 Experiment with different grouping of queries that get fed into the DBN
 ResolveddcausseT177302 Relax the default AND of the retrieval query filter with minimum_should_match
 ResolveddebtT177502 Deploy A/B test to test relaxing the retrieval query filter
 ResolveddcausseT177956 Turn off: A/B test to test relaxing the retrieval query filter
 Resolved• chelsyxT177957 Analysis of: A/B test to test relaxing the retrieval query filter
 DeclinedNoneT177519 Build and AB test an ML Model with all the features exploded into individual pieces
 ResolvedEBernhardsonT182616 Re-run AB test for Hebrew Wikipedia (has > 1% of search traffic) with new model
 Resolved• chelsyxT183024 Analysis of hewiki's A/B test (> 1% of search traffic with a new model)

Event Timeline

• Deskana created this task.Mar 28 2017, 5:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 28 2017, 5:15 PM
• Deskana assigned this task to EBernhardson.Mar 28 2017, 5:28 PM
• Deskana updated the task description. (Show Details)
• Deskana moved this task from needs triage to Current work on the Discovery-Search board.
• Deskana edited projects, added Discovery-Search (Current work); removed Discovery-Search.
• Deskana added a subscriber: EBernhardson.

Moving in to current sprint so we can break this up; @EBernhardson volunteered to take a first pass at that. :-)

• Deskana triaged this task as High priority.Mar 28 2017, 5:29 PM
EBernhardson created subtask T162053: backend data engineering and plumbing for LTRank.Apr 3 2017, 3:31 PM
EBernhardson created subtask T162061: Evaluate libraries for ranking with LambdaRank.Apr 3 2017, 4:26 PM
EBernhardson created subtask T162062: Specialize elasticsearch learning to rank plugin for our use case.
EBernhardson created subtask T162279: Collect ideas for feature engineering of LTRank.Apr 5 2017, 3:49 PM
EBernhardson created subtask T162369: Evaluate rescore windows for learning to rank.Apr 6 2017, 3:55 PM
EBernhardson moved this task from Current work to Up Next on the Discovery-Search board.Apr 17 2017, 6:06 PM
EBernhardson edited projects, added Discovery-Search; removed Discovery-Search (Current work).
EBernhardson created subtask T168664: Evaluate training set sizes for LTR models.Jun 22 2017, 4:42 PM
debt lowered the priority of this task from High to Medium.Jun 22 2017, 5:12 PM
debt closed subtask T162061: Evaluate libraries for ranking with LambdaRank as Resolved.Jun 23 2017, 8:58 PM
EBernhardson created subtask T169002: Load test the ltr-query plugin.Jun 27 2017, 5:57 PM
TJones created subtask T169103: Quick RelForge analysis of impact of initial LTR model.Jun 28 2017, 3:13 PM
debt closed subtask T169002: Load test the ltr-query plugin as Resolved.Jun 30 2017, 9:20 PM
debt closed subtask T168664: Evaluate training set sizes for LTR models as Resolved.
EBernhardson created subtask T170009: Evaluate training speed and accuracy for 1M and 30M sample training sets with different worker counts.Jul 7 2017, 6:54 PM
TJones mentioned this in T85770: multi term search does not return expected results.Jul 13 2017, 7:41 PM
debt closed subtask T169103: Quick RelForge analysis of impact of initial LTR model as Resolved.Jul 14 2017, 10:10 PM
debt closed subtask T162062: Specialize elasticsearch learning to rank plugin for our use case as Resolved.Jul 14 2017, 10:15 PM
• EBjune added a parent task: T174064: [FY 2017-18 Objective] Implement advanced search methodologies.Aug 31 2017, 4:38 PM
EBernhardson created subtask T175771: Deploy AB test for all wikipedias with > 1% of search traffic.Sep 13 2017, 12:48 AM
EBernhardson created subtask T175772: Deploy MLR as default content search to enwiki.Sep 13 2017, 12:51 AM
debt closed subtask T175772: Deploy MLR as default content search to enwiki as Resolved.Sep 20 2017, 6:01 PM
debt closed subtask T175771: Deploy AB test for all wikipedias with > 1% of search traffic as Resolved.
debt subscribed.Sep 22 2017, 3:07 PM

We've deployed a new test to 18 other wikis that have less than 1% of traffic: T175771

debt added a subtask: T177520: Experiment with different grouping of queries that get fed into the DBN.Oct 5 2017, 5:08 PM
debt added a subtask: T177302: Relax the default AND of the retrieval query filter with minimum_should_match.
debt added a subtask: T177519: Build and AB test an ML Model with all the features exploded into individual pieces.Oct 5 2017, 5:11 PM
debt closed subtask T177302: Relax the default AND of the retrieval query filter with minimum_should_match as Resolved.Oct 6 2017, 7:41 PM
Liuxinyu970226 subscribed.Oct 20 2017, 3:30 AM
debt closed subtask T170009: Evaluate training speed and accuracy for 1M and 30M sample training sets with different worker counts as Resolved.Nov 9 2017, 10:10 PM
debt closed subtask T162369: Evaluate rescore windows for learning to rank as Resolved.Nov 16 2017, 5:43 PM
debt added a subtask: T182616: Re-run AB test for Hebrew Wikipedia (has > 1% of search traffic) with new model.Dec 11 2017, 6:46 PM
debt closed subtask T162053: backend data engineering and plumbing for LTRank as Resolved.Dec 15 2017, 6:27 PM
debt closed subtask T182616: Re-run AB test for Hebrew Wikipedia (has > 1% of search traffic) with new model as Resolved.Mar 22 2018, 5:16 PM
EBernhardson closed subtask T177519: Build and AB test an ML Model with all the features exploded into individual pieces as Declined.Sep 10 2018, 5:55 PM
EBernhardson closed subtask T162279: Collect ideas for feature engineering of LTRank as Resolved.Sep 10 2018, 6:06 PM
EBernhardson closed this task as Resolved.Sep 10 2018, 6:09 PM

Deployed! (some time ago).

Liuxinyu970226 unsubscribed.Sep 11 2018, 11:09 AM
Gehel closed subtask T177520: Experiment with different grouping of queries that get fed into the DBN as Declined.Aug 11 2022, 7:56 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL