Current MLR on Search could probably be improved significantly by reviewing the current training process.
Problem statement
MLR models are trained regularly but have not been deployed recently. While offline evaluation metrics indicate potential improvements through redeploying these models, we must ensure that the current training data is free from bias.
Our working assumption is that a large number of queries in the training set is "easy" for the model to predict an interaction for (and might cause overfit). Experimental results across large wikis show that,
depending on the heuristic used to define "easy," roughly 20% of the queries can be classified as such.
Our hypothesis is that MLR can be improved by learning from more complex queries.
The goal of this task is to evaluate potential improvements by training models that incorporate knowledge of query complexity, including establishing a meaningful definition of a "easy query".
Approach
We train a model with knowledge of easy queries and evaluate its performance (offline) compared to the baseline. Our approach can be broken down into three steps:
- Create a dataset of (query, page) pairs to compute heuristics for labelling "easy queries".
- Implement heuristics to estimate whether a query is easy or not.
- A query is considered easy if there is high similarity between the query and the page title. Additionally, we can train a mixture of experts to classify easy queries and use their predictions to normalize label assignments (e.g., how confident we are that a query is easy).
- We compare several heuristics for (query, title) similarity, including exact match, cosine, Jaccard, and Levenshtein metrics. The assumption that "we have a lot of easy queries" seems valid, although the value of "a lot" depends on the similarity metric applied. We follow a data-driven approach to identify cutoff values for labeling a query as easy.
- Train XGBoost models with knowledge of easy queries and compare the results with baseline models.
- MLR models are trained on a dataset consisting (among other features) of queries performed by users and the resulting page interactions. Our approach involves adding a new dimension to the training data to label a query as "easy" and using this information during training for instance weighting in the construction of the training DMatrix (e.g. we want to "penalize" easy queries). Other methods can be explored (e.g. cost sensitive learning), if the general approach proves to be viable.
- This step requires reproducing (locally, on a stat host) Mjolnir's training logic.
Note that steps 2–3 are iterative. "Easy query" is a hyperparameter that we may need to tune to find an optimal value after several training and evaluation rounds.
Next Steps
This work aims to determine whether there are immediate performance gains with models tuned to account for easy queries.
Team consultation on the outcomes of offline model evaluation will decide whether we should proceed to implement these changes in Mjolnir.
If results are positive, we will move to an A/B testing phase and further testing to identify potential regressions
(e.g., improved performance on complex queries at the cost of poor performance on easy queries).
AC
- documented understanding of the current limitation and potential improvements to MLR training
- hypothesis describing the user impact of potential improvements