Page MenuHomePhabricator

Ensure mjolnir models have positive scores
Closed, ResolvedPublic

Description

Elasticsearch 7 requires scores to be positive values, but the models mjolnir uploads to the elasticsearch cluster have their values centered around zero. The models will need to be adjusted similar to the example from esltr repo.

AC: Models run in production only produce positive valued scores.

Event Timeline

Change 779924 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[search/MjoLniR@master] Adjust uploaded model to always have a positive score.

https://gerrit.wikimedia.org/r/779924

Change 779924 merged by jenkins-bot:

[search/MjoLniR@master] Adjust uploaded model to always have a positive score.

https://gerrit.wikimedia.org/r/779924

Patch is only the first step, once deployed we will need to wait for new models to build and then update prod to use them.

Change 787830 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[search/MjoLniR/deploy@master] adjust uploaded models to always have a positive score

https://gerrit.wikimedia.org/r/787830

While double checking realized I did not actually deploy this change. Will need to be deployed monday.

Change 787830 merged by Ebernhardson:

[search/MjoLniR/deploy@master] adjust uploaded models to always have a positive score

https://gerrit.wikimedia.org/r/787830

Mentioned in SAL (#wikimedia-operations) [2022-05-02T17:50:47Z] <ebernhardson@deploy1002> Started deploy [search/mjolnir/deploy@f94bb01]: T306123: adjust uploaded models to always have a positive score

Mentioned in SAL (#wikimedia-operations) [2022-05-02T17:51:33Z] <ebernhardson@deploy1002> Finished deploy [search/mjolnir/deploy@f94bb01]: T306123: adjust uploaded models to always have a positive score (duration: 00m 45s)

Daemon has been deployed and restarted. Redeploying the last model upload as follows:

  • Delete already uploaded models (repeat for eqiad and codfw):
cat ltr_wikis | while read wiki; do 
  curl -XDELETE https://search.svc.codfw.wmnet:9243/_ltr/_model/$wiki-20220421-20180215-query_explorer
done
  • Clear the upload step of the latest dag run on airflow1001 so it will re-upload:
sudo -u analytics-search airflow clear -t '^upload-' -s 2022-04-21 -e 2022-04-22 mjolnir

Change 788386 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/mediawiki-config@master] cirrus: Update MLR models to 20220421 deployment

https://gerrit.wikimedia.org/r/788386

Verified the deployed models for 2022-04-21 all have positive leafs ( and a quick check that 2022-04-14 fails the same check) with:

cat ltr_wikis | while read wiki; do
  echo $wiki
  curl "https://search.svc.eqiad.wmnet:9243/.ltrstore/_search?q=name.prefix=$wiki-20220414" \
    | jq '.. | .leaf? | select(. < 0 and . != null)'
done

Change 788386 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: Update MLR models to 20220421 deployment

https://gerrit.wikimedia.org/r/788386

Mentioned in SAL (#wikimedia-operations) [2022-05-02T20:14:44Z] <catrope@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:788386|cirrus: Update MLR models to 20220421 deployment (T306123)]] (duration: 00m 48s)