Page MenuHomePhabricator

Deploy "add a link" to 12th round of wikis
Closed, ResolvedPublic

Description

  • Training models
    • Eastern Mari Wikipedia mhr
    • Maori Wikipedia mi
    • Minangkabau Wikipedia min
    • Macedonian Wikipedia mk
    • Malayalam Wikipedia ml
    • Mongolian Wikipedia mn
    • Mon Wikipedia mnw see T308137#8690680
    • Marathi Wikipedia mr
    • Western Mari Wikipedia mrj
    • Malay Wikipedia ms
    • Maltese Wikipedia mt
    • Mirandese Wikipedia mwl
    • Burmese Wikipedia my see T308137#8690680
    • Erzya Wikipedia myv
    • Mazanderani Wikipedia mzn
    • Nauru Wikipedia na locked wiki
    • Nāhuatl Wikipedia nah
    • Neapolitan Wikipedia nap
    • Low German Wikipedia nds
    • Low Saxon Wikipedia nds-nl
    • Nepali Wikipedia ne
    • Newari Wikipedia new
    • Norwegian Nynorsk Wikipedia nn
  • Models verification
  • Publish Datasets
  • Populate the excluded section titles
  • Deploy back-end
  • Check how the model works on the wikis
  • In Search, use hasrecommendation:link to find articles
  • Test them on https://api.wikimedia.org/service/linkrecommendation/apidocs/#/default/get_v1_linkrecommendations__project___domain___page_title_
  • Inform communities
  • Deploy front-end

Event Timeline

23/23 models were trained successfully in the 12th round of wikis.

Model evaluation has been completed and below are the backtesting results:

Precision@0.5Recall@0.5
mhrwiki0.930.34
miwiki0.940.75
minwiki0.970.96
mkwiki0.870.57
mlwiki0.690.14
mnwiki0.720.30
mnwwiki0.000.00
mrwiki0.970.67
mrjwiki0.930.54
mswiki0.900.68
mtwiki0.750.35
mwlwiki0.840.56
mywiki0.630.06
myvwiki0.790.32
mznwiki0.930.71
nawiki0.950.86
nahwiki0.830.51
napwiki0.910.52
ndswiki0.890.44
nds_nlwiki0.820.55
newiki0.740.15
newwiki1.000.24
nnwiki0.860.64

CCing @MGerlach, in case he would like to add comments on the backtesting evaluation.

The conclusion on the backtesting results is that most of the languages look fine besides:

  • mnwwiki's precision and recall are bad. Both are nil.
  • mnwiki (0.72) and newiki (0.74) have a precision slightly lower than the recommended one (0.75).
  • mlwiki's precision (0.69) and recall (0.14) are lower than the recommended one (0.75 and 0.2 respectively).
  • mywiki has a low precision (0.63) and very low recall (0.06).

Talked to @MGerlach about these results and agreed mnwiki, newiki, and mlwiki should be deployed but mnwwiki and mywiki shouldn't.

@kostajh, we published datasets for all 21/23 models that passed the evaluation in this round.

elukey moved this task from In Progress to Watching on the Machine-Learning-Team board.
elukey added a subscriber: kevinbazira.
Sgs changed the task status from Open to In Progress.Jul 5 2023, 2:07 PM
Sgs claimed this task.
Sgs triaged this task as Medium priority.
Sgs moved this task from Triaged to Sprint 0 (Growth Team) on the Growth-Team board.
Sgs edited projects, added Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team.
Sgs moved this task from Incoming to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.

Change 935723 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[operations/mediawiki-config@master] GrowthExperiments: Enable backend of link recommendation 10, 11, 12th round wikis

https://gerrit.wikimedia.org/r/935723

I ran this script for adding the link-recommendation task type and populating the excluded sections entries:

for WIKI in mhrwiki miwiki minwiki mkwiki mlwiki mnwiki mrwiki mrjwiki mswiki mtwiki mwlwiki myvwiki mznwiki nawiki nahwiki napwiki ndswiki nds_nlwiki newiki newwiki nnwiki; do
    ORIGIN=`mwscript getConfiguration.php $WIKI --settings 'wgCanonicalServer' --format json | jq --raw-output '.wgCanonicalServer'`
    mwscript extensions/GrowthExperiments/maintenance/changeWikiConfig.php $WIKI \
            --page MediaWiki:NewcomerTasks.json \
            --create-only \
            --json \
            --summary "Growth features configuration boilerplate ([[phab:$PHAB]])" \
            link-recommendation \
            '{ "type": "link-recommendation", "group": "easy" }'
    jq "select(.wiki==\"$WIKI\" and .probability > 0.25) | .section" wiki_sections.jsonl \
        | jq --slurp --compact-output "unique" \
        | mwscript extensions/GrowthExperiments/maintenance/changeWikiConfig.php $WIKI \
            --page MediaWiki:NewcomerTasks.json \
            --json \
            --summary "machine-generated configuration for excluding sections from link recommendations ([[phab:$PHAB]]), feel free to improve" \
            link-recommendation.excludedSections \
            "`cat`"
    echo "$ORIGIN/wiki/MediaWiki:NewcomerTasks.json"
    echo "$ORIGIN/w/index.php?title=MediaWiki:NewcomerTasks.json&diff=next"
    echo "Press <Enter> to continue"
    read # give time for manual verification
done

Change 935723 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: Enable backend of link recommendation 10, 11, 12th round wikis

https://gerrit.wikimedia.org/r/935723

Mentioned in SAL (#wikimedia-operations) [2023-07-11T13:03:28Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:935723|GrowthExperiments: Enable backend of link recommendation 10, 11, 12th round wikis (T308135 T308136 T308137)]]

Mentioned in SAL (#wikimedia-operations) [2023-07-11T13:04:58Z] <urbanecm@deploy1002> sgimeno and urbanecm: Backport for [[gerrit:935723|GrowthExperiments: Enable backend of link recommendation 10, 11, 12th round wikis (T308135 T308136 T308137)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-07-11T13:13:13Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:935723|GrowthExperiments: Enable backend of link recommendation 10, 11, 12th round wikis (T308135 T308136 T308137)]] (duration: 09m 45s)

Status update, as per today all wikis have produced results except for nawiki.

For more context:

I think we can proceed with the rollout of this round except for nawiki. cc @Trizek-WMF.

Change 948144 had a related patch set uploaded (by Sergio Gimeno; author: Sergio Gimeno):

[operations/mediawiki-config@master] GrowthExperiments: enable add a link in 12th round of wikis

https://gerrit.wikimedia.org/r/948144

Sgs updated the task description. (Show Details)
Sgs subscribed.
Trizek-WMF updated the task description. (Show Details)

Status update, as per today all wikis have produced results except for nawiki.

For more context:

I think we can proceed with the rollout of this round except for nawiki. cc @Trizek-WMF.

Thank you for conducting the tests. :)

We won't deploy at na.wp. It was locked after I created the lists.

I set the date to deploy on Sept 6th.

Change 948144 merged by jenkins-bot:

[operations/mediawiki-config@master] GrowthExperiments: enable add a link in 12 and 13th round of wikis

https://gerrit.wikimedia.org/r/948144

Mentioned in SAL (#wikimedia-operations) [2023-09-06T20:03:03Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:948144|GrowthExperiments: enable add a link in 12 and 13th round of wikis (T308137 T308138)]]

Mentioned in SAL (#wikimedia-operations) [2023-09-06T20:04:40Z] <taavi@deploy1002> taavi and sgimeno: Backport for [[gerrit:948144|GrowthExperiments: enable add a link in 12 and 13th round of wikis (T308137 T308138)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-09-06T20:13:20Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:948144|GrowthExperiments: enable add a link in 12 and 13th round of wikis (T308137 T308138)]] (duration: 10m 16s)