Page MenuHomePhabricator

Deploy Add a link to a third round of wikis
Open, Stalled, MediumPublic

Description

Add a link have been tested on our pilot wikis T278710), and a second set of wikis gor them too (T284481). Now it is time for a third set of wikis.

We will go with a shortlist of wikis, to have an opportunity to monitor these wikis closely, before deploying to all wikis. The wikis we select have to be rather active wikis, who know how to contact us.

Wikipedias to potentially work with:

  • Catalan Wikipedia
  • Hebrew Wikipedia
  • Hindi Wikipedia
  • Korean Wikipedia
  • Norwegian Bokmål Wikipedia
  • Portuguese Wikipedia
  • Simple English Wikipedia
  • Swedish Wikipedia
  • Ukrainian Wikipedia

Development tasks

Communities relation tasks

  • Week XX: announcement to communities
  • Week XX: deployment

Event Timeline

@Trizek-WMF is the list in the description finalized? If not, please let us know when it is, so we can begin training models, generating the datasets, etc.

kostajh triaged this task as Medium priority.Sep 13 2021, 10:03 AM

@Trizek-WMF is the list in the description finalized? If not, please let us know when it is, so we can begin training models, generating the datasets, etc.

@kostajh, you can start the work.

Regarding community announcement, I'm off on week 38. We would announce it to the communities starting on Monday September 27.

@Trizek-WMF is the list in the description finalized? If not, please let us know when it is, so we can begin training models, generating the datasets, etc.

@kostajh, you can start the work.

@kostajh Should we help with running the training pipeline to generate datasets and models? I am currently working with @kevinbazira to transition the algorithm to the ML-team. Therefore, going through these steps together to train for a new set of languages might be a good exercise.

@Trizek-WMF is the list in the description finalized? If not, please let us know when it is, so we can begin training models, generating the datasets, etc.

@kostajh, you can start the work.

@kostajh Should we help with running the training pipeline to generate datasets and models? I am currently working with @kevinbazira to transition the algorithm to the ML-team. Therefore, going through these steps together to train for a new set of languages might be a good exercise.

@MGerlach, sounds good, please work with @kevinbazira to do the steps on stats1008. Do you think you would be able to do it early next week?

Tentatively assigning to @kevinbazira per the discussion above. Please let us know if you will be able to work on it in the coming days, as we'd like to announce it to communities a week from now (T290011#7352557), although we could push that back if needed (cc @MMiller_WMF).

Thanks @kostajh. @MGerlach and I have started running the training pipelines on stat1008. We shall keep you updated on the progress.

Change 722623 had a related patch set uploaded (by Kosta Harlan; author: MGerlach):

[research/mwaddlink@main] Fix parsing error when generating the backtesting data

https://gerrit.wikimedia.org/r/722623

Change 722623 merged by jenkins-bot:

[research/mwaddlink@main] Fix parsing error when generating the backtesting data

https://gerrit.wikimedia.org/r/722623

@MGerlach, sounds good, please work with @kevinbazira to do the steps on stats1008. Do you think you would be able to do it early next week?

@kevinbazira generated the models and the datasets for all languages mentioned above on stat1008.

The backtesting results are in this spreadsheet (you can also compare to the numbers in previous rounds).
For the default threshold (0.5) we get the following results:

  • overall, results are comparable to previous rounds and no major red flags
  • cawiki, nowiki, ptwiki, simplewiki, svwiki, ukwiki have high precision (>=79%) and pretty good recall numbers (>=40%) so I am very confident in these models
  • hewiki, hiwiki, kowiki have a bit lower precision (72-75%) and quite a bit lower recall (25-28%) so there might be issues in generating enough recommendations; however, I dont think it is extremely concerning as this is comparable to the backtesting results we got for arwiki and bnwiki in the previous round (see the next tab in the spreadsheet).

@kostajh would you like us to continue with publishing the datasets?

@MGerlach, sounds good, please work with @kevinbazira to do the steps on stats1008. Do you think you would be able to do it early next week?

@kevinbazira generated the models and the datasets for all languages mentioned above on stat1008.

The backtesting results are in this spreadsheet (you can also compare to the numbers in previous rounds).
For the default threshold (0.5) we get the following results:

  • overall, results are comparable to previous rounds and no major red flags
  • cawiki, nowiki, ptwiki, simplewiki, svwiki, ukwiki have high precision (>=79%) and pretty good recall numbers (>=40%) so I am very confident in these models
  • hewiki, hiwiki, kowiki have a bit lower precision (72-75%) and quite a bit lower recall (25-28%) so there might be issues in generating enough recommendations; however, I dont think it is extremely concerning as this is comparable to the backtesting results we got for arwiki and bnwiki in the previous round (see the next tab in the spreadsheet).

@kostajh would you like us to continue with publishing the datasets?

@MGerlach yes, please go ahead and publish.

kevinbazira updated the task description. (Show Details)

Thank you for the confirmation @kostajh.

@MGerlach and I have published the models and datasets for the 9 languages mentioned above.

Thank you for the confirmation @kostajh.

@MGerlach and I have published the models and datasets for the 9 languages mentioned above.

Thank you @kevinbazira & @MGerlach :)

Assigning to myself now for the MediaWiki specific configuration.

Change 723517 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[operations/mediawiki-config@master] GrowthExperiments: Enable AddLink for next round of wikis

https://gerrit.wikimedia.org/r/723517

Change 723518 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Remove unnecessary GELinkREcommendationsFrontendEnabled config

https://gerrit.wikimedia.org/r/723518

Change 723518 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] Remove unnecessary GELinkREcommendationsFrontendEnabled config

https://gerrit.wikimedia.org/r/723518

I ended up moving this to T291687: Remove GELinkREcommendationsFrontendEnabled config flag

Trizek-WMF changed the task status from Open to Stalled.Sep 29 2021, 2:51 PM

Given the feedback we received from German Wikipedia and Arabic Wikipedia, we have decided to stall this deployment until we work on limiting the number of links each newcomer can add. The work will resume when "Add an image" will be done.

MMiller_WMF renamed this task from Deploy Add a link to a third set of wikis to Deploy Add a link to a third round of wikis.Sep 29 2021, 4:17 PM

Given the feedback we received from German Wikipedia and Arabic Wikipedia, we have decided to stall this deployment until we work on limiting the number of links each newcomer can add. The work will resume when "Add an image" will be done.

Do you mean: 1) number of add link tasks the user can complete per day/week? or 2) number of links that can be added per task?

If it's the latter, then implementing that is relatively simple (and is also not directly related or scheduled to be done with the Image-Suggestions work) – we could use the existing maximumLinksPerTask option (this can be adjusted in NewcomerTasks.json) so that the user can only see e.g. a maximum of 5 links on a task, instead of the current default of 10. We would still want to fetch an extra amount of tasks in case we're not able to find some of the results in the UI, though.

@kostajh, the plan is to limit the number of tasks. We will discuss about this in a forthcoming team meeting.

kostajh moved this task from In Progress to Incoming on the Growth-Team (Current Sprint) board.

@kostajh, the plan is to limit the number of tasks. We will discuss about this in a forthcoming team meeting.

Did we talk about this in a meeting? We now have the ability to limit number of tasks done per day by a user; that functionality was added for #image-suggestion in T293316: Add an image: minimal quality gate, so it would be trivial to introduce for Add-Link as well.

Change 737402 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/GrowthExperiments@master] [WIP] Implement dailyLimit quality gate for add link

https://gerrit.wikimedia.org/r/737402

@kostajh, the plan is to limit the number of tasks. We will discuss about this in a forthcoming team meeting.

Did we talk about this in a meeting?

I can't remember which meeting covered it, but I think we have.

We now have the ability to limit number of tasks done per day by a user; that functionality was added for #image-suggestion in T293316: Add an image: minimal quality gate, so it would be trivial to introduce for Add-Link as well.

Oh, good! :)

Change 737402 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Implement dailyLimit quality gate for add link

https://gerrit.wikimedia.org/r/737402