In T304110#7814510, @Trizek-WMF wrote:

In T304110#7814487, @kostajh wrote:

In T304110#7814354, @Trizek-WMF wrote:

Thank you @kostajh. The wikis already ready are listed on T304542: Deploy "add a link" to third round of wikis. I'm editing the task accordingly.

We can announce a deployment the week after the training is done, and for announcement purposes, on a Wednesday (time to have Tech News reaching at all wikis, and enough time before Friday).

I would adjust the timing – after the backend patch is enabled, we need to wait some period of time (days, maybe 1-2 weeks) to see that there are enough link recommendation tasks for the wiki (see the task pool section of https://grafana.wikimedia.org/d/vGq7hbnMz/special-homepage-and-suggested-edits?orgId=1). Once we see there are enough tasks, then we can enable the frontend.

So it would be a deployment around April 15, on a Wednesday.

Maybe, with everything happening in Growth-Team (Sprint 0 (Growth Team)) + various people being away around spring holidays, I am not confident on that date.

Let's pause and have a proper schedule being set then. :)

T304953: Schedule the deployment of "Add a link" to more wikis

@kevinbazira, all rounds have been created, they are sub-tasks of the current one. Please proceed on model training as you can! :)

@Trizek-WMF, thank you for creating all the rounds. I am working on generating datasets and models round by round and will be sharing updates on the sub-tasks.

The task generation for the Add Link task pool runs on one thread per DB server group (s1-s7). The two large DB server groups (in terms of number of wikis) are s5 (21 Wikipedias) and s3 (280 Wikipedias). For mid-sized wikis task generation took about 5-6 hours for a single wiki. It will probably take the same time for large wikis as well (we are going for a constant number of wikis, regardless of size); might take less time for small wikis where candidates can be exhausted before the required number of tasks are found. But assuming it doesn't, 280 Wikipedias is ~70 days, so we should probably run 4 instances of the refresh script in parallel on s3.

The script creates a lock with the ID of the wiki it is processing, so running multiple instances in parallel on the same dblist should be fine - the second thread will just skip the wiki that the first is already processing.

Proposal for streamlining the completion of remaining rounds:

1. Train models, verify models, publish datasets for all remaining wikis. This involves work from Machine-Learning-Team (cc @kevinbazira) and Research (cc @MGerlach). Growth doesn't need to be consulted on this phase; once these teams are happy with the models, the datasets can be published.
2. Growth engineers will populate the excluded section titles for all remaining wikis
3. Growth engineers will enable the backend for all remaining wikis, so the task pools begin to fill up
4. @Trizek-WMF can then verify the hasrecommendation:link results and check the API https://api.wikimedia.org/service/linkrecommendation/apidocs/#/default/get_v1_linkrecommendations__project___domain___page_title_
5. @Trizek-WMF can inform communities, and Growth engineers can enable the front-end, either staggered or en masse, depending on what works better for @Trizek-WMF

tl;dr I think we would all save time with context switching if we can do the model training, section title population, and backend enabling all at once, rather than in phases over a period of months. Then the actual presentation to communities could be done in a more staggered way if that is better from a community relations standpoint.

@KStoller-WMF @kevinbazira @MGerlach what do you think?

@kostajh +1 on doing the initial phases at once to avoid context switching.

The ML team will proceed with training models, evaluating them, and publishing datasets for all the remaining rounds.

As far as we keep a staggered deployment, I'm fine. :)

+1
This sounds like a great approach!

In T304110#8531763, @Trizek-WMF wrote:

When all wikis will have trained models, then we will resume deployments.

@kevinbazira
Do we have a rough ETA for when model training will be done for all Wikipedias? Thanks!

@KStoller-WMF, we are currently working on the 9th out of 18 rounds of wikis. Each of the 9 remaining rounds has ~20 models. ETA to train, evaluate, and publish all these models is about a month or more depending on the size of each of these wikis and whether or not we have to fine-tune the link recommendation algorithm to support a wiki's language-specific characters.

Will be sharing progress updates on the sub-tasks.

@Trizek-WMF Please further describe in this Phab ticket, what level of involvement is needed from CRS