Page MenuHomePhabricator
Create Task
Maniphest T361926

Improve training and inference pipeline for multilingual link recommendation model
Closed, ResolvedPublic
Actions

Description

In T354659, we did exploratory work showing that it is feasible to build a single model that works for many (>50) or even all languages.
However, the current architecture of the model is not suitable to easily maintain such a model; for example, the training pipeline using a bashscript is not suitable to automate the creation/updating of multilingual models.

In this task, we want to improve the pipelines for training and inference.

  • developing an end-to-end training pipeline for the multilingual model; ideally with airflow
  • using the pipeline to train one or few (>=5) models for all languages (i.e. figure out the best way to group languages)
  • revise the inference pipeline for the new model (keeping in mind the requirements to serve it via LiftWing later on) (this will be sketched out in a separate task)

Related Objects
Search...

Event Timeline

MGerlach created this task.Apr 5 2024, 10:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 5 2024, 10:36 AM
MGerlach added a project: Research (FY2023-24-Research-April-June).Apr 5 2024, 10:38 AM
MGerlach updated the task description. (Show Details)
MGerlach mentioned this in T361929: [Research Engineering Request] Building end-to-end training pipeline for the add-a-link model.Apr 5 2024, 11:04 AM
AKhatun_WMF added a comment.Apr 15 2024, 3:46 AM

Update week 8 to 14 April 2024:

  • Went over airflow and research dataset repos
  • Sketched an overview of our current code base workflow and a few research airflow repos.
AKhatun_WMF added a comment.Apr 20 2024, 4:42 AM

Update week 15 to 21 April 2024:

  • Discussed how to change our code to Airflow friendly version
  • Identifying changes and decisions to be made wrt add-a-link repo
AKhatun_WMF added a comment.Apr 28 2024, 2:44 PM

Update week of 22-28 April, 2024:

  • Refactor code and add CLI arguments.
  • Set up dev airflow instances.
AKhatun_WMF added a comment.May 2 2024, 9:32 PM

Update week of 29 April - 5 May, 2024:

  • Set up code in research_datasets, create a test file
  • Chat with Fabian on how to connect code to airflow_dags repo
  • Set up airflow dags repo properly and resolve issues setting up a test dag
AKhatun_WMF added a comment.May 11 2024, 2:28 PM

Update week of 6 - 12 May, 2024:

  • successfully ran test airflow dag, included a main file and fsspec for testing
  • created all required functions in main file.
  • creating dag for the entire pipeline.
AKhatun_WMF added a comment.May 19 2024, 11:59 AM

Update week of 13 - 19 May, 2024:

  • Rewrite code for airflow
  • Start testing code
AKhatun_WMF added a comment.May 23 2024, 2:51 AM

Update week of 20 - 26 May, 2024:

  • Debug and fix code to run airflow dag
AKhatun_WMF added a comment.May 31 2024, 3:34 PM

Update week of 27 May - 02 June, 2024:

  • Finish debugging. Fix dag and research-datasets code. Push updated code to both repos.
  • fix moving embedding to hdfs
  • fix ICU import
  • push MR
AKhatun_WMF added a comment.Jun 11 2024, 9:17 PM

Update week of 03 June - 09 June, 2024:

  • fixed pre-commit errors
  • find bug that causes drop in performance
AKhatun_WMF added a comment.Jun 17 2024, 3:41 PM

Update week of 10 June - 16 June, 2024:

  • found and fixed bug causing precision loss
  • updated dag to follow guidelines, fixed formatting and ran tests
  • added shards in dag (to be used later)
  • airflow-dags MR sent
  • worked on fixing memory issues. converted a script to use spark instead if purely python.
  • Rebase MR and make changes in dag (to incorporate new changes in research-datasets) [New changes not used yet]
AKhatun_WMF added a comment.Jun 27 2024, 2:46 PM

Update week of 17 June - 23 June, 2024:

  • Change read/write methods to fix memory issues. Fixed for small-medium wikis. Additional errors for larger wikis (e.g. enwiki, jawiki)
  • Refactor generate_anchor_dictionary script to modularize better.

Update week of 24 June - 30 June, 2024:

MGerlach moved this task from FY2023-24-Research-April-June to FY2024-25-Research-July-September on the Research board.Jul 1 2024, 10:07 AM
MGerlach edited projects, added Research (FY2024-25-Research-July-September); removed Research (FY2023-24-Research-April-June).
MGerlach added a comment.Jul 1 2024, 10:13 AM

Moving this to the next quarter (FY2024-25-Research-July-September) as the work is not yet fully completed

  • the MR for the pipeline in airflow is submitted.
  • this needs some code-review by research-engineering before it can be merged. this should be completed by mid-July (see T361929#9935541)
  • we expect to resolve the task in the next 1-2 weeks
Miriam added a project: Essential-Work.Aug 12 2024, 4:46 PM
Miriam claimed this task.Aug 13 2024, 5:47 PM
Miriam added a subtask: T361929: [Research Engineering Request] Building end-to-end training pipeline for the add-a-link model.
Miriam moved this task from FY2024-25-Research-July-September to In Progress on the Research board.
Miriam edited projects, added Research; removed Research (FY2024-25-Research-July-September).
fkaelin closed subtask T361929: [Research Engineering Request] Building end-to-end training pipeline for the add-a-link model as Resolved.Aug 24 2024, 2:26 AM
MGerlach claimed this task.Oct 18 2024, 3:03 PM
  • training pipeline with airflow has been merged (MR)
  • will run some tests of the pipeline in the next week(s)
  • inference pipeline will need a separate task as it requires some additional discussion on how to best approach that. see some context in Fabian's recent presentation (slidedeck)
MGerlach updated the task description. (Show Details)Oct 18 2024, 3:04 PM
MGerlach mentioned this in T276438: Establish processes for running the dataset pipeline.Oct 24 2024, 12:37 PM
Isaac mentioned this in T383614: [Q3 FY 24-25 Applied Science] Knowledge Gaps Research.Jan 13 2025, 7:37 PM
Isaac triaged this task as Medium priority.Jan 13 2025, 11:00 PM
Michael subscribed.Feb 13 2025, 4:35 PM
MGerlach updated the task description. (Show Details)Feb 19 2025, 4:37 PM
MGerlach closed this task as Resolved.Feb 19 2025, 4:53 PM
MGerlach added subscribers: • MunizaA, fkaelin.

Validated the airflow-dag for the training pipeline.

  • Successfully ran the pipeline to train individual models for 6 different languages
  • Backtesting evaluation is similar to results from previous models
wiki_dbthresholdN_testPrecision (prev)Recall (prev)
arwiki0.510000.996 (0.75)0.394 (0.34)
bnwiki0.510000.666 (0.75)0.257 (0.30)
cswiki0.510000.873 (0.78)0.447 (0.44)
viwiki0.510000.943 (0.89)0.806 (0.65)
simplewiki0.510000.802 (0.79)0.406 (0.45)
enwiki0.510000.846 (0.81)0.410 (0.45)

Updated documentation: https://meta.wikimedia.org/wiki/Research:Improving_multilingual_support_for_link_recommendation_model_for_add-a-link_task#Building_the_Pipeline

Notes:

  • Marking this task as completed. However, this completes the work only for the training pipeline. This can be used to train either individual models for each language or aggregate models for several (or all) languages.
  • Additional work needs to be done on how to integrate this new training pipeline into the deployed addalink service. There are several ways one could do inference once we have trained the models.
  • Aligning the training pipeline with the inference service will be captured in a separate task.

Many thanks to @MunizaA and @fkaelin for their incredible support in moving the pipeline to an airflow-dag.

MGerlach mentioned this in T387556: Add Link training pipeline gets stuck for certain wikis.Feb 28 2025, 11:59 AM
MGerlach mentioned this in T388258: Make airflow-dag for addalink training pipeline output compatible with deployed model.Mar 7 2025, 4:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits