Page MenuHomePhabricator
Create Task
Maniphest T416877

Remove pipeline training code from research/mwaddlink
Open, LowPublic
Actions

Description

Historically, the research/mwaddlink repository on Gerrit contained two related-but-distinct codebases:

  1. The training code to generate Add-Link-Structured-Task models
  2. The Link Recommendation service to generate recommendations based on those models

With the migration towards Airflow, this has changed. Nowadays, the repository is only used for the Link Recommendation service itself. Model training code is present in the Airflow DAGs repository on GitLab (more specific link).

In a discussion with @OKarakaya-WMF, @Urbanecm_WMF and Ozge decided to remove the pipeline training code from research/mwaddlink to avoid confusion (example patch from a confused author), replacing it with a link to the new location.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
linkrecommendation: Bump versionoperations/deployment-chartsmaster+1 -1
cleanup: Remove training coderesearch/mwaddlinkmain+6 -2 K
Customize query in gerrit

Related Objects

Event Timeline

Urbanecm_WMF created this task.Feb 9 2026, 1:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 9 2026, 1:55 PM
Urbanecm_WMF mentioned this in T416878: Update Add Link Wikitech documentation.Feb 9 2026, 1:57 PM
gerritbot added a comment.Feb 9 2026, 2:30 PM

Change #1237917 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[research/mwaddlink@main] [WIP] cleanup: Remove training code

https://gerrit.wikimedia.org/r/1237917

gerritbot added a project: Patch-For-Review.Feb 9 2026, 2:30 PM
Urbanecm_WMF mentioned this in T416880: research/mwaddlink has a broken CI.Feb 9 2026, 2:43 PM
Urbanecm_WMF triaged this task as Low priority.Feb 9 2026, 5:02 PM
Urbanecm_WMF added a project: Growth Team Maintenance.Feb 10 2026, 11:46 AM
Urbanecm_WMF added a comment.Feb 10 2026, 11:49 AM

Let's start with removing the code itself. I'll work on cleaning up requirements.txt (and hopefully the overall size of the image) once we know the code removal doesn't break anything.

Urbanecm mentioned this in rRMWAd67ce05117c4: build: Install slightly newer version of Pandas.Wed, Feb 11, 1:13 PM
gerritbot added a comment.Tue, Feb 17, 5:57 PM

Change #1237917 merged by jenkins-bot:

[research/mwaddlink@main] cleanup: Remove training code

https://gerrit.wikimedia.org/r/1237917

Urbanecm mentioned this in rRMWA4372b3e3533c: cleanup: Remove training code.Tue, Feb 17, 6:01 PM
Maintenance_bot removed a project: Patch-For-Review.Tue, Feb 17, 6:31 PM
gerritbot added a comment.Tue, Feb 17, 8:22 PM

Change #1240045 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/deployment-charts@master] linkrecommendation: Bump version

https://gerrit.wikimedia.org/r/1240045

gerritbot added a project: Patch-For-Review.Tue, Feb 17, 8:22 PM
gerritbot added a comment.Tue, Feb 17, 8:27 PM

Change #1240045 merged by jenkins-bot:

[operations/deployment-charts@master] linkrecommendation: Bump version

https://gerrit.wikimedia.org/r/1240045

Maintenance_bot removed a project: Patch-For-Review.Tue, Feb 17, 8:30 PM
DMburugu subscribed.Thu, Mar 12, 1:53 PM

@Urbanecm_WMF was this work concluded? I remember we had considered working on it in a certain sprint

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits