Page MenuHomePhabricator
Create Task
Maniphest T392944

Enable the iterative way of refreshing LinkRecommendations for all wikis
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

In the context of the parent task, we created the functionality to iterate through all the articles of a wiki and check them for link suggestions.

This was enabled for a first set of wikis: fr, es, fa, arz, cs, pt, and id via the GELinkRecommendationsRefreshByIteratingThroughAllTitles configuration option. So far, we have not reeived any negative feedback and it has led to a lot more suggestions in those wikis without making any of the risk-predicting metrics noticeably worse.

Since it doesn't seem to cause any problems, let's enable it on all wikis.

Acceptance criteria:

  • actually enable the iterative mode for all wikis
  • clean up the per-topic mode related code

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
[Growth] Remove unused config variablesUrbanecmoperations/mediawiki-configmaster+0 -14
refreshLinkRecommendations: Remove obsolete double-count guardUrbanecmmediawiki/extensions/GrowthExperimentsmaster+2 -10
listTaskCounts: Correct the --statsd option descriptionUrbanecmmediawiki/extensions/GrowthExperimentsmaster+1 -1
NewcomerTasks: Remove ORES topic based Add Link task pool populationUrbanecmmediawiki/extensions/GrowthExperimentsmaster+24 -452
[Growth] Enable iterative Add Link task pool population on all wikisUrbanecmoperations/mediawiki-configmaster+1 -18
Enable GELinkRecommendationsRefreshByIteratingThroughAllTitles by defaultUrbanecmmediawiki/extensions/GrowthExperimentsmaster+1 -1
Growth: enable new way of refreshing LinkRecommendations for more wikisMichael Großeoperations/mediawiki-configmaster+10 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Michael created this task.Apr 29 2025, 6:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 29 2025, 6:21 PM
Michael added a comment.May 21 2025, 2:49 PM

Copying my comment from the parent T386250:

Since we enabled the new way of doing things a while ago and things are looking good so far, we maybe can take the next step and enable it on a bunch more wikis.

My suggestion, separated by db-sections would be:

Section 1:

  • enwiki?

Section 2:

  • itwiki
  • nlwiki

Section 5:

  • dewiki
  • srwiki
  • shwiki

Section 6:

  • ruwiki

Section 7:

  • cawiki
  • heviki
  • viwiki
DMburugu subscribed.May 21 2025, 2:55 PM

it has led to a lot more suggestions in those wikis without making any of the risk-predicting metrics noticeably worse.

Based on that, is this something we can do in the next sprint to continue improving Link Recommendation efficiency?

Michael added a comment.May 21 2025, 4:57 PM
In T392944#10844666, @DMburugu wrote:

it has led to a lot more suggestions in those wikis without making any of the risk-predicting metrics noticeably worse.

Based on that, is this something we can do in the next sprint to continue improving Link Recommendation efficiency?

Yes! I wanted to wait until the migration to kubernetes for that job is finished, but this has just happened (see T385782#10845115) and so this task should now be ready for the next sprint.

Sgs moved this task from Inbox to Backlog on the Growth-Team board.Jun 9 2025, 4:45 PM
KStoller-WMF triaged this task as Medium priority.Jun 19 2025, 3:33 PM
KStoller-WMF subscribed.
gerritbot added a comment.Jun 26 2025, 6:17 PM

Change #1164287 had a related patch set uploaded (by Michael Große; author: Michael Große):

[operations/mediawiki-config@master] Growth: enable new way of refreshing LinkRecommendations for more wikis

https://gerrit.wikimedia.org/r/1164287

gerritbot added a project: Patch-For-Review.Jun 26 2025, 6:17 PM
gerritbot added a comment.Jul 28 2025, 1:27 PM

Change #1164287 merged by jenkins-bot:

[operations/mediawiki-config@master] Growth: enable new way of refreshing LinkRecommendations for more wikis

https://gerrit.wikimedia.org/r/1164287

Stashbot added a comment.Jul 28 2025, 1:27 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-28T13:27:24Z] <lucaswerkmeister-wmde@deploy1003> Started scap sync-world: Backport for [[gerrit:1164287|Growth: enable new way of refreshing LinkRecommendations for more wikis (T386250 T392944)]], [[gerrit:1171548|Echo: be explicit about special wikis using Wikipedia logo (T400070)]]

Stashbot mentioned this in T386250: Rewrite refreshLinkRecommendations to not iterate through article topics.Jul 28 2025, 1:27 PM
Stashbot mentioned this in T400070: Clean up Echo setting on ex-wikipedia special wikis.

Mentioned in SAL (#wikimedia-operations) [2025-07-28T13:29:21Z] <lucaswerkmeister-wmde@deploy1003> lucaswerkmeister-wmde, migr: Backport for [[gerrit:1164287|Growth: enable new way of refreshing LinkRecommendations for more wikis (T386250 T392944)]], [[gerrit:1171548|Echo: be explicit about special wikis using Wikipedia logo (T400070)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2025, 1:32 PM
Stashbot added a comment.Jul 28 2025, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2025-07-28T13:35:31Z] <lucaswerkmeister-wmde@deploy1003> Finished scap sync-world: Backport for [[gerrit:1164287|Growth: enable new way of refreshing LinkRecommendations for more wikis (T386250 T392944)]], [[gerrit:1171548|Echo: be explicit about special wikis using Wikipedia logo (T400070)]] (duration: 08m 06s)

KStoller-WMF moved this task from Backlog to Triaged on the Growth-Team board.Oct 27 2025, 5:34 PM
Urbanecm_WMF renamed this task from Enable the iterative way of refreshing LinkRecommendations for more wikis to Enable the iterative way of refreshing LinkRecommendations for all wikis.Tue, Sep 8, 3:32 PM
Urbanecm_WMF subscribed.

Since this task is not closed, I renamed it to be about doing it for all wikis.

Urbanecm_WMF updated the task description. (Show Details)Tue, Sep 8, 3:33 PM
Urbanecm_WMF set the point value for this task to 2.Tue, Sep 8, 3:38 PM
Urbanecm_WMF moved this task from Triaged to Up next Sprint on the Growth-Team board.

This would be important to do with the switch to interest based topics. Estimated within a conversation with @Michael, per agreement with them, moving for consideration in next sprint.

KStoller-WMF assigned this task to Urbanecm_WMF.Tue, Sep 8, 4:53 PM
KStoller-WMF moved this task from Up next Sprint to FY2026-27 Q1 Sprint 6 on the Growth-Team board.
KStoller-WMF edited projects, added: Growth-Team (FY2026-27 Q1 Sprint 6); removed: Growth-Team.
Urbanecm_WMF updated the task description. (Show Details)Tue, Sep 8, 8:29 PM
gerritbot added a comment.Tue, Sep 8, 8:33 PM

Change #1338031 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] [Growth] Enable iterative Add Link task pool population on all wikis

https://gerrit.wikimedia.org/r/1338031

gerritbot added a project: Patch-For-Review.Tue, Sep 8, 8:33 PM
gerritbot added a comment.Tue, Sep 8, 8:38 PM

Change #1338038 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] Enable GELinkRecommendationsRefreshByIteratingThroughAllTitles by default

https://gerrit.wikimedia.org/r/1338038

gerritbot added a comment.Wed, Sep 9, 10:31 AM

Change #1338038 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Enable GELinkRecommendationsRefreshByIteratingThroughAllTitles by default

https://gerrit.wikimedia.org/r/1338038

ReleaseTaggerBot added a project: MW-1.47-notes (1.47.0-wmf.20; 2026-09-15).Wed, Sep 9, 11:00 AM
gerritbot added a comment.Wed, Sep 9, 11:30 AM

Change #1338031 merged by jenkins-bot:

[operations/mediawiki-config@master] [Growth] Enable iterative Add Link task pool population on all wikis

https://gerrit.wikimedia.org/r/1338031

Maintenance_bot removed a project: Patch-For-Review.Wed, Sep 9, 11:30 AM
Stashbot added a comment.Wed, Sep 9, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-09T11:31:03Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]]

Stashbot added a comment.Wed, Sep 9, 11:35 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-09T11:35:26Z] <urbanecm@deploy1003> urbanecm: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Stashbot added a comment.Wed, Sep 9, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-09T11:53:01Z] <urbanecm@deploy1003> Finished scap sync-world: Backport for [[gerrit:1338031|[Growth] Enable iterative Add Link task pool population on all wikis (T392944)]] (duration: 21m 58s)

Stashbot added a comment.Wed, Sep 9, 3:17 PM

Mentioned in SAL (#wikimedia-operations) [2026-09-09T15:17:44Z] <urbanecm> Delete all running periodic jobs starting with growthexperiments-refreshlinkrecommendations-* (to pick up new configuration; T392944)

gerritbot added a comment.Wed, Sep 9, 3:51 PM

Change #1338242 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] NewcomerTasks: Remove ORES topic based Add Link task pool population

https://gerrit.wikimedia.org/r/1338242

gerritbot added a project: Patch-For-Review.Wed, Sep 9, 3:51 PM
Urbanecm_WMF moved this task from Incoming to Doing on the Growth-Team (FY2026-27 Q1 Sprint 6) board.Wed, Sep 9, 3:52 PM
Urbanecm_WMF moved this task from Doing to Code Review on the Growth-Team (FY2026-27 Q1 Sprint 6) board.Thu, Sep 10, 9:17 AM
gerritbot added a comment.Fri, Sep 11, 1:49 PM

Change #1338242 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] NewcomerTasks: Remove ORES topic based Add Link task pool population

https://gerrit.wikimedia.org/r/1338242

Maintenance_bot removed a project: Patch-For-Review.Fri, Sep 11, 2:31 PM
gerritbot added a comment.Mon, Sep 14, 12:21 PM

Change #1341161 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] [Growth] Remove unused config variables

https://gerrit.wikimedia.org/r/1341161

gerritbot added a project: Patch-For-Review.Mon, Sep 14, 12:21 PM
gerritbot added a comment.Mon, Sep 14, 12:26 PM

Change #1341164 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] listTaskCounts: Correct the --statsd option description

https://gerrit.wikimedia.org/r/1341164

gerritbot added a comment.Mon, Sep 14, 12:27 PM

Change #1341165 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[mediawiki/extensions/GrowthExperiments@master] refreshLinkRecommendations: Remove obsolete double-count guard

https://gerrit.wikimedia.org/r/1341165

Urbanecm_WMF moved this task from Code Review to Blocked / Needs Work on the Growth-Team (FY2026-27 Q1 Sprint 6) board.Mon, Sep 14, 12:29 PM

Cleanup config should be deployed on Sept 21. Moving to Blocked.

Urbanecm_WMF mentioned this in T437889: Clean-up and speed up GrowthExperiments:listTaskCounts.php.Mon, Sep 14, 1:03 PM
gerritbot added a comment.Mon, Sep 14, 1:19 PM

Change #1341164 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] listTaskCounts: Correct the --statsd option description

https://gerrit.wikimedia.org/r/1341164

gerritbot added a comment.Mon, Sep 14, 1:19 PM

Change #1341165 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] refreshLinkRecommendations: Remove obsolete double-count guard

https://gerrit.wikimedia.org/r/1341165

Urbanecm_WMF added a comment.Mon, Sep 14, 1:35 PM

Logged a possible cleanup from this as T437889: Clean-up and speed up GrowthExperiments:listTaskCounts.php, as it would need some extra scrutiny.

DMburugu added a project: Essential-Work.Mon, Sep 14, 1:55 PM
Urbanecm_WMF mentioned this in T438474: GrowthExperiments: Remove the unused /suggestions/info REST endpoint.Fri, Sep 18, 12:52 PM
gerritbot added a comment.Mon, Sep 21, 8:28 AM

Change #1341161 merged by jenkins-bot:

[operations/mediawiki-config@master] [Growth] Remove unused config variables

https://gerrit.wikimedia.org/r/1341161

Stashbot added a comment.Mon, Sep 21, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-21T08:29:03Z] <urbanecm@deploy1003> Started scap sync-world: Backport for [[gerrit:1341161|[Growth] Remove unused config variables (T392944)]]

Maintenance_bot removed a project: Patch-For-Review.Mon, Sep 21, 8:31 AM
Urbanecm_WMF moved this task from Blocked / Needs Work to QA on the Growth-Team (FY2026-27 Q1 Sprint 6) board.Mon, Sep 21, 8:31 AM
Stashbot added a comment.Mon, Sep 21, 9:01 AM

Mentioned in SAL (#wikimedia-operations) [2026-09-21T09:01:57Z] <urbanecm@deploy1003> Finished scap sync-world: Backport for [[gerrit:1341161|[Growth] Remove unused config variables (T392944)]] (duration: 32m 54s)

DMburugu edited projects, added: Growth-Team (FY2026-27 Q1 Sprint 7); removed: Growth-Team (FY2026-27 Q1 Sprint 6).Tue, Sep 22, 4:38 PM
DMburugu moved this task from Incoming to QA on the Growth-Team (FY2026-27 Q1 Sprint 7) board.
Etonkovidova closed this task as Resolved.Wed, Sep 30, 2:29 PM
Etonkovidova subscribed.

Checked wmf.21 - Link Recommendation task pool is stable (not declining) link; logstash Link Recommendation dashboard (link) looks good too.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits