Page MenuHomePhabricator

Recommendation API: Support pagination for single page collection recommendations
Closed, ResolvedPublic8 Estimated Story Points

Assigned To
Authored By
ngkountas
Jan 22 2025, 3:21 PM
Referenced Files
F72135058: 2026-02-16_11-47-00.png
Feb 17 2026, 1:06 AM
F72135056: 2026-02-16_11-45-45.png
Feb 17 2026, 1:06 AM
F72135053: 2026-02-16_11-21-46.png
Feb 17 2026, 1:06 AM
F72135036: 2026-02-16_11-21-40.png
Feb 17 2026, 1:06 AM
F71592037: 2026-01-22_09-49-21.png
Jan 22 2026, 11:47 PM
F71592035: 2026-01-22_09-49-00.png
Jan 22 2026, 11:47 PM
F71592033: 2026-01-22_09-48-52.png
Jan 22 2026, 11:47 PM

Description

In order to properly support the collection view dialog in CX unified dashboard (T379273), we need to add support for pagination for single page collection recommendations.

More specifically, we want to support a new page URL parameter and when this parameter is present, the API will return the corresponding page for the page collection recommendations, without any shuffling.

Edit:
Fetching single page collection recommendations in "pages", also allows us to fetch fewer suggestions per request, without the risk of fetching duplicate recommendations per request. However, the final approach for this "pagination" was decided to use a continue feature. That means that the API generates and returns a continue_seed that when used always shuffles the page collection in the same way, and a continue_offset which indicates the index of the last processed page in the collection. Using these "continuation" parameters, the caller can obtain the page collection recommendations in "pages" of any size, but only in sequential requests.

E.g. first page request (page size=6):
https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=6

Now using the returned from API continue_offset and continue_seed parameters we can get the second page. The returned offset from the above request is 8, thus we'll use a continue_offset equal to 9:
https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=6&continue_seed=887932336&continue_offset=9

In similar way, we can fetch the following pages.


Derived Requirement

Ensure that the Recommendation API supports pagination for single page collection recommendations by using continuation parameters. When a client provides continue_seed and continue_offset, the API must return the next sequential page of recommendations from the same deterministically shuffled collection, without introducing duplicate or reshuffled results across requests.

Test Steps

Test Case 1: Ensure initial collection recommendation request returns continuation parameters

  1. Send a request to the Recommendation API for single page collection recommendations with collections=true and a defined count value.
  2. Inspect the API response payload.
  3. ✅❓❌⬜ AC1: The response includes both continue_seed and continue_offset parameters along with the first page of recommendations.

Test Case 2: Ensure subsequent paginated request returns the next sequential set of recommendations

  1. Using the continue_seed and continue_offset returned from the initial request, send a second API request including these continuation parameters.
  2. Inspect the list of returned recommendations.
  3. ✅❓❌⬜ AC2: The response returns the next sequential page of recommendations from the same collection, without duplicates and without reshuffling compared to the initial request.

QA Results - TestWiki

ACStatusDetails
1T384485#11547237
2T384485#11621349

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ngkountas triaged this task as Medium priority.Jan 22 2025, 3:22 PM
ngkountas added a project: OKR-Work.
ngkountas added a project: Recommendation-API.
ngkountas moved this task from Prioritized to In-progress on the LPL Hypothesis board.

Change #1113513 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[research/recommendation-api@master] Add support for pagination for single page collections

https://gerrit.wikimedia.org/r/1113513

Change #1206409 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[research/recommendation-api@master] add support for "continue_from" for single page collections

https://gerrit.wikimedia.org/r/1206409

Change #1206999 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[mediawiki/extensions/ContentTranslation@master] CX api: Support new recommendation api "continue" feature

https://gerrit.wikimedia.org/r/1206999

Change #1113513 abandoned by Nik Gkountas:

[research/recommendation-api@master] Add support for pagination for single page collections

Reason:

A different approach followed in I3a435ef0545740797d1b8a2c4f4e1a6e9a8c26a4. Abandoning this one, in favor of the other patch.

https://gerrit.wikimedia.org/r/1113513

Change #1211223 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+20251125

https://gerrit.wikimedia.org/r/1211223

Change #1211679 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@wmf/1.46.0-wmf.3] CX3 Build 1.0.0+20251126

https://gerrit.wikimedia.org/r/1211679

Change #1211223 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+20251126

https://gerrit.wikimedia.org/r/1211223

Change #1211679 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@wmf/1.46.0-wmf.3] CX3 Build 1.0.0+20251126

https://gerrit.wikimedia.org/r/1211679

Mentioned in SAL (#wikimedia-operations) [2025-11-26T14:16:33Z] <sbisson@deploy2002> Started scap sync-world: Backport for [[gerrit:1211679|CX3 Build 1.0.0+20251126 (T384485)]]

Mentioned in SAL (#wikimedia-operations) [2025-11-26T14:18:47Z] <sbisson@deploy2002> sbisson: Backport for [[gerrit:1211679|CX3 Build 1.0.0+20251126 (T384485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-11-26T14:25:25Z] <sbisson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1211679|CX3 Build 1.0.0+20251126 (T384485)]] (duration: 08m 52s)

Change #1206409 merged by jenkins-bot:

[research/recommendation-api@master] add support for pagination for single page collections

https://gerrit.wikimedia.org/r/1206409

Change #1214195 had a related patch set uploaded (by KartikMistry; author: Sbisson):

[operations/deployment-charts@master] Update rec-api to 2025-12-02-200719-production

https://gerrit.wikimedia.org/r/1214195

Change #1214195 merged by jenkins-bot:

[operations/deployment-charts@master] Update rec-api to 2025-12-02-200719-production

https://gerrit.wikimedia.org/r/1214195

Change #1214539 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[research/recommendation-api@master] fix lead section size filtering for single page collections

https://gerrit.wikimedia.org/r/1214539

Change #1214540 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[research/recommendation-api@master] fix section suggestion fetching for single page collections

https://gerrit.wikimedia.org/r/1214540

Change #1214580 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@wmf/1.46.0-wmf.4] CX3 Build 1.0.0+20251126

https://gerrit.wikimedia.org/r/1214580

Change #1214580 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@wmf/1.46.0-wmf.4] CX3 Build 1.0.0+20251126

https://gerrit.wikimedia.org/r/1214580

Mentioned in SAL (#wikimedia-operations) [2025-12-03T17:31:41Z] <sbisson@deploy2002> Started scap sync-world: Backport for [[gerrit:1214580|CX3 Build 1.0.0+20251126 (T384485)]]

Mentioned in SAL (#wikimedia-operations) [2025-12-03T17:34:36Z] <sbisson@deploy2002> sbisson: Backport for [[gerrit:1214580|CX3 Build 1.0.0+20251126 (T384485)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-12-03T17:40:48Z] <sbisson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1214580|CX3 Build 1.0.0+20251126 (T384485)]] (duration: 09m 07s)

Change #1206999 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX api: Support new recommendation api "continue" feature

https://gerrit.wikimedia.org/r/1206999

Change #1214539 merged by jenkins-bot:

[research/recommendation-api@master] fix lead section size filtering for single page collections

https://gerrit.wikimedia.org/r/1214539

Change #1216781 had a related patch set uploaded (by Nik Gkountas; author: Nik Gkountas):

[research/recommendation-api@master] Fix continue offset and seed for section translation recommendations

https://gerrit.wikimedia.org/r/1216781

Change #1216781 merged by jenkins-bot:

[research/recommendation-api@master] Fix continue offset and seed for section translation recommendations

https://gerrit.wikimedia.org/r/1216781

Change #1214540 merged by jenkins-bot:

[research/recommendation-api@master] fix section suggestion fetching for single page collections

https://gerrit.wikimedia.org/r/1214540

Change #1216835 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+2025120

https://gerrit.wikimedia.org/r/1216835

Change #1217181 had a related patch set uploaded (by Sbisson; author: Sbisson):

[mediawiki/extensions/ContentTranslation@wmf/1.46.0-wmf.5] CX3 Build 1.0.0+20251209

https://gerrit.wikimedia.org/r/1217181

Change #1217182 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommendation API to 2025-12-09-164214-production

https://gerrit.wikimedia.org/r/1217182

Change #1217182 merged by jenkins-bot:

[operations/deployment-charts@master] Update Recommendation API to 2025-12-09-164214-production

https://gerrit.wikimedia.org/r/1217182

Mentioned in SAL (#wikimedia-operations) [2025-12-10T13:53:38Z] <kart_> Updated Recommendation API to 2025-12-09-164214-production (T384485, T409338, T409332)

Change #1216835 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@master] CX3 Build 1.0.0+20251209

https://gerrit.wikimedia.org/r/1216835

Change #1217181 merged by jenkins-bot:

[mediawiki/extensions/ContentTranslation@wmf/1.46.0-wmf.5] CX3 Build 1.0.0+20251209

https://gerrit.wikimedia.org/r/1217181

Mentioned in SAL (#wikimedia-operations) [2025-12-10T14:07:02Z] <sbisson@deploy2002> sbisson: Backport for [[gerrit:1217181|CX3 Build 1.0.0+20251209 (T384485 T408845 T409332 T409337 T409338 T411779)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-12-10T14:13:57Z] <sbisson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1217181|CX3 Build 1.0.0+20251209 (T384485 T408845 T409332 T409337 T409338 T411779)]] (duration: 09m 01s)

ngkountas set the point value for this task to 8.Jan 8 2026, 3:10 PM

@ngkountas Can you please review AC2, and let me know what you think?

Test Result - API

Status: ✅ PASS / ❓ Need More Info
Environment: API
OS: macOS Tahoe 26.2
Browser: Chrome 143
Device: MBA
Emulated Device: NA

Test Artifact(s):
https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=6&continue_seed=2743614298
https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=1&continue_seed=2743614298&continue_offset=3

Test Steps

Test Case 1: Ensure initial collection recommendation request returns continuation parameters

  1. Send a request to the Recommendation API for single page collection recommendations with collections=true and a defined count value.
  2. Inspect the API response payload.
  3. AC1: The response includes both continue_seed and continue_offset parameters along with the first page of recommendations.
CountContinue_seed
2026-01-22_09-48-52.png (1×1 px, 138 KB)
2026-01-22_09-49-00.png (1×1 px, 130 KB)

Test Case 2: Ensure subsequent paginated request returns the next sequential set of recommendations

  1. Using the continue_seed and continue_offset returned from the initial request, send a second API request including these continuation parameters.
  2. Inspect the list of returned recommendations.
  3. AC2: The response returns the next sequential page of recommendations from the same collection, without duplicates and without reshuffling compared to the initial request.

Based on AC1 screenshots. Besides "Hack", shouldn't it be "Proprietary software", since that's 3rd on the list? My offset is 3, not 2.

Continue_offset
2026-01-22_09-49-21.png (567×1 px, 85 KB)
GMikesell-WMF changed the task status from Open to In Progress.Jan 22 2026, 11:48 PM
GMikesell-WMF updated the task description. (Show Details)
GMikesell-WMF moved this task from Needs QA to In-progress on the LPL Hypothesis board.

@GMikesell-WMF the continue_offset indicates the number of page collection articles that have been processed, until the requested (count) amount of recommendations is found. For this reason, it cannot be used to skip any number of results, like you try to do in your example. The continue_offset parameter is intended to be used in the following way:

  1. Send a request without offset and seed(optional): https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=6
  2. The above request returns the seed number (e.g. 918523796) and the index of the last collection article (e.g. 18) that was processed before the 6 requested recommendations are returned. Now we can use these parameters to skip the 6 first recommendations like this by providing the given seed and setting the offset to returned_offset + 1: https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=6&continue_seed=918523796&continue_offset=19

@GMikesell-WMF the continue_offset indicates the number of page collection articles that have been processed, until the requested (count) amount of recommendations is found. For this reason, it cannot be used to skip any number of results, like you try to do in your example. The continue_offset parameter is intended to be used in the following way:

  1. Send a request without offset and seed(optional): https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=6
  2. The above request returns the seed number (e.g. 918523796) and the index of the last collection article (e.g. 18) that was processed before the 6 requested recommendations are returned. Now we can use these parameters to skip the 6 first recommendations like this by providing the given seed and setting the offset to returned_offset + 1: https://api.wikimedia.org/service/lw/recommendation/api/v1/translation?source=en&target=el&collections=true&seed=Software&count=6&continue_seed=918523796&continue_offset=19

I see now regarding the processed articles, until the count is found. I will move this to Sign-off. Thanks for all your work!

Software&count=6Seed & offset"continue_offset": 19"continue_offset": 20
2026-02-16_11-21-40.png (1×1 px, 136 KB)
2026-02-16_11-21-46.png (1×1 px, 140 KB)
2026-02-16_11-45-45.png (1×1 px, 163 KB)
2026-02-16_11-47-00.png (1×1 px, 153 KB)