Page MenuHomePhabricator
Create Task
Maniphest T406854

Collection data unavailable in several rec-api hosts
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

It was discovered Oct 8 2025 around noon UTC that collections were not working as expected in the CX dashboard.

Upon closer inspection, it looks like some recommendation api instances have their collection cache filed and some don't.

Based on a test deployment to staging, it looks like the calls from the rec-api to mediawiki are refused by the localhost:6500 proxy. (example)

Derived Requirement

All rec-api hosts must successfully retrieve and serve collection data to the CX dashboard. Calls from the recommendation API to MediaWiki must not be blocked or refused by the localhost:6500 proxy. Each rec-api instance must have a functioning and populated collection cache so that collection-based recommendations consistently load for all users.

Test Steps

Test Case 1: Verify rec-api Host Serves Collection Data

  1. Directly query a rec-api host endpoint that returns collection data (staging or beta).
  2. Observe the response payload.
  3. ✅❓❌⬜ AC1: Confirm that the rec-api host returns populated collection data rather than empty or missing results.

QA Results - Logstash

ACStatusDetails
1✅T406854#11403708

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update Recommendation API to 2025-11-17-092813-productionoperations/deployment-chartsmaster+2 -2
Error handling and update scheduleresearch/recommendation-apimaster+17 -22
Update Recommnedation API to 2025-11-07-162011-productionoperations/deployment-chartsmaster+2 -6
Update Recommnedation API to 2025-11-05-230545-productionoperations/deployment-chartsmaster+2 -6
Improve periodic update flow and error handlingresearch/recommendation-apimaster+17 -15
Update Recommendation API to 2025-10-09-145754-productionoperations/deployment-chartsmaster+2 -2
Common log prefix for cache update code pathsresearch/recommendation-apimaster+4 -4
Customize query in gerrit

Related Objects

Event Timeline

SBisson created this task.Oct 9 2025, 12:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 9 2025, 12:19 PM
SBisson triaged this task as Unbreak Now! priority.Oct 9 2025, 12:19 PM
gerritbot added a comment.Oct 9 2025, 2:31 PM

Change #1194958 had a related patch set uploaded (by Sbisson; author: Sbisson):

[research/recommendation-api@master] Common log prefix for cache update code paths

https://gerrit.wikimedia.org/r/1194958

gerritbot added a project: Patch-For-Review.Oct 9 2025, 2:31 PM
gerritbot added a comment.Oct 9 2025, 2:57 PM

Change #1194958 merged by jenkins-bot:

[research/recommendation-api@master] Common log prefix for cache update code paths

https://gerrit.wikimedia.org/r/1194958

gerritbot added a comment.Oct 9 2025, 3:16 PM

Change #1194971 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommendation API to 2025-10-09-145754-production

https://gerrit.wikimedia.org/r/1194971

gerritbot added a comment.Oct 9 2025, 3:23 PM

Change #1194971 merged by jenkins-bot:

[operations/deployment-charts@master] Update Recommendation API to 2025-10-09-145754-production

https://gerrit.wikimedia.org/r/1194971

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2025, 3:31 PM
KartikMistry subscribed.Oct 9 2025, 4:06 PM
SBisson updated the task description. (Show Details)Oct 9 2025, 4:08 PM
SBisson added a project: Machine-Learning-Team.

Tagging Machine-Learning-Team in case they can provide insight into what's happening with the localhost:6500 proxy

eamedina subscribed.Oct 9 2025, 5:38 PM
SBisson lowered the priority of this task from Unbreak Now! to Medium.Oct 10 2025, 3:17 PM

It looks like all rec-api instances have been able to update their cache in the last hour.

I'm hesitant to close this task since there was clearly a problem that we don't understand but I'm not sure what should be the next step of the investigation. Maybe even more logging?

SBisson removed SBisson as the assignee of this task.Oct 10 2025, 3:18 PM
SBisson updated Other Assignee, removed: KartikMistry.
SBisson moved this task from In-progress to Incoming on the LPL Hypothesis board.

Pushing back to incoming since I'm not working on it and I'm not even sure what should be done.

SBisson closed this task as Resolved.Oct 24 2025, 6:46 PM
SBisson claimed this task.

I don't think we can do anything about it at this point.

Nikerabbit removed a project: LPL Hypothesis.Oct 27 2025, 11:18 AM
achou moved this task from Unsorted to 2025-2026 Q2 Done on the Machine-Learning-Team board.Oct 28 2025, 3:07 PM
SBisson reopened this task as Open.Oct 29 2025, 6:45 PM

We're seeing this problem again right now.

SBisson added a project: LPL Hypothesis.Oct 29 2025, 6:48 PM
SBisson changed the task status from Open to In Progress.Oct 30 2025, 5:54 PM
SBisson mentioned this in T407894: Recommendation api should remove untagged page collections.
SBisson mentioned this in T407895: Recommendation api should not show page collections with no articles.
SBisson moved this task from Incoming to In-progress on the LPL Hypothesis board.
gerritbot added a comment.Nov 4 2025, 4:18 PM

Change #1201718 had a related patch set uploaded (by Sbisson; author: Sbisson):

[research/recommendation-api@master] Improve periodic update flow and error handling

https://gerrit.wikimedia.org/r/1201718

gerritbot added a project: Patch-For-Review.Nov 4 2025, 4:18 PM
SBisson moved this task from In-progress to Needs Code Review on the LPL Hypothesis board.Nov 4 2025, 8:42 PM
gerritbot added a comment.Nov 5 2025, 11:58 AM

Change #1201718 merged by jenkins-bot:

[research/recommendation-api@master] Improve periodic update flow and error handling

https://gerrit.wikimedia.org/r/1201718

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2025, 12:31 PM
gerritbot added a comment.Nov 6 2025, 5:57 AM

Change #1202376 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-05-230545-production

https://gerrit.wikimedia.org/r/1202376

gerritbot added a project: Patch-For-Review.Nov 6 2025, 5:57 AM
gerritbot added a comment.Nov 6 2025, 1:31 PM

Change #1202376 merged by jenkins-bot:

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-05-230545-production

https://gerrit.wikimedia.org/r/1202376

Maintenance_bot removed a project: Patch-For-Review.Nov 6 2025, 2:32 PM
SBisson moved this task from Needs Code Review to Pending deployment on the LPL Hypothesis board.Fri, Nov 7, 9:02 PM
Nikerabbit added a project: LPL Projects (Other).Mon, Nov 10, 1:07 PM
gerritbot added a comment.Mon, Nov 10, 1:27 PM

Change #1203450 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-07-162011-production

https://gerrit.wikimedia.org/r/1203450

gerritbot added a project: Patch-For-Review.Mon, Nov 10, 1:27 PM
gerritbot added a comment.Mon, Nov 10, 1:37 PM

Change #1203450 merged by jenkins-bot:

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-07-162011-production

https://gerrit.wikimedia.org/r/1203450

Stashbot mentioned this in T408936: Error when calling the wikidata API with no titles.Mon, Nov 10, 2:31 PM
Stashbot mentioned this in T408937: Faulty error handling when fetching language pairs.
Stashbot mentioned this in T408934: Production error: AttributeError: 'NoneType' object has no attribute 'keys'.

Mentioned in SAL (#wikimedia-operations) [2025-11-10T14:31:12Z] <kart_> Update Recommnedation API to 2025-11-07-162011-production (T405000, T406854, T408936, T408937, T408934)

Stashbot mentioned this in T405000: Handle failure to load languages from cx server.Mon, Nov 10, 2:31 PM
Maintenance_bot removed a project: Patch-For-Review.Mon, Nov 10, 2:32 PM
SBisson moved this task from Pending deployment to Needs QA on the LPL Hypothesis board.Wed, Nov 12, 1:10 PM
SBisson raised the priority of this task from Medium to High.Thu, Nov 13, 6:07 PM
SBisson moved this task from Needs QA to In-progress on the LPL Hypothesis board.

This is happening again.

According to log stash all instances have updated their cache successfully but when queried, the /page-collections endpoint is returning [] about 75% of the time.

gerritbot added a comment.Thu, Nov 13, 6:48 PM

Change #1204944 had a related patch set uploaded (by Sbisson; author: Sbisson):

[research/recommendation-api@master] Error handling and update schedule

https://gerrit.wikimedia.org/r/1204944

gerritbot added a project: Patch-For-Review.Thu, Nov 13, 6:48 PM
SBisson moved this task from In-progress to Needs Code Review on the LPL Hypothesis board.Thu, Nov 13, 7:27 PM
gerritbot added a comment.Thu, Nov 13, 10:11 PM

Change #1204944 merged by jenkins-bot:

[research/recommendation-api@master] Error handling and update schedule

https://gerrit.wikimedia.org/r/1204944

Maintenance_bot removed a project: Patch-For-Review.Thu, Nov 13, 10:30 PM
SBisson moved this task from Needs Code Review to Pending deployment on the LPL Hypothesis board.Fri, Nov 14, 4:43 PM
Nikerabbit added a project: Unplanned-Sprint-Work.Mon, Nov 17, 2:20 PM
Nikerabbit set the point value for this task to 8.
gerritbot added a comment.Tue, Nov 18, 12:16 PM

Change #1206859 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommendation API to 2025-11-17-092813-production

https://gerrit.wikimedia.org/r/1206859

gerritbot added a project: Patch-For-Review.Tue, Nov 18, 12:16 PM
gerritbot added a comment.Tue, Nov 18, 1:11 PM

Change #1206859 merged by jenkins-bot:

[operations/deployment-charts@master] Update Recommendation API to 2025-11-17-092813-production

https://gerrit.wikimedia.org/r/1206859

Maintenance_bot removed a project: Patch-For-Review.Tue, Nov 18, 1:30 PM
Stashbot added a comment.Tue, Nov 18, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2025-11-18T13:35:51Z] <kart_> Update Recommendation API to 2025-11-17-092813-production (T406854)

SBisson moved this task from Pending deployment to Needs QA on the LPL Hypothesis board.Wed, Nov 19, 2:51 AM
GMikesell-WMF updated Other Assignee, added: GMikesell-WMF.Tue, Nov 25, 12:27 AM
GMikesell-WMF updated the task description. (Show Details)
GMikesell-WMF subscribed.Tue, Nov 25, 12:30 AM

@SBisson Recommendation API is showing that it's not emptying the cache, as seen in the screenshot. I will move this to Sign-off. Thanks for all your work!

Test Result - Logstash

Status: ✅ PASS
Environment: Logstash
OS: macOS Tahoe 26.1
Browser: Chrome 142
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 1: Verify rec-api Host Serves Collection Data

  1. Directly query a rec-api host endpoint that returns collection data (staging or beta).
  2. Observe the response payload.
  3. ✅ AC1: Confirm that the rec-api host returns populated collection data rather than empty or missing results.

2025-11-24_16-15-38.png (881×1 px, 409 KB)

GMikesell-WMF updated Other Assignee, removed: GMikesell-WMF.Tue, Nov 25, 12:30 AM
GMikesell-WMF updated the task description. (Show Details)
GMikesell-WMF moved this task from Needs QA to Design Signoff on the LPL Hypothesis board.
Nikerabbit closed this task as Resolved.Tue, Nov 25, 1:15 PM
Nikerabbit moved this task from Design Signoff to Product Signoff on the LPL Hypothesis board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits