Page MenuHomePhabricator
Create Task
Maniphest T408934

Production error: AttributeError: 'NoneType' object has no attribute 'keys'
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

https://logstash.wikimedia.org/app/discover#/doc/0001c630-87fc-11f0-a6ef-d336524cfa1c/ecs-ml-1-1.11.0-7-2025.44?id=feOWOZoBlLUf3R62Iu77

This has been happening for about a month as a consequence of the recommendation api being unable to reach CX Server. This is documented in T405004: High volume of requests from rec-api to cxserver fail with 503 service unavailable

Screenshot 2025-10-31 at 1.30.12 PM.png (672×1 px, 56 KB)

Test Steps

Test Case 1: Verify Recommendation API Does Not Produce 'NoneType' AttributeError When CX Server Is Unreachable

  1. Trigger a recommendation API request that requires CX Server (e.g., a section or article recommendation request).
  2. Ensure CX Server returns a 503 response or is unavailable.
  3. Observe the API response.
  4. Review Logstash for associated logs tied to the request.
  5. ✅❓❌⬜ AC1: Confirm no unhandled exception (AttributeError: 'NoneType' object has no attribute 'keys') is generated.

QA Results - Logstash

ACStatusDetails
1✅T408934#11374869

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update Recommnedation API to 2025-11-07-162011-productionoperations/deployment-chartsmaster+2 -6
Update Recommnedation API to 2025-11-05-230545-productionoperations/deployment-chartsmaster+2 -6
Gracefully handle empty section sizesresearch/recommendation-apimaster+13 -2
Customize query in gerrit

Related Objects

Event Timeline

SBisson created this task.Oct 31 2025, 5:32 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 31 2025, 5:32 PM
gerritbot added a comment.Oct 31 2025, 5:42 PM

Change #1200393 had a related patch set uploaded (by Sbisson; author: Sbisson):

[research/recommendation-api@master] Gracefully handle empty section sizes

https://gerrit.wikimedia.org/r/1200393

gerritbot added a project: Patch-For-Review.Oct 31 2025, 5:42 PM
SBisson triaged this task as High priority.Oct 31 2025, 5:42 PM
SBisson moved this task from Backlog to Needs Code Review on the LPL Hypothesis board.
gerritbot added a comment.Oct 31 2025, 7:32 PM

Change #1200393 merged by jenkins-bot:

[research/recommendation-api@master] Gracefully handle empty section sizes

https://gerrit.wikimedia.org/r/1200393

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2025, 8:31 PM
SBisson moved this task from Needs Code Review to Pending deployment on the LPL Hypothesis board.Nov 3 2025, 3:16 PM
Nikerabbit added a project: LPL Projects (CX observability).Wed, Nov 5, 2:48 PM
gerritbot added a comment.Thu, Nov 6, 5:57 AM

Change #1202376 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-05-230545-production

https://gerrit.wikimedia.org/r/1202376

gerritbot added a project: Patch-For-Review.Thu, Nov 6, 5:57 AM
gerritbot added a comment.Thu, Nov 6, 1:31 PM

Change #1202376 merged by jenkins-bot:

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-05-230545-production

https://gerrit.wikimedia.org/r/1202376

Maintenance_bot removed a project: Patch-For-Review.Thu, Nov 6, 2:31 PM
gerritbot added a comment.Mon, Nov 10, 1:27 PM

Change #1203450 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-07-162011-production

https://gerrit.wikimedia.org/r/1203450

gerritbot added a project: Patch-For-Review.Mon, Nov 10, 1:27 PM
gerritbot added a comment.Mon, Nov 10, 1:37 PM

Change #1203450 merged by jenkins-bot:

[operations/deployment-charts@master] Update Recommnedation API to 2025-11-07-162011-production

https://gerrit.wikimedia.org/r/1203450

Stashbot mentioned this in T408936: Error when calling the wikidata API with no titles.Mon, Nov 10, 2:31 PM
Stashbot mentioned this in T408937: Faulty error handling when fetching language pairs.

Mentioned in SAL (#wikimedia-operations) [2025-11-10T14:31:12Z] <kart_> Update Recommnedation API to 2025-11-07-162011-production (T405000, T406854, T408936, T408937, T408934)

Stashbot mentioned this in T406854: Collection data unavailable in several rec-api hosts.Mon, Nov 10, 2:31 PM
Stashbot mentioned this in T405000: Handle failure to load languages from cx server.
Maintenance_bot removed a project: Patch-For-Review.Mon, Nov 10, 2:32 PM
SBisson moved this task from Pending deployment to Needs QA on the LPL Hypothesis board.Mon, Nov 10, 6:29 PM
GMikesell-WMF updated Other Assignee, added: GMikesell-WMF.Fri, Nov 14, 4:30 PM
GMikesell-WMF updated the task description. (Show Details)
GMikesell-WMF subscribed.Fri, Nov 14, 4:33 PM

@SBisson No errors in Logstash since it has been updated on 11/1, as seen in the screenshot. I will move this to Sign-off. Thanks for all your work!

**Test Result - Logstash*

Status: ✅ PASS
Environment: Logstash
OS: macOS Tahoe 26.1
Browser: Chrome 142
Device: MBA
Emulated Device: NA

Test Artifact(s):

Test Steps

Test Case 1: Verify Recommendation API Does Not Produce 'NoneType' AttributeError When CX Server Is Unreachable

  1. Trigger a recommendation API request that requires CX Server (e.g., a section or article recommendation request).
  2. Ensure CX Server returns a 503 response or is unavailable.
  3. Observe the API response.
  4. Review Logstash for associated logs tied to the request.
  5. ✅ AC1: Confirm no unhandled exception (AttributeError: 'NoneType' object has no attribute 'keys') is generated.

2025-11-12_11-28-50.png (914×1 px, 642 KB)

GMikesell-WMF updated Other Assignee, removed: GMikesell-WMF.Fri, Nov 14, 4:33 PM
GMikesell-WMF updated the task description. (Show Details)
GMikesell-WMF moved this task from Needs QA to Design Signoff on the LPL Hypothesis board.
SGautam_WMF subscribed.Mon, Nov 17, 8:17 AM

no design review needed for this task so moving it to product signoff.

SGautam_WMF moved this task from Design Signoff to Product Signoff on the LPL Hypothesis board.Mon, Nov 17, 8:18 AM
Nikerabbit closed this task as Resolved.Mon, Nov 17, 1:20 PM
Nikerabbit set the point value for this task to 2.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits