Page MenuHomePhabricator

Rec API tests failing intermittently
Closed, ResolvedPublic2 Estimated Story Points

Description

At least 2 tests in the recommendation/test/test_api.py suite have been failing intermittently and blocking merges. Almost every time for this patch: https://gerrit.wikimedia.org/r/c/research/recommendation-api/+/1143605

test_section_recommendations
    @pytest.mark.anyio
    async def test_section_recommendations(client: AsyncClient):
        response = await client.get(
            "/v1/translation/sections?source=en&target=es&seed=Apple&search_algorithm=morelike&count=12"
        )
        assert response.status_code == 200
        results = response.json()
        assert len(results) == 12
        assert results[0].get("source_title")
        assert results[0].get("target_title")
        assert results[0].get("source_sections")
        assert results[0].get("target_sections")
        assert results[0].get("present")
>       assert results[0].get("missing")
E       AssertionError: assert {}
E        +  where {} = <built-in method get of dict object at 0x104ec0940>('missing')
E        +    where <built-in method get of dict object at 0x104ec0940> = {'collection': None, 'missing': {}, 'present': {'External links': 'Enlaces externos', 'References': 'Referencias', 'Uses': 'Usos'}, 'source_sections': ['Uses', 'References', 'External links'], ...}.get

The absence of the missing property would suggest an article for which no section is missing in the target language. Therefore returning an invalid section recommendation.

test_recommendations_mostpopular
    @pytest.mark.anyio
    async def test_recommendations_mostpopular(client: AsyncClient):
        response = await client.get("/v1/translation?source=en&target=es&seed=Moon&search_algorithm=morelike")
        assert response.status_code == 200
        results = response.json()
        assert len(results) > 0
        assert results[0].get("title")
        assert results[0].get("pageviews") == 0
>       assert results[0].get("wikidata_id")
E       AssertionError: assert None
E        +  where None = <built-in method get of dict object at 0x7f5eee546900>('wikidata_id')
E        +    where <built-in method get of dict object at 0x7f5eee546900> = {'collection': None, 'langlinks_count': 0, 'pageviews': 0, 'rank': 302.0, ...}.get

Absence of wikidata_id? Is it a problem for the CX workflow?

Event Timeline

Change #1149719 had a related patch set uploaded (by Sbisson; author: Sbisson):

[research/recommendation-api@master] Fix: Filter out section recommendations with no missing sections

https://gerrit.wikimedia.org/r/1149719

SBisson triaged this task as Medium priority.
SBisson moved this task from Prioritized to Needs Code Review on the LPL Hypothesis board.

The patch above has a fix for test_section_recommendations and some extra logging for test_recommendations_mostpopular, which I can't get to fail today despite running it many many times.

Change #1149719 merged by jenkins-bot:

[research/recommendation-api@master] Fix: Filter out section recommendations with no missing sections

https://gerrit.wikimedia.org/r/1149719

Change #1150664 had a related patch set uploaded (by KartikMistry; author: KartikMistry):

[operations/deployment-charts@master] Update recommendation-api to 2025-05-26-081343-production

https://gerrit.wikimedia.org/r/1150664

Change #1150664 merged by jenkins-bot:

[operations/deployment-charts@master] Update recommendation-api to 2025-05-26-081343-production

https://gerrit.wikimedia.org/r/1150664

Mentioned in SAL (#wikimedia-operations) [2025-05-26T12:55:46Z] <kart_> Update Recommendation-API to 2025-05-26-081343-production (T394441, T395026, T306508, T391230)

Change #1150741 had a related patch set uploaded (by Sbisson; author: Sbisson):

[research/recommendation-api@master] test_section_recommendations: 'present' section is optional

https://gerrit.wikimedia.org/r/1150741

Change #1150741 merged by jenkins-bot:

[research/recommendation-api@master] test_section_recommendations: 'present' section is optional

https://gerrit.wikimedia.org/r/1150741

Recommendation-API deployment expected this week.

Nikerabbit changed the task status from Open to In Progress.Aug 12 2025, 7:19 AM

This is now deployed and those tests have been passing for a while.

Nikerabbit set the point value for this task to 2.Sep 4 2025, 10:56 AM
Nikerabbit moved this task from Pending deployment to Product Signoff on the LPL Hypothesis board.