Page MenuHomePhabricator

Recommendation-API CI testing is flaky due to frequent 429s from Wikidata Query Service
Closed, ResolvedPublic

Description

The recommendation API CI tests frequently fail due to a 429 response from WDQS in the one test that consumes it.

Example: https://integration.wikimedia.org/ci/job/service-pipeline-test/2270/console

Is CI really consuming WDQS to the point of regularly hitting the intended rate limiting threshold, or is this a WDQS bug? If not a bug, can some provision for CI be made in WDQS, or should this be addressed on the recommendation API side (perhaps by running the offending test on something like one run in ten)?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Mholloway renamed this task from CI testing is flaky due to frequent 429s from Wikidata Query Service to Recommendation-API CI testing is flaky due to frequent 429s from Wikidata Query Service.Jun 21 2019, 3:05 PM

Could you give some details on what recommendation API is actually doing there? Maybe post a link to the relevant code and probably explanation of which queries it is running and how often? Generally 429 means it is sending too many queries too often. Which shouldn't really happen on the test. But need some details.

Sure. The specific URL that the flaky test is hitting is https://de.wikipedia.org/api/rest_v1/data/recommendation/article/creation/translation/en.

Under the hood, the service is querying WDQS with this up to 10 times in rapid succession, for a batched set of ~500 Wikidata item IDs:

SELECT ?item (COUNT(?sitelink) as ?count) WHERE {
  VALUES ?item { <series of 50 item ids> }
  FILTER NOT EXISTS { ?item wdt:P31 wd:Q4167410 . }
  OPTIONAL { ?sitelink schema:about ?item }
  FILTER NOT EXISTS {
    ?article schema:about ?item .
    ?article schema:isPartOf <https://${target}.${projectDomain}/> .
  }
} GROUP BY ?item

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/recommendation-api/+/master/lib/article.creation.translation.js#158

Change 518349 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[mediawiki/services/recommendation-api@master] Temporarily disable flaky tests

https://gerrit.wikimedia.org/r/518349

Change 518349 merged by jenkins-bot:
[mediawiki/services/recommendation-api@master] Temporarily disable flaky tests

https://gerrit.wikimedia.org/r/518349