Page MenuHomePhabricator

For external traffic release, perform lookup in GrowthExperiments cache before generating recommendations
Closed, DeclinedPublic

Description

As we've discussed in T269581: Add Link engineering: Allow external traffic to linkrecommendation service, T277297: 504 timeout and 503 errors when accessing linkrecommendation service and related patches, generating link recommendations is slow. This is not as big of a problem for the internal consumer of the service (refreshLinkRecommendations.php) but it's not great for the external traffic release. We've hacked around that some by lengthening the timeout for the service (potentially problematic) and returning all possible results within a timeout period.

Another idea would be to query the GrowthExperiments cache for the relevant article link recommendations before (re)generating link recommendations for the article.

The workflow would be:

For larger wikis, there would be mostly cache misses but for smaller wikis there'd be a higher cache hit-to-miss ratio (our initial task pool size will be 25,000 articles per wiki).

Event Timeline

We could just look at what recommendations are cached in the production wiki instead of picking random articles.

We could just look at what recommendations are cached in the production wiki instead of picking random articles.

Could you elaborate please? I'm not sure that I follow.

kostajh renamed this task from Perform lookup in GrowthExperiments cache before generating recommendations to For external traffic release, perform lookup in GrowthExperiments cache before generating recommendations.Mar 29 2021, 8:36 PM

I mean, instead of picking random pages we could pull pages from the upstream search index and see if they exist locally. Not sure if it's worth the complexity, but that's true for this task in general.

I mean, instead of picking random pages we could pull pages from the upstream search index and see if they exist locally. Not sure if it's worth the complexity, but that's true for this task in general.

Yeah, agreed. Declining this.