Page MenuHomePhabricator

[Investigation] Evaluate improving database by checking in bulk if articles have suggestions
Open, Needs TriagePublic

Description

When creating the original approach to iterate through all articles in a wiki to find new suggestions with a maintenance script, @Urbanecm_WMF made the good point that our current database queries might be somewhat inefficient by checking for every page individually whether they have suggestions in our GrowthExperiments database table growthexperiments_link_recommendations.

As we are now rolling out the new way of refreshing those Link Recommendations to more wikis, we should investigate if batching these queries for ~500 pages at a time would be better than doing them individually. This is about the database query coming from LinkRecommendationStore::getRecommendationStateByRevision which goes to the primary database.

Acceptance criteria:

  • Investigate if there is benefit from batching these queries for by 500 at a time when checking for whether they are known in the growthexperiments_link_recommendations table
  • Investigate whether with the new way of doing things (iterating through all the articles in a wiki), we actually still have to query the primary database
  • if we can improve things here, outline a plan of how to implement those improvements and estimate the effort required

Event Timeline

What is the percentage of articles that have suggestions? If it's around 1 to 10%. Maybe you could use a bloom filter instead?