Page MenuHomePhabricator

Performance improvement: Fetch pageviews asynchronously
Closed, ResolvedPublic

Description

During the update process, one of the slowest steps is fetching pageviews (depending on how many pages are being processed). This is currently done synchronously. The API allows us to make up to 100 requests at the same time. We should take advantage of this. The performance improvement should be in the orders of magnitude.

Event Timeline

Restricted Application added a project: Community-Tech. · View Herald TranscriptMar 8 2019, 5:55 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

This is certainly a "nice to have" and nothing urgent. I'm only logging it here so we don't forget about it.

MusikAnimal moved this task from Ready to In Development on the Community-Tech-Sprint board.
MusikAnimal updated the task description. (Show Details)Apr 8 2019, 5:23 PM

PR: https://github.com/wikimedia/eventmetrics/pull/288

To give an example, fetching pageviews for https://eventmetrics-dev.wmflabs.org/programs/76/events/370 takes around 2.5 minutes. With the async implementation it takes around 4.8 seconds.

MusikAnimal closed this task as Resolved.Apr 18 2019, 1:20 AM

Merged and deployed