Page MenuHomePhabricator

Improve performance of Popular Pages bot
Closed, ResolvedPublic


Popular pages bot takes about a full month to process all the WikiProjects. The community has continually come back to us asking why it takes so long, and/or why it didn't create a report at all. The fundamental issue is that it's just too slow.

Currently the implementation loops through each page in a WikiProject, one by one, and only fetches pageviews for the target page + redirects. This usually amounts to maybe 5-10 pages. So we're only making a handful of asynchronous requests to the pageviews API when we are allowed to make up to 100.

Proposed solutions

  • Use the database to get the list of pages per WikiProject, and also the redirects for each page. We're currently doing this through the MediaWiki API which is much slower.
  • Queue up 100 or so pages and get pageviews all at once, rather then in small batches of just the target article + its redirects.

If done correctly, we should see a significant performance improvement. I estimate the runtime to be cut at least in half, but probably much more.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 10 2019, 9:55 PM
MusikAnimal changed the subtype of this task from "Bug Report" to "Task".Jul 10 2019, 9:56 PM
MusikAnimal updated the task description. (Show Details)

PR at

From my testing, a WikiProject with about 150,000 pages takes 20-30 minutes to process, when before it was 3-4 hours.

MusikAnimal added a subscriber: Niharika.EditedJul 16 2019, 5:01 AM

PR merged, thanks @Niharika!

For some reason the script will gradually get slower and slower on Toolforge. I gave the job 3 GB of RAM, which should be plenty sufficient. Everything runs very fast on my local, with no hiccups.

Out of curiosity I set up the bot on VPS; the instance is commtech-bot.commtech.eqiad.wmflabs, and the repo at /var/bot/popularpages. Indeed the bot is flying through the pages like it does on my local. So I'm going to keep it running there for now. It has done a few very large reports and there's no sign of slowing down, and memory usage is relatively low. I'll monitor and report back about performance.

aezell added a subscriber: aezell.Jul 16 2019, 5:45 PM

That might be interesting information to share with Cloud-Services.

Eventually the script starts to slow down on VPS too, but not nearly as much as it does on Toolforge. I think it's simply a matter of CPU power, managing arrays that can contain hundreds of thousands of items. Once its gets to the next WikiProject, and the array is cleared, it goes back to being really fast.

I'm convinced VPS is the best home for this bot. Running out of memory doesn't seem to be a problem, and as explained above, the script runs much faster overall. I've added documentation to (along with the two other instances belonging to our project).

MusikAnimal closed this task as Resolved.Jul 23 2019, 9:42 PM

The bot finished in record time, and everything seems correct as far as I can tell. I'm going to be bold and save you all time and just mark this as resolved.