Page MenuHomePhabricator

[Phragile] Investigate timeouts when connecting to very large projects
Closed, ResolvedPublic2 Estimated Story Points

Description

We need to investigate the reasons for timeouts when Phragile connects to very large projects like VisualEditor.

Event Timeline

Tobi_WMDE_SW raised the priority of this task from to High.
Tobi_WMDE_SW updated the task description. (Show Details)
Tobi_WMDE_SW moved this task from Incoming to v1.1 Backlog (Q1/2016) on the Phragile board.

The reason for the timeouts is that Phabricator's API does not seem to be able to handle the amount of data associated with the almost 6000 tasks in the VisualEditor project at once. The API call maniphest.query alone takes more than 15 seconds to return all the task data and the response contains over 12mb of JSON data.

To graph the project Phragile additionally needs the transactions of these 6000 tasks which results in even more data (~42mb). Currently Phragile tries to load all the transactions at once which fails after about a minute with an exception on Phabricator’s side. One solution is to fetch the transactions in batches. Instead of one API call loading all transactions there will be several calls loading e.g. transactions of 200 tasks at a time. It takes about 50 - 60 seconds on a labs instance to retrieve the raw data. Showing the sprint overview on Phragile's test instance for the VisualEditor project took about 70 seconds in total.

Tobi_WMDE_SW moved this task from v1.1 Backlog (Q1/2016) to Work in progress on the Phragile board.
Tobi_WMDE_SW moved this task from Work in progress to Done on the Phragile board.