I agree that the load time for the burndown view is unacceptable.

The problem is directly related to LiskDAO and this query "SELECT x.objectPHID, x.oldValue, x.newValue, x.dateCreated FROM maniphest_transaction x WHERE transactionType = 'core:customfield' ORDER BY x.dateCreated ASC". This is a full table scan and is executed 3 times on the burndown view page load. The number of transactions in maniphest transaction is too large to be doing this kind of query.

The answer to this is directly relates to using Facts to store transaction data for Sprint. T78127 is the blocker here.

One other possibility is to add an index to the transactionType column in phabricator_maniphest.maniphest_transaction.

CREATE INDEX key_type ON phabricator_maniphest.maniphest_transaction (transactionType);

I may propose this as an upstream change, because transaction grouping is necessary for any kind of report about maniphest.

https://secure.phabricator.com/T6771 created. This could manually be done on production, though not as readily as if it were released as a patch and done with ./storage upgrade

See patches here:

https://gerrit.wikimedia.org/r/#/c/180564/
https://gerrit.wikimedia.org/r/#/c/180673/

I am quite sure that the performance will improve with 0.6.1.8. All of the table scans have been removed. The transaction type lookup on maniphest transactions is only necessary for the SprintFactsDaemon aggregator now. Upstream may still implement the index, but I have no idea when.

@Christopher, you should find a way to inject your own personal performance to the Sprint app. You're fast, man!

Change 180774 had a related patch set uploaded (by Christopher Johnson (WMDE)):
DataView Performance enhancements

https://gerrit.wikimedia.org/r/180774

Patch-For-Review

Change 180774 merged by Christopher Johnson (WMDE):
DataView Performance enhancements

https://gerrit.wikimedia.org/r/180774

I created "Collaboration-Team-Sprint-2014-12-17", here's its Burndown view. Its Sprint board was very slow to load, and after adding about 20 tasks to the project it gets timeouts:

Request aborted by debug time limit after 30 seconds.

STACK TRACE
api.php:32 PhabricatorStartup->onDebugTick()
SprintBoardViewController.php:274 celerity_generate_unique_node_id()
AphrontController.php:33 SprintBoardViewController->processRequest()
index.php:103 AphrontController->handleRequest()

That sounds like an extreme example of this bug so I'm not filing a separate task. Some of the other boards at https://phabricator.wikimedia.org/sprint/ aren't quite so bad, so perhaps the Collaboration-Team is doing it wrong. E.g. none of our tasks have story points yet.

Good luck fixing the bug, thanks.

Are these fixes deployed? The Sprint-extension managed pages are still extremely slow or time-out altogether.

We are still ironing the process for deploying new versions of the Sprint extension, and right now I'm not sure whether this task should be blocked by T78243: Phabricator upgrade on 2015-01-14. CCing @chasemp.

It seems better, but §Collaboration-Team-Sprint-2014-12-17 is still very slow or I get a timeout. Thelatest timeout isn't in celerity_generate_unique_node_id(), now:

Request aborted by debug time limit after 30 seconds.

STACK TRACE
  AphrontBaseMySQLDatabaseConnection.php:146 PhabricatorStartup->onDebugTick()
queryfx.php:17 AphrontBaseMySQLDatabaseConnection->selectAllResults()
SprintQuery.php:137 queryfx_all()
SprintQuery.php:73 SprintQuery->getXactionData()
SprintBoardTaskCard.php:65 SprintQuery->getPointsTransactions()
SprintBoardViewController.php:282 SprintBoardTaskCard->getItem()
AphrontController.php:33 SprintBoardViewController->processRequest()
index.php:103 AphrontController->handleRequest()

(The Collaboration team may remove the '§' from the name to make the board usable.)

This is definitely coming up as a pretty big challenge for Fundraising... is there any news on when the fix might go out? Thanks!

In T78679#964722, @atgo wrote:

is there any news on when the fix might go out?

That's T78243

Thanks @Aklapper

Also, this happened today:

Request aborted by debug time limit after 30 seconds.

STACK TRACE
AphrontBaseMySQLDatabaseConnection.php:168 PhabricatorStartup->onDebugTick()
queryfx.php:6 AphrontBaseMySQLDatabaseConnection->executeRawQuery()
-:- queryfx()
queryfx.php:16 call_user_func_array()
SprintQuery.php:137 queryfx_all()
SprintQuery.php:73 SprintQuery->getXactionData()
SprintBoardTaskCard.php:65 SprintQuery->getPointsTransactions()
SprintBoardViewController.php:282 SprintBoardTaskCard->getItem()
AphrontController.php:33 SprintBoardViewController->processRequest()
index.php:103 AphrontController->handleRequest()

is this fixed in Sprint 0.6.2.7 ?

@chasemp

yes, 0.6.2.7 contains the fixes. You could run a query plan analysis from
dark console on production and compare it to phab08.eqiad.wmflabs to
confirm.

Thanks @Christopher. side note, are you on irc at all?

Ran into an error upgrading to the tag release/2015-01-08/1

I think this is saying "We run through and look for a DB for every application and this application has none....oh no!" Which I believe is ok as we aren't using a dedicated DB, but if you could let me know I won't upgrade tomorrow without confirmation this is ignorable (that is if I can get to that point at all).

FYI, https://phab-01.wmflabs.org/ should have upgraded: phab/arc/libphutil/sprint app/ security extension and I am trying to run through things now.

In T78679#974033, @chasemp wrote:

Thanks @Christopher. side note, are you on irc at all?

Ran into an error upgrading to the tag release/2015-01-08/1

P212 sprint app error upgrading schema

1	root@phab-01:/var/log# /srv/phab/phabricator/bin/storage upgrade
2	Before running storage upgrades, you should take down the Phabricator web
3	interface and stop any running Phabricator daemons (you can disable this
4	warning with --force).
5
6	Are you ready to continue? [y/N] y
7
8	Applying patch 'phabricator:20141210.maniphestsubscribersmig.2.sql'...
9	Applying patch 'phabricator:20141210.reposervice.sql'...
10	Applying patch 'phabricator:20141212.conduittoken.sql'...
11	Applying patch 'phabricator:20141215.almanacservicetype.sql'...
12	Applying patch 'phabricator:20141217.almanacdevicelock.sql'...
13	Applying patch 'phabricator:20141217.almanaclock.sql'...
14	Applying patch 'phabricator:20141218.maniphestcctxn.php'...
15	Converting Maniphest CC transactions to modern SUBSCRIBER transactions...
16	Done.
17	Applying patch 'phabricator:20141222.maniphestprojtxn.php'...
18	Converting Maniphest project transactions to modern EDGE transactions...
19	Done.
20	Applying patch 'phabricator:20141223.daemonloguser.sql'...
21	Applying patch 'phabricator:20141223.daemonobjectphid.sql'...
22	Applying patch 'phabricator:20141230.pasteeditpolicycolumn.sql'...
23	Applying patch 'phabricator:20141230.pasteeditpolicyexisting.sql'...
24	Applying patch 'phabricator:20150102.policyname.php'...
25	Migrating policies...
26	Applying patch 'phabricator:20150102.tasksubscriber.sql'...
27	Applying patch 'phabricator:20150105.conpsearch.sql'...
28	Storage is up to date. Use 'storage status' for details.
29	Verifying database schemata...
30
31
32	Database Table Name Issues
33	phabricator_worker worker_activetask key_object Missing Key
34	phabricator_worker worker_archivetask key_object Missing Key
35
36	Found 2 issues(s) with schemata, detailed above.
37
38	You can review issues in more detail from the web interface, in Config > Database Status. To better understand the adjustment workflow, see "Managing Storage Adjustments" in the documentation.
39
40	MySQL needs to copy table data to make some adjustments, so these migrations may take some time.
41
42
43	Fix these schema issues? [y/N] y
44
45	Dropping caches, for faster migrations...
46	Purging remarkup cache...done.
47	Purging changeset cache...done.
48	Purging general cache...done.
49	Fixing schema issues...
50	Done.
51	Completed fixing all schema issues.
52
53	Target Error
54	phabricator_sprint Missing
55
56	SCHEMATA ERRORS
57
58	The schemata have errors (detailed above) which the adjustment workflow can
59	not fix.
60
61	If you are not developing Phabricator itself, report this issue to the
62	upstream.
63
64	If you are developing Phabricator, these errors usually indicate that your
65	schema specifications do not agree with the schemata your code actually
66	builds.
67	root@phab-01:/var/log#

I think this is saying "We run through and look for a DB for every application and this application has none....oh no!" Which I believe is ok as we aren't using a dedicated DB, but if you could let me know I won't upgrade tomorrow without confirmation this is ignorable (that is if I can get to that point at all).

FYI, https://phab-01.wmflabs.org/ should have upgraded: phab/arc/libphutil/sprint app/ security extension and I am trying to run through things now.

We ran through a bunch of sprint extension functionality and logic and I believe this is indeed benign. It would still be good @Christopher if you could confirm.

In T78679#973628, @Christopher wrote:

@chasemp

yes, 0.6.2.7 contains the fixes. You could run a query plan analysis from
dark console on production and compare it to phab08.eqiad.wmflabs to
confirm.

double checking, this is deployed with tag release/2015-01-08/1 right?

@chasemp RE: the missing schema error. This is ignorable for now. I will try to bypass the workflow hook that checks classes for schema if they implement LiskDAO. Created new task here T86773.

In T78679#976110, @Christopher wrote:

@chasemp RE: the missing schema error. This is ignorable for now. I will try to bypass the workflow hook that checks classes for schema if they implement LiskDAO. Created new task here T86773.

Understood and thanks