Civi's added new tables, we're adding more API calls, let's not get surprised in December by queue consumers that can't keep up. Run Eileen's performance tests every couple weeks to catch slowdowns.
Description
Description
Related Objects
Related Objects
Event Timeline
Comment Actions
Run performance tests on staging for a baseline and to catch regressions before they're merged to deployment. Measure production performance and compare against baseline. Maybe pause the donations consumer for an hour before these tests in order to force a backlog of messages.
Comment Actions
We had an informal test last night when we turned the queue consumer off for over an hour. After we turned it back on, these were the total number imported for the next bunch of runs:
347
350
331
311
310
316
335
111 (reached end of queue in 33 sec)
These runs last 90 seconds and happen every two minues.
This was with two de-dupe jobs also running.
Thank You email jobs lagged behind:
368 in 2 min 56 sec
400 in 3 min 19 sec
400 in 3 min 23 sec
400 in 3 min 12 sec
400 in 3 min 16 sec
400 in 3 min 7 sec
328 in 2 min 44 sec
This was with CiviMail record creation turned on, sampled at 50%
Comment Actions
For what it is worth there is one unnecessary stand out query that runs very often and could be fixed through caching. It runs at least once every api call - not sure if there is a ticket on that - might be interesting to fix it & re-run https://phabricator.wikimedia.org/T148688
Comment Actions
We just had another informal test due to an artificially low paypal job runner limit. When that was lifted, the donations queue was topped up to about 8,000.
This test comes a day after deploying @Eileenmcnaughton's API performance fix: rWFCHb9fcc61d815a8ce
Donation queue import counts (90 second runs) between 2016-11-17 17:20:00 and 18:20:00
398
357
315
330
322
239
181
410
374
366
429
335
290
264
348
349
374
331
313
307
248
179
393
441
422
323
303
323
240
393
A lot more variable, with an average of 165 donations imported per minute
Comment Actions
Donation imports are good and fast these days, if a bit uneven. Average of 840 inserted on every 75 second run.
882
623
605
1357
610
577
1376
562
582
1388
558
635
1302
547
574
1352
554
598
1402
537
578
1295
581
549
1449
561
688
1333
556
1238
558
602
1135
533
1090
Comment Actions
Thank you letter send stats:
8/23: [no CiviMail records, 40 second runs] average ~245 per run
8/30: [100% CiviMail records, 40 second runs] average ~155 per run
8/30: [no CiviMail records, 40 second runs] average ~245 per run
After deploying https://gerrit.wikimedia.org/r/374915
8/31: [no CiviMail records, 40 second run] 455 sent