Page MenuHomePhabricator

Prepare queue messages for testing on upgraded Civi. Test them.
Closed, ResolvedPublic4 Story Points

Description

We should integration test our modules by wiring the CRM up to our other systems, or at least run end-to-end tests starting at queue consumption. Feed these messages through the staging server using "drush dpm" and friends.

  • Generate a bunch of test data in the form of serialized queue messages and put it somewhere in the repo.
  • drush -v -v unsubscribe-queue-consume
  • drush -v -v recurring-queue-consume
  • drush -v - v fredge-queue-consume
  • drush -v -v donations-process-messagefile
  • Test some broken messages that throw Civi errors and other exceptions.

Event Timeline

awight created this task.Oct 2 2015, 7:24 AM
awight raised the priority of this task from to Normal.
awight updated the task description. (Show Details)
awight renamed this task from Prepare queue messages for testing on upgraded Civi to Prepare queue messages for testing on upgraded Civi. Test them..Oct 7 2015, 6:23 PM
awight set Security to None.
awight edited a custom field.
Ejegg added a subscriber: Ejegg.Oct 21 2015, 12:10 AM

Fake data sets

Change 247765 had a related patch set uploaded (by Ejegg):
Queue consumer can import multiple messages from JSON

https://gerrit.wikimedia.org/r/247765

Ejegg added a comment.Oct 21 2015, 8:24 PM

I just tried importing 1000 fake messages locally (with contribution tracking entries set up).
On Civi 4.2, it took 2m 11s
On Civi 4.6, it took 4m 55s
The Civi 4.6 db was freshly migrated from a copy of the Civi 4.2 db, so they started with the same number of rows.

Didn't see any errors in the 4.6 import. Hooks to update the donor rollup tables worked fine, and imported contacts and contributions look good in the UI.

Ejegg added a comment.EditedOct 21 2015, 9:48 PM

Ran another batch of 15 in each version with the general query log on. Analyzed the last one inserted for each version, to allow caching of option ids.

Some observations: 4.6 adds an extra query 'SET NAMES utf8' before EVERYTHING, sometimes not even followed by another query. Can't be very costly individually, but it's a lot more round trips to the db.

Civi4.2: 1 delete, 18 inserts, 18 selects, 8 updates
Civi4.6: 1 delete, 29 inserts, 29 selects, 8 updates
(simple grepping, some selects might be nested)

Civi4.6 creates a contact activity record for the contribution, while 4.2 does not
4.6 also inserts 4 rows in civicrm_entity_financial_trxn, 2 into civicrm_financial_item, 2 into civicrm_financial_trxn, and 1 into civicrm_line_item

Did you test some rejects to ensure the exception logging works ok?

awight updated the task description. (Show Details)Oct 23 2015, 7:56 PM
awight updated the task description. (Show Details)
awight updated the task description. (Show Details)Oct 23 2015, 8:01 PM
awight updated the task description. (Show Details)Oct 25 2015, 9:00 AM
awight claimed this task.Oct 27 2015, 8:20 PM

Change 249334 had a related patch set uploaded (by Awight):
Add test message fixtures for the fredge queue

https://gerrit.wikimedia.org/r/249334

awight updated the task description. (Show Details)Oct 28 2015, 12:32 AM
Ejegg added a comment.Oct 28 2015, 9:06 PM

Benchmarking queue consumption on production: 854 donations imported in 90 seconds, before the upgrade. Peak donation rate last year was 3 per second, so even if the new version is half as fast we should be OK.

Change 249334 merged by jenkins-bot:
Add test message fixtures for the fredge queue

https://gerrit.wikimedia.org/r/249334

Eileenmcnaughton closed this task as Resolved.Nov 10 2015, 5:44 PM

Change 247765 merged by jenkins-bot:
Queue consumer can import multiple messages from JSON

https://gerrit.wikimedia.org/r/247765

mmodell removed a subscriber: awight.Jun 22 2017, 9:44 PM