Page MenuHomePhabricator

Prepare queue messages for testing on upgraded Civi. Test them.
Closed, ResolvedPublic4 Estimated Story Points

Description

We should integration test our modules by wiring the CRM up to our other systems, or at least run end-to-end tests starting at queue consumption. Feed these messages through the staging server using "drush dpm" and friends.

  • Generate a bunch of test data in the form of serialized queue messages and put it somewhere in the repo.
  • drush -v -v unsubscribe-queue-consume
  • drush -v -v recurring-queue-consume
  • drush -v - v fredge-queue-consume
  • drush -v -v donations-process-messagefile
  • Test some broken messages that throw Civi errors and other exceptions.

Event Timeline

awight raised the priority of this task from to Medium.
awight updated the task description. (Show Details)
awight renamed this task from Prepare queue messages for testing on upgraded Civi to Prepare queue messages for testing on upgraded Civi. Test them..Oct 7 2015, 6:23 PM
awight set Security to None.
awight edited a custom field.

Fake data sets

Change 247765 had a related patch set uploaded (by Ejegg):
Queue consumer can import multiple messages from JSON

https://gerrit.wikimedia.org/r/247765

I just tried importing 1000 fake messages locally (with contribution tracking entries set up).
On Civi 4.2, it took 2m 11s
On Civi 4.6, it took 4m 55s
The Civi 4.6 db was freshly migrated from a copy of the Civi 4.2 db, so they started with the same number of rows.

Didn't see any errors in the 4.6 import. Hooks to update the donor rollup tables worked fine, and imported contacts and contributions look good in the UI.

Ran another batch of 15 in each version with the general query log on. Analyzed the last one inserted for each version, to allow caching of option ids.

Some observations: 4.6 adds an extra query 'SET NAMES utf8' before EVERYTHING, sometimes not even followed by another query. Can't be very costly individually, but it's a lot more round trips to the db.

Civi4.2: 1 delete, 18 inserts, 18 selects, 8 updates
Civi4.6: 1 delete, 29 inserts, 29 selects, 8 updates
(simple grepping, some selects might be nested)

Civi4.6 creates a contact activity record for the contribution, while 4.2 does not
4.6 also inserts 4 rows in civicrm_entity_financial_trxn, 2 into civicrm_financial_item, 2 into civicrm_financial_trxn, and 1 into civicrm_line_item

Did you test some rejects to ensure the exception logging works ok?

awight updated the task description. (Show Details)

Change 249334 had a related patch set uploaded (by Awight):
Add test message fixtures for the fredge queue

https://gerrit.wikimedia.org/r/249334

Benchmarking queue consumption on production: 854 donations imported in 90 seconds, before the upgrade. Peak donation rate last year was 3 per second, so even if the new version is half as fast we should be OK.

Change 249334 merged by jenkins-bot:
Add test message fixtures for the fredge queue

https://gerrit.wikimedia.org/r/249334

Change 247765 merged by jenkins-bot:
Queue consumer can import multiple messages from JSON

https://gerrit.wikimedia.org/r/247765