Page MenuHomePhabricator

Data Axle project - List pull from Civi
Closed, ResolvedPublic

Description

We have confirmed the criteria for the list pull from Civi. There are a few different segments we would like to bring into one list. There may be overlap and we would need to figure out how to dedupe the list for those cases.

Universal criteria:

  • Individual contact type
  • Not opted out
  • US based with mailing address (don’t include those with invalid addresses)

ie

FROM civicrm_contact c INNER JOIN civicrm_address a ON a.id = c.id AND a.is_primary = 1
AND a.country_id = 1228 AND a.street_address <> '' 
WHERE contact_type = 'Individual' AND is_deceased = 0 AND is_deleted = 0
AND is_opt_out = 0

Segments:

  1. Gift of $50+ cumulatively in the last 5 years (Jan 1, 2018 - today)
  2. HNWI MG Audience (Donors who have a HNWI of $1M based on the TSmart data and have donated between $50 and $249 in the FY2122) - there is a group created by Major Gifts here. select * FROM civicrm_group_contact WHERE group_id = 1204 AND status = 'Added';
  3. Given $250+ ever (Data Axle 250+ single donation group ) select * FROM civicrm_group_contact WHERE group_id = 1678 AND status = 'Added';
  4. DAF donors of any threshold who gave in last 5 years (gift source = Donor Advised Fund) select * FROM civicrm_group_contact WHERE group_id = 1609 AND status = 'Added';
  5. QCD donors any threshold who gave in last 5 years (gift source = Retirement fund / retirement fund - endowment specified / retirement fund - annual fund specified) civicrm_value_1_gift_data_7.campaign IN (''Retirement Fund - Endowment Specified'', ''Retirement Fund')
  6. Given at least $20+ each year for the last 3 years

The format for the list requested by Data Axle is:

  • CSV or Excel file
  • 1 list file

Separate columns for:

  • First Name
  • Last name
  • Address (123 Main St)
  • City
  • State
  • Zip
  • Email
  • Phone(if available)
  • DOB

I know this can get complicated, so please follow up for any clarification needed. Thank you for your help pulling this list together and exporting from Civi! Any idea on how long this might take to create and pull would be appreciated so we can let Data Axle know.

We did T225446 and T254517 last time

Event Timeline

AnnWF updated the task description. (Show Details)

To see who is in a group

select count(*) FROM civicrm_group_contact WHERE group_id IN( 1678,1677) AND status = 'Added';

AnnWF updated the task description. (Show Details)

View final result from https://civicrm.wikimedia.org/civicrm/search#/display/Data_Axel_export_contacts
includes 6 segments:
1 and 6 created from upgrader.php
2 to 4 all manually created:
saved at searchkit and import all qualified contacts to groups:

Once the pr merged, will then provide the csv to data axle

Change 928649 had a related patch set uploaded (by Wfan; author: Wfan):

[wikimedia/fundraising/crm@master] Data Axle project - List pull from Civi

https://gerrit.wikimedia.org/r/928649

Change 928649 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Data Axle project - List pull from Civi

https://gerrit.wikimedia.org/r/928649

@AnnWF - I ran that update script

Thanks Eileen! The final version I have download the csv from https://civicrm.wikimedia.org/civicrm/search#/display/Data_Axel_export_contacts, and before upload to data axel, do anyone @MDemosWMF needs to review it?
the idea is have all 6 segments in a group, then use this searchkit https://civicrm.wikimedia.org/civicrm/admin/search#/edit/984 to get the final version.

Thanks @AnnWF and @Eileenmcnaughton! Just want to confirm we have all of the 6 segments I listed above included in the list. I see the searchkit links for segments 2-5, but not for 1 and 6. It seems like you may have pulled in those segments in a different way though, is that right?

A couple of questions as well before we upload this to the FTP:

  • Did we remove duplicates where there might be overlap between segments?
  • What is the total number of contacts in the final file? I will need to send them the file name and quantity of contacts.

Hi @MDemosWMF,

Eileen already run the upgrade on production so segment 1 and 6 also got contacts imported therefore all 6 segments are there
1: we did remove duplicates since we group by contact_id
2: the total number is 229256: https://civicrm.wikimedia.org/civicrm/search#/display/Data_Axel_export_contacts

Let me know if the final list looks ok to submit to data axel :P
Thanks

Thanks @AnnWF and @Eileenmcnaughton! Just want to confirm we have all of the 6 segments I listed above included in the list. I see the searchkit links for segments 2-5, but not for 1 and 6. It seems like you may have pulled in those segments in a different way though, is that right?

A couple of questions as well before we upload this to the FTP:

  • Did we remove duplicates where there might be overlap between segments?
  • What is the total number of contacts in the final file? I will need to send them the file name and quantity of contacts.

Great, that sounds good. Could we also include the CID as a field in the export? I would like to wait for @ERoden-WMF to sign-off as well before we move forward. Thanks!

Great, that sounds good. Could we also include the CID as a field in the export? I would like to wait for @ERoden-WMF to sign-off as well before we move forward. Thanks!

sure, I just add contact id to the list, https://civicrm.wikimedia.org/civicrm/search#/display/Data_Axel_export_contacts and the total number is now 229247, and I have the csv downloaded on hand.
once this query been sign-off, please let me know then I will submit the csv to data axle link directly. thanks @ERoden-WMF and @MDemosWMF

Hi @AnnWF! I just spoke with Erica and we have a few questions:

  • Could we get the the total of each list for segments 1 and 6? We also would like to make sure that these results include donors who make recurring gifts - we are guessing they do but want to be sure.
  • I see in the searchkits linked above the address is optional, but in the final list it looks like all the results have an address. I just want to confirm that all results in the list have a street address.

I see the DOB is written out instead of numeric format. I am double checking with Data Axle to see what format they need for that and will let you know.

Thanks!

Hi @MDemosWMF,

All 6 segments have it's own list , go to https://civicrm.wikimedia.org/civicrm/group?reset=1 and keyword: T336891, they are:
1: https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1685
2: https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1677
3: https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1678
4: https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1684
5: https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1682
6: https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1686
and yes all contacts have valid address, not optional
let me know what DOB format is needed.

Hi @AnnWF! I just spoke with Erica and we have a few questions:

  • Could we get the the total of each list for segments 1 and 6? We also would like to make sure that these results include donors who make recurring gifts - we are guessing they do but want to be sure.
  • I see in the searchkits linked above the address is optional, but in the final list it looks like all the results have an address. I just want to confirm that all results in the list have a street address.

I see the DOB is written out instead of numeric format. I am double checking with Data Axle to see what format they need for that and will let you know.

Thanks!

Thanks! I don't think I worded the criteria for segment 1 correctly. We would like to pull donors who have given $50+ each year for the last 5 years - not $50+ total for the last 5 years. Can you update that list?

I just heard back from Data Axle and they do not require any specific format for the DOB so we should be good.

Can you also confirm that recurring gifts are included in the lists where applicable? I think they are, but just want to be thorough in making sure the results are what we are looking for. Thank you!

Thanks! I don't think I worded the criteria for segment 1 correctly. We would like to pull donors who have given $50+ each year for the last 5 years - not $50+ total for the last 5 years. Can you update that list?

I just heard back from Data Axle and they do not require any specific format for the DOB so we should be good.

@MDemosWMF Sure, will update the segment 1 to 50+ each years, and for the recurring gifts, do you mean recurring donation? if so yes we include it, the new segment 1 is https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1703

Change 929796 had a related patch set uploaded (by Wfan; author: Wfan):

[wikimedia/fundraising/crm@master] Data Axle project - update segment 1

https://gerrit.wikimedia.org/r/929796

Change 929796 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Data Axle project - update segment 1

https://gerrit.wikimedia.org/r/929796

Change 930731 had a related patch set uploaded (by Wfan; author: Wfan):

[wikimedia/fundraising/crm@deployment] Data Axle project - update segment 1

https://gerrit.wikimedia.org/r/930731

Hi @AnnWF just wanted to check I haven't missed a message here (Melanie is out until next week). Is the updated list ready to view, or are you still working on it?

Hi @AnnWF just wanted to check I haven't missed a message here (Melanie is out until next week). Is the updated list ready to view, or are you still working on it?

Hi @ERoden-WMF,

The new segment code got merged, and I will do the production deployment after meeting and the new segment 1 is https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1703, you can review that this afternoon,
thanks!

Thanks @AnnWF, appreciate it. I'll take a look once it's ready and then we should be able to send over the Data Axle.

Change 930731 merged by jenkins-bot:

[wikimedia/fundraising/crm@deployment] Data Axle project - update segment 1

https://gerrit.wikimedia.org/r/930731

Thanks @AnnWF I've just reviewed and all looks good. We can send to Data Axle in one file. Please name the file "Wikimedia_Data_061623". I will let them know the file name now.

Hi @ERoden-WMF, files contain 113114 results, named Wikimedia_Data_061623.csv sent to the link Data Axle provided. Please let me know if this ticket is resolvable. Thanks

Thanks @AnnWF I've just reviewed and all looks good. We can send to Data Axle in one file. Please name the file "Wikimedia_Data_061623". I will let them know the file name now.

XenoRyet set Final Story Points to 4.