Page MenuHomePhabricator

Creating New Matching Gift Import
Closed, ResolvedPublic4 Estimated Story Points

Description

This is the next import we would like to create for Engage (moving them from hand entering gifts to imports). These matching gift check donations are currently being hand entered into Civi by Engage so we are hoping to receive the data files from them and then be able to import straight into Civi.

This import still involves check gifts so it uses many of the fields that our Engage imports do, but also has some similarity to the Benevity import structure. These gifts will be similar to the Benevity import because the hard credit record is the Organization providing the match and the soft credit is the Individual.

Here are the static fields/structure:

-Contribution type = Engage
-Payment instrument = Check
-Restrictions = Restricted - Foundation
-Gift always goes on organization record and soft credits the individual
-Organization will be matched by name only and bounce back if no exact match is in Civi upon import (Similar to Benevity)
-All address/phone/email fields in the file will be associated with the soft credited Individual
-We would like an Employer relationship link to be created between the Org and soft credited Individual (if not already)

We need all fields currently used for the Engage check imports with the addition/removal of a few below.

Add:
-Soft Credit First Name
-Soft Credit Last Name
-Employer of - *See below under Questions
-Optional CID column - Since there can be many Individuals associated with the same Organization check, Engage has requested that in these cases it is easier to key in a CID than repeatedly write out the full Organization name in the file. This doesn’t occur for every check so it could be optional and only used if the Organization name field is null.

Remove:
-No Thank You
-Thank You Date
There are no TY/acknowledgements involved with these gifts

I have attached the Matching gift import file draft of what this might look like with new fields highlighted here:

Challenges/Questions
We would like your guidance and advice on the following:

-How to link the employer relationships in file?*
Would we use an ‘Employer of’ field in the file (included in the draft template) or can this be built in to automate that relationship link upon import? I see the Benevity import doesn't have any field like that since the file is from them and was wondering how they are linked.

-How to dedupe the soft credited individual during import?
We get limited information about the soft credited Individual, often only their name is provided. Would it be possible to match based on employer relationship if no email or address is present? For example, there is already a John Smith employed by Texas Instruments in Civi and would match if they have the same name and same employer.

Thank you for your help!! We would love to have this ready by end of Q4, but if not we are flexible to push it to Q1. Please reach out with any questions/concerns :)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Hi All - this is on hold at the moment due to the possible shift in lockbox service. We will follow up when we are a go on this again. Thanks!

@MDemosWMF @EMartin @Ppena @greg @NNichols

I was under the impression this was needed in the next few weeks or no later than the end of this fiscal quarter. The Engage contract ends in August. We should discuss this more if you really need it. We don't know how or when we'll change to another company.

@DStrine Right now we are taking some time to explore our options and figure out next steps. We were thinking a pause on this for the next week or so would avoid the possibility of duplicate work on your end as we decide what direction to move in.

*Update* - At this point we have decided it would be worth it to move forward with this import (time permitting). Since this had to be postponed last minute it may not fit within the timeline of priorities for fr-tech, so if we could have this completed by start of Q1 we can move forward. However, if this is not realistic we can continue to put this on hold until next year. Thank you for your flexibility after getting thrown this curveball! Let me know what you think.

^^Apologies I meant Q2 not Q1! Sprint 'L' this task is in works for us, but if July/August is needed that's ok on our end.

From office hours: Nov/Dec/Jan ideally.

@NNichols Nora, please note, this is being worked on by FRTEch and should be something you are planning for on the matching gifts side of things you are thinking about.

From mid-sprint checkin: This work is happening but mostly directly upstream right now and we're looking at after the next Civi upgrade (first wednesday of the month) to get this out.

We just sat down with Ellen for a walkthrough in her process of entering gifts and learned some new things! This unfortunately means a few changes from the original import outline, so I have attached a new potential template.

The main change is we found out we need to include an Individual hard credit as well as the Organization hard credit. So sometimes it is a gift going onto the Org record and soft crediting an Individual, and other times it is a gift being credited to the Individual and soft crediting the Organization.

There are two options I see here:

  1. We can add address/email fields to align with the hard credit Individual field, to better find a record match if they are already in Civi or create a new record if not. This seems like it could overburden the import with all of the other soft credit fields involved. (I defer to your expertise if this is fine or would cause issues)
  2. Or the second option is use a CID match for the Individual and not add any extra address/email fields for that hard credit. Since deduping won’t be needed it will match directly to that CID and an Individual CID will always need to be entered if it is an Individual hard credit gift. Ellen can look for them in Civi and use the CID if they are already there. If not, she can create the record and use that CID in the import.

Overall if a CID is present in the file, we always want to use that since it is the best match to the record. If it is not present, we can have it create a new record with the information provided if there is no match (for the soft credit fields). For the Organization hard credit field if there is no CID present, it will still bounce that Org name line if there is not an exact name match. Across the import if Ellen is using the CID as a match the name/address fields can be left blank since it will create a direct match and also cut down on data entry.

Ellen also mentioned it would be nice to have the option to use the CID for any soft credits whether Organization or Individual so I’ve added those fields to the template as well.

As a side note - there are EFT matching gifts (different from the check matching gifts on this import) that we believe could use the same import template once this is created. The only differences would be the payment method would be EFT instead of Check and the contribution type would be Cash instead of Engage. Just something I wanted to add in case there is anything you would need to do with the import as you build it now to allow that down the road.

As usual please let me know what is doable and what won’t work. I know this is pretty dense so please reach out with any questions! This is definitely the most complex import to date so I really appreciate your help tackling this. Thank you!!

@MDemosWMF How about we find a time next week & just look at what the core CiviCRM import does on staging together & how we could use that more & custom code less as a first step

@Eileenmcnaughton Sure! I can put something on the calendar

I caught up with @MDemosWMF today & we are going to catch up next week.

We hit a couple of things

  • either overly strict required field validation OR user error
  • DB error on staging search kit screen
  • expressed preference for Melanie that we prioritise being able to save the full job configuration

@MDemosWMF when I tried again the things that didn't work started working - weird. Let's get a sample file & try again

@MDemosWMF testing locally I was choosing 'Primary Email' from the list - but it should be 'Email' - I logged the confusing label upstream https://lab.civicrm.org/dev/core/-/issues/4096

image.png (524×1 px, 136 KB)

I caught up with @mdemos today & we will catch up again next week. We identified some tasks to work on to get this usable

Most important

  1. Fix the broken search kit actions
  2. intermittent search kit loading issue
  3. figure out why import test today failed
  4. To be able to have the soft credit dedupe rule include considering the existance of an employer relationship
  5. To ensure there IS an employer relationship between the contacts if a matching gift is imported to them

Desirable

  1. Be able to save the full import configuration as a template
  2. wording change for 'is primary' to make it less confusing https://lab.civicrm.org/dev/core/-/issues/4096

Needed for other imports

  1. To be able to handle currency conversion (this is down the track for other imports)
  2. Minor side issue - state province id field label should be state province imho
  3. Ideally be able to add defaults for fields not in the csv (down the track)

Change 885420 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Civi ports

https://gerrit.wikimedia.org/r/885420

Change 885420 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Civi ports

https://gerrit.wikimedia.org/r/885420

Just noting I've made progress on items under Most important

  1. should be fixed on staging now - this fix is also included in core
  2. progress on staging - it no longer kills search kit. I'm working to exclude expired searches from the listing per https://github.com/civicrm/civicrm-core/pull/25516
  3. in discussion with the community I have established there is demand for this beyond WMF

Change 899187 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] CiviCRM 5.61.alpha [not for merge]

https://gerrit.wikimedia.org/r/899187

Change 899188 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] [Not for merge] import config code

https://gerrit.wikimedia.org/r/899188

Change 901733 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] [do not merge] Upstream PR

https://gerrit.wikimedia.org/r/901733

Change 901735 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] [not for merge]

https://gerrit.wikimedia.org/r/901735

Change 902223 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Move Organization name resolver to extension

https://gerrit.wikimedia.org/r/902223

Change 902223 abandoned by Eileen:

[wikimedia/fundraising/crm@master] Move Organization name resolver to extension

Reason:

https://gerrit.wikimedia.org/r/902223

Change 902231 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Move anonymousContact retrieval to extension helper

https://gerrit.wikimedia.org/r/902231

Change 902232 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Simplify create contact

https://gerrit.wikimedia.org/r/902232

Change 899188 abandoned by Eileen:

[wikimedia/fundraising/crm@master] [Not for merge] import config code

Reason:

Merged upstream

https://gerrit.wikimedia.org/r/899188

Change 901733 abandoned by Eileen:

[wikimedia/fundraising/crm@master] [do not merge] Upstream PR

Reason:

Merged upstream

https://gerrit.wikimedia.org/r/901733

Change 901735 abandoned by Eileen:

[wikimedia/fundraising/crm@master] [not for merge] Fix import url

Reason:

Merged upstream

https://gerrit.wikimedia.org/r/901735

Change 902231 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Move anonymousContact retrieval to extension helper

https://gerrit.wikimedia.org/r/902231

Change 902232 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Simplify create contact

https://gerrit.wikimedia.org/r/902232

Change 904387 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] CiviCRM port

https://gerrit.wikimedia.org/r/904387

Change 904387 abandoned by Eileen:

[wikimedia/fundraising/crm@master] CiviCRM port

Reason:

https://gerrit.wikimedia.org/r/904387

@MDemosWMF As of yesterday I did an Individual import on staging that I think hits all the bases. I think we need to throw a few more tests at it.

Next steps

  1. more testing on staging with more data
  2. I'm working to document here
  3. when the CiviCRM point release goes out - hopefully later this week we can enable on live
  4. I want to check in with @JMando on what data needs Joseph has. The one that is top of mind for me is how important it is to add a record to ContributionTracking (which does currently happen with other imports but not with manual data entry).

@Eileenmcnaughton @MDemosWMF That depends on what would be lost if not creating a record in ContributionTracking. So long as I can tell where the gift comes from (country sourced from address), if it is endowment or not (financial_type_id=26 for example), gateway, and other relevant payment info then it should be fine for these gifts. I do not believe we use ContributionTracking for that in instances really outside of banner, email, and other online gifts where we need to tie back to a specific campaign.

@Eileenmcnaughton are there any gifts currently imported with this new process in the database? I could do a normal data pull with our usual queries and make sure they look right for reporting.

@JMando we only have them on staging so far (& not many good records as yet). After the next CiviCRM point release goes out (which I hope to do early next week) we can do some trials on prod

@MDemosWMF - should we be suppressing thank you emails for the Matching Gift import?

Change 899187 merged by Eileen:

[wikimedia/fundraising/crm@master] CiviCRM 5.61 rc

https://gerrit.wikimedia.org/r/899187

Change 908013 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Further import updates gateway_trxn_id & no_thank_you

https://gerrit.wikimedia.org/r/908013

Change 908015 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Add import handling for individual look up and soft credits.

https://gerrit.wikimedia.org/r/908015

Change 908050 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Add check for the contribution already existing

https://gerrit.wikimedia.org/r/908050

Change 908015 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add import handling for individual look up and soft credits.

https://gerrit.wikimedia.org/r/908015

Change 908013 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Further import updates gateway_trxn_id & no_thank_you

https://gerrit.wikimedia.org/r/908013

Change 908050 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add check for the contribution already existing

https://gerrit.wikimedia.org/r/908050

Change 910616 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Fix typo in Import Mapping Managed entity

https://gerrit.wikimedia.org/r/910616

Change 910617 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Avoid type error when organization_name is not set

https://gerrit.wikimedia.org/r/910617

Change 910616 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Fix typo in Import Mapping Managed entity

https://gerrit.wikimedia.org/r/910616

Change 910617 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Avoid type error when organization_name is not set

https://gerrit.wikimedia.org/r/910617