@LeanneS it seems there are a number of contacts in the engage file with first_name = Anonymous, last_name = Anonymous and no email. When that happens with Benevity we match them to the Anonymous contact AND we ignore any address data that may be associated with that contact. Are you happy for that handling for Engage too

@LeanneS a couple more questions - say we have an existing contact:

'first_name' => 'Minnie',
  'last_name' => 'Mouse',
  'contact_type' => 'Individual',
  'email' => 'minnie@example.com',
    'postal_code' => 98210,
    'city' => 'Mouseville',
    'street_address' => '35 Mousey Lane',
    'location_type_id' => 'Home',
));

Is it possible that our new contact might be a match but have LESS address info - ie. the new contact has no city & in that case should we blank out the city?

actually only one new question - other per above

@LeanneS I put a copy of new code on staging that 'does the easy matches' - ie. first name + last name + street address + city + postal code. or first name + last name + email (or equivalent for orgs).

Change 419639 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Add an output file showing which contacts resulted in new contacts being created

https://gerrit.wikimedia.org/r/419639

Ok - also on staging you can get a list of NOT matched contacts

I wanted to make it easier to see who wasn't matched so we could look for more opportunities

@Eileenmcnaughton I think it's fine to handle anonymous gifts similarly to the Benevity import.

Usually address information provided with checks is consistent for existing contacts (not always), but I'd say we should keep address information if we have it.

Thanks for that feature on non-matched contacts! I'll test this out.

@Eileenmcnaughton I just tried testing a couple of files on staging and both timed out. When I refreshed the page, the smaller organization file did show as processed, but most of the individual import failed.

Test files added here: /Fundraising/Tech/Major Gifts/Engage Import/March Import test file

I think I can see why it did that - what does 'failed' mean in this context -

I see a few error messages from my attempts: 'Error in line 5: (DB Error: already exists),' 'Unknown import error: Import aborted due to 10 consecutive errors, last error was at row 14: Could not delete entity id X,' 'Error in line 6: (Could not delete entity id X)'

Hmm error already exists is already imported - but ok - that delete entity id is weird

I fixed up an issue on staging (T190119) and re-tried the import & it seemed OK. That file is now imported so it would be good to test with a different one.

Note that I also changed the appearance of Civi on staging - thought I'd try out the shoreditch them. Am undecided personally

@Eileenmcnaughton Thanks! I tested a couple new files and they overall seemed to import. I'm having some trouble using Find Contribs to check how they imported as it is very slow/won't load.

I'm curious how the records are merging for those with many duplicate records. For example, we have some monthly check donors in which the most recent March record merged with their October record, not February or their first record created. I'm trying to send a few examples, but the Recent Items links lead to different records than the names indicated.

I also noticed that the communications preferences aren't being reflected, such as do not mail.

Change 421803 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Fix deduped contact imports to update privacy fields

https://gerrit.wikimedia.org/r/421803

@LeanneS I tested today on staging & got better results (civicrm/contact/view?reset=1&cid=25885282) - we were in a bit of a weird place last week with DB state so would be keen to test one more on staging before going live .

I have added another patch (https://gerrit.wikimedia.org/r/#/c/421803/) (check with me if it is deployed to staging before testing) to ensure the following fields are updated:
Do Not Email,Do Not Phone,Do Not SMS,Is Opt Out

(I left off 'Do Not Solicit' which is not in the files you sent me but added 'Do Not Mail' which is basically the same format as the above).

2 other questions from @Ejegg in review

1. if we have 2 contacts with an email match & one is the most recent donor but the address is not their primary and the other has an older donation but it is their primary which do we match to? (ditto address)
2. I am matching first_name='Anonymous' + last_name = 'Anonymous' + no email to the anonymous donor - is it possible that we might have contacts with no email & first_name = 'Anonymous' but an actual name for last name (if so, do we match to anonymous?)
3. Do we need to handle empty first & last names or will there always be either a name or 'Anonymous' - the file I'm looking at has no empty names

Answers

1. Go with most recent, make incoming the primary
2. If we have anonymous+anonymous +address do not discard address, create new contact.
3. conservative approach (skipping contacts with only first or last name OK for now)

Change 421803 abandoned by Eileen:
Fix deduped contact imports to update privacy fields

https://gerrit.wikimedia.org/r/421803

@LeanneS @NNichols - most recent version of this is now on staging - it has handling for

• privacy field updates
• only matches to anon if all of email, street_address & postal code are blank (I figured just city/ state wasn't enough to create a new contact)?
• has handling for trailing . in street address

Change 419639 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Add an output file showing which contacts resulted in new contacts being created

https://gerrit.wikimedia.org/r/419639

@LeanneS @NNichols OK to close?

@Eileenmcnaughton it's been working great for me! For the last three imports I went back and double checked the file created with new contacts and did not find any that should have been merged - they were also pretty small to begin with though.
@LeanneS Is there anything else you think we should be testing before this gets closed?

Thanks! I think we can proceed with closing and then continue to monitor the next few imports.