@SHust - yes - that functionality was always intended for names that are in different scripts, as well as nick names - ie the last option refers to 'interchangeable' (assuming we don't want to prioritise one script or the other).

@SHust are you talking abou this functionality? ie any name pairs saved here will be picked up y the script in future

Just noting we have got a lot of functionality into the Core CiviCRM import framework now - some interventions we might need to do

I'm moving this to done cos I think we discussed it further & it is resolved. However, I note that next upgrade we will bring in the label improvement merged upstream https://github.com/civicrm/civicrm-core/pull/25700

@SHust maybe we can go over this on the screen at Civi fortnightly - I just merged the contacts you gave and

OK - it looks like Txn Amt, Posted Dt. and Doc Dt all need to be adjusted to have the second word capitalised

@MDemosWMF those imports work under the assumption the headers will be exactly the same each time - in this case

I just pulled this in as I wound up digging into it today.

@SHust I looked a little into this & the issue is largely non-technical. When we implemented the dedupe handling the advice we got from stakeholders was to only keep the latest donor's address. What you are wanting to do differs from that - so we need to figure out who decides that behaviour

I'm gonna close this cos I'm pretty sure it got fixed along the line- the main issue is the ticket is kinda confusing in terms of what is required but bulk updates can be done using search kit now

There are PRs against the library we use that might help

Just updating this to comment what things should be done.

Just noting a couple of merged PRs I'm gonna pull in / consider / may have already been pulled in

@SHust we do have these happen from time to time & it happens when server load is super high - in the meantime you can undelete the contact, re-add the email & re-merge them to fix the specific contact (let me know if you want to run through it on a call)

@SHust I put this in done - but if you want me to do the same thing for email matches when you have worked through that group open another phab

So the main difference is a change in the view link which we could alter with a hook - I put up an upstream PR to get activity_type_id added

@SHust I put the contacts in a group - try this

View link existing - we can remove that dumb text easily

Differences as a result of updating the type to 'Thank You email'

@SHust - I think most of the non-English matches did turn out to relate to bad data - especially old bad data - we don't collect much address data a lot of the time I guess.

As I suggested might be the case you are having trouble finding contacts with the Name and Address rule because there are not many to find.

OK - gonna re-create my temp tables with those rows gone....

OK - I was able to find the 'mother load' of bad addresses & it warranted an upgrade script to clean up a bit - then I can start looking for the real dupes again

268 contacts with a first name of '.' - gone
319 addresses with a city of . - fixed
235 addresses with a postal code of . - fixed
512 addresses with a street address of . - fixed

OK 1927 contacts with a last name of '.' - will remove those dots

OK running the above for BETWEEN 2000000 AND 3000000 found an insane number & seems to be a data issue.

INSERT INTO eileen_temp_dupe_pairs Select DISTINCT c1.id as id1, c2.id as id2, c1.first_name, c1.last_name, c1.street_address, c1.preferred_language FROM eileen_temp_dupe_contacts c1 INNER JOIN eileen_temp_dupe_contacts c2 ON c1.first_name = c2.first_name AND c1.last_name = c2.last_name AND c1.street_address = c2.street_address AND c1.id < c2.id AND c1.id BETWEEN 1000000 AND 2000000;
Query OK, 34417 rows affected (9.437 sec)

OK that create was way too slow - breaking it up

Am making temp tables with queries

@SHust I am looking in the DB & have put a couple of contacts that are dedupeable in this group

Just noting I've made progress on items under Most important

There is something merged upstream that should deal with this in future

I caught up with @mdemos today & we will catch up again next week. We identified some tasks to work on to get this usable

Upstream patch now merged

@MDemosWMF testing locally I was choosing 'Primary Email' from the list - but it should be 'Email' - I logged the confusing label upstream https://lab.civicrm.org/dev/core/-/issues/4096

@MDemosWMF when I tried again the things that didn't work started working - weird. Let's get a sample file & try again

I caught up with @MDemosWMF today & we are going to catch up next week.

@jgleeson - no they feel out during upgrade in favour of https://phabricator.wikimedia.org/T327225

I hit one issue during upgrade https://lab.civicrm.org/dev/core/-/issues/4095

@Ejegg I am including a patch from this in deploy - does that mean this is done now?

I'm dragging this back to triage as I've been looking at the activity indexes in the context of the upgrade & think we should possible remove

@Jgreen - it looks right - but once again that table is one of the location ones which are heavy on deletes - let's see how activity, activity_contact, contribution & contact look. If they are good we can try to figure out what is sane with these 2 tables - if not we might be on the wrong track

@MDemosWMF How about we find a time next week & just look at what the core CiviCRM import does on staging together & how we could use that more & custom code less as a first step