User Details
- User Since
- Sep 15 2015, 12:37 AM (392 w, 21 h)
- Availability
- Available
- LDAP User
- Eileen
- MediaWiki User
- Eileenmcnaughton [ Global Accounts ]
Today
@SHust - yes - that functionality was always intended for names that are in different scripts, as well as nick names - ie the last option refers to 'interchangeable' (assuming we don't want to prioritise one script or the other).
@SHust are you talking abou this functionality? ie any name pairs saved here will be picked up y the script in future
Sat, Mar 18
Just noting we have got a lot of functionality into the Core CiviCRM import framework now - some interventions we might need to do
Fri, Mar 17
Thu, Mar 16
Tue, Mar 14
Mon, Mar 13
I'm moving this to done cos I think we discussed it further & it is resolved. However, I note that next upgrade we will bring in the label improvement merged upstream https://github.com/civicrm/civicrm-core/pull/25700
Wed, Mar 8
@SHust maybe we can go over this on the screen at Civi fortnightly - I just merged the contacts you gave and
Mon, Mar 6
OK - it looks like Txn Amt, Posted Dt. and Doc Dt all need to be adjusted to have the second word capitalised
@MDemosWMF those imports work under the assumption the headers will be exactly the same each time - in this case
Wed, Mar 1
I just pulled this in as I wound up digging into it today.
Tue, Feb 28
@SHust I looked a little into this & the issue is largely non-technical. When we implemented the dedupe handling the advice we got from stakeholders was to only keep the latest donor's address. What you are wanting to do differs from that - so we need to figure out who decides that behaviour
Thu, Feb 23
I'm gonna close this cos I'm pretty sure it got fixed along the line- the main issue is the ticket is kinda confusing in terms of what is required but bulk updates can be done using search kit now
Wed, Feb 22
There are PRs against the library we use that might help
Tue, Feb 21
Feb 16 2023
Feb 14 2023
Just updating this to comment what things should be done.
Just noting a couple of merged PRs I'm gonna pull in / consider / may have already been pulled in
@SHust we do have these happen from time to time & it happens when server load is super high - in the meantime you can undelete the contact, re-add the email & re-merge them to fix the specific contact (let me know if you want to run through it on a call)
@SHust I put this in done - but if you want me to do the same thing for email matches when you have worked through that group open another phab
So the main difference is a change in the view link which we could alter with a hook - I put up an upstream PR to get activity_type_id added
@SHust I put the contacts in a group - try this
View link existing - we can remove that dumb text easily
Differences as a result of updating the type to 'Thank You email'
Feb 9 2023
@SHust - I think most of the non-English matches did turn out to relate to bad data - especially old bad data - we don't collect much address data a lot of the time I guess.
As I suggested might be the case you are having trouble finding contacts with the Name and Address rule because there are not many to find.
OK - gonna re-create my temp tables with those rows gone....
OK - I was able to find the 'mother load' of bad addresses & it warranted an upgrade script to clean up a bit - then I can start looking for the real dupes again
String | field | number | notes |
--- | City | 35 | |
--- | Postal code | 62 | |
--- | Street Address | 79 | |
--- | First Name | 11 | |
--- | Last Name | 19 | |
... | City | 37 | |
... | Postal code | 24 | |
... | Street Address | 77 | |
... | First Name | 22 | |
... | Last Name | 57 | |
1 | Postal code | 902 | Selectively fixed for obviously bad - is this ever legit |
268 contacts with a first name of '.' - gone
319 addresses with a city of . - fixed
235 addresses with a postal code of . - fixed
512 addresses with a street address of . - fixed
Feb 8 2023
OK 1927 contacts with a last name of '.' - will remove those dots
OK running the above for BETWEEN 2000000 AND 3000000 found an insane number & seems to be a data issue.
INSERT INTO eileen_temp_dupe_pairs Select DISTINCT c1.id as id1, c2.id as id2, c1.first_name, c1.last_name, c1.street_address, c1.preferred_language FROM eileen_temp_dupe_contacts c1 INNER JOIN eileen_temp_dupe_contacts c2 ON c1.first_name = c2.first_name AND c1.last_name = c2.last_name AND c1.street_address = c2.street_address AND c1.id < c2.id AND c1.id BETWEEN 1000000 AND 2000000;
Query OK, 34417 rows affected (9.437 sec)
OK that create was way too slow - breaking it up
Am making temp tables with queries
@SHust I am looking in the DB & have put a couple of contacts that are dedupeable in this group
Feb 7 2023
Just noting I've made progress on items under Most important
Jan 31 2023
There is something merged upstream that should deal with this in future
I caught up with @mdemos today & we will catch up again next week. We identified some tasks to work on to get this usable
Upstream patch now merged
Jan 30 2023
Jan 26 2023
@MDemosWMF testing locally I was choosing 'Primary Email' from the list - but it should be 'Email' - I logged the confusing label upstream https://lab.civicrm.org/dev/core/-/issues/4096
@MDemosWMF when I tried again the things that didn't work started working - weird. Let's get a sample file & try again
I caught up with @MDemosWMF today & we are going to catch up next week.
Jan 25 2023
@jgleeson - no they feel out during upgrade in favour of https://phabricator.wikimedia.org/T327225
I hit one issue during upgrade https://lab.civicrm.org/dev/core/-/issues/4095
@Ejegg I am including a patch from this in deploy - does that mean this is done now?
Jan 24 2023
Jan 23 2023
Jan 19 2023
I'm dragging this back to triage as I've been looking at the activity indexes in the context of the upgrade & think we should possible remove
@Jgreen - it looks right - but once again that table is one of the location ones which are heavy on deletes - let's see how activity, activity_contact, contribution & contact look. If they are good we can try to figure out what is sane with these 2 tables - if not we might be on the wrong track
Jan 18 2023
@MDemosWMF How about we find a time next week & just look at what the core CiviCRM import does on staging together & how we could use that more & custom code less as a first step