I'm thinking here to use an extension for this & to convert our 'pre-checks' (ie. our throttle logic) to a hook. Undecided as yet whether just to track through settings like in other places or to investigate Civi's batch mechanism - will have a discussion with Tim

There is a problem with the csv file - we have a contact with the name ríg in their name- (505553) but in the target smart file it is
r<CC><C4><E5>_g

I've also found a few cases where the address has '<' char that are mangled - eg 774744 came back as <8A><FC>_<BE><B5><E1>

Auditing for char < finds

831645 - no additional info
290489 - unrelated - actual html but no additional info
7486779 spam contact
8400248 pre-existing La<Mesa city

OK - that's if for zip 1 which is 1/9 of all of them - I think we can just ignore any field with '<' & move on

@LeanneS just wondering - why do we update names from target smart data?

@LeanneS there are now 181 records imported into this group - civicrm/group/search?force=1&context=smog&gid=669

• I'm not quite sure why the number added is slightly less than the number of rows I processed but I think there are actually duplicate rows in the import doc

Can you check they look ok?

The first file is now importing (first of 9 with about 500k contacts in each)

@LeanneS just looking - we are importing vf_party
but not party_score_rollup - we don't have 2 fields in civi & I thought they might be kinda the same but the party_score_rollup DOES seem to have extra info