I'll also note that {Phttps://phabricator.wikimedia.org/T411896} causes the same issue. Something similar (missing data, but in this case a webform opt out if that was the method used) may have happened for cid 2450685.

Scoping out the required changes here:
Acoustic unsubscribes via form (as opposed to directly from the mailing) do not opt out the contact in CiviCRM. We create an Unsubscribe activity with subject "Unsubscribed via Acoustic Email link" in CRM_Omnimail_Omniactivity::getResult, but we don't opt them out. For the unsubscribes directly from the mailing, we process them in civicrm_api3_omnirecipient_process_unsubscribes and opt them out there, but since there is no mailing event for the kind of unsubscribes we are concerned with here, we cannot use this api action. So we need parity between these two pathways.

I've put the emails on hold for the contacts that were Invalid E-mail address (except the couple that had recent Mailings) and now disabled the field, to be deleted in the next maintenance window (after trigger updates have been done so we lose the insert for this field).

We are sending Faircom a complete list of US-primary-address contacts every month. Will need to ask them if they could just delete from their universe the contacts who are no longer in the list or do they actually need to do the deduping on their end as well. This also brings up the question of if they are removing contacts who no longer are included in the file Joseph sends because they have moved outside the US (probably pretty minor but would be good to handle this as well).

@MDemosWMF Yes, we'll aim for early 2026.

@SHust @AMJohnson @LWadleigh @MBeat33 The team has identified a lack of caching in CiviCRM was causing a large number of unnecessary queries when editing contact details (and quite likely in other areas as well). We've implemented a fix, so please let us know if the general slowness seems improved on your end. We hope this improves slow editing and loading, but I don't think this has anything to do with being unable to load pages at all.

@MDemosWMF Just want to make sure I understand this correctly:

We are still getting about 1/4 of inserted phones since Nov 27 without a source:

@MDemosWMF Could be the Mac CSV thing, you definitely don't want that (hasn't been a thing for decades).

@MDemosWMF I've split the incorrect url issue above off into T411438: Add correct links for generic imports to Import Templates SearchKit, so we can work on a general solution for all imports.
In the meantime, you can just go to https://civicrm.wikimedia.org/civicrm/import/option_value and select the template there instead of using the Import Template report.

@krobinson Since this takes ~10 minutes to run, we've decided to put it off until next year to avoid slowing anything down. The backfill is ready to go, so we'll just revive it then.

@MDemosWMF I have imported the packages in your file, but it looks like 30 of them already existed: https://civicrm.wikimedia.org/civicrm/search#/display/Import_3653/Import_3653
You might note there is one less rows than your original file, which is because I imported that one separately to test.

There's an additional problem with this one in that the link to use the import template looks like https://civicrm.wikimedia.org/civicrm/import/%?reset=1&template_id=3648 when it should be https://civicrm.wikimedia.org/civicrm/import/option_value?reset=1&template_id=3648 (i.e. we have % instead of option_value).

@MDemosWMF and if it is Excel, are you doing File -> Save As ... -> csv?

@MDemosWMF The November Package codes.csv file has the incorrect (old-style MacOS) line endings in it again, as we saw where you had that file that wouldn't import earlier. Which program are you using to create these files?

@SHust This has been fixed. Note that if you try to create an address with only country = US it will still not work (this is upstream to prevent addresses from being created by mistake on the create new contact form), but now if you edit an address to make it country = US with no other data, it will save correctly.

@SHust The goal is just to get rid of the Invalid E-mail address field entirely. It does not seem to be used except for these 28 contacts and it isn't taken into account for Acoustic exports, so we would still be trying to email these contacts. My thought is that we should just put emails on hold rather than using this separate field.

Merged upstream, but we can wait for January for this.

The additional request here is also to add something similar in Deduper (we'll just add these to the extension in general, down by the last contribution date).

The additional request here is also to add something similar in Deduper (we'll just add these to the extension in general, down by the last contribution date).

@SHust Thanks, I think we are seeing the same bug, which is more specifically that when an address has country = US, but no other data, it is deleted. If you try the same thing with a city included, you'll see the address isn't deleted. This is actually an upstream bug which is that if an address has no other data other than country and country = the default country, then the address is deleted.

@SHust I'm not sure I understand the bug here, as I've been unable to make this happen myself. Can you provide step by step instructions to replicate?

@LWadleigh Thanks for those times. Are they Pacific or UTC?

@Ejegg No worries. I'd test this but my email-pref-center has stopped working, so I'll leave this to someone else right now.

Duplicate T411042: Prevent any @lists.wikimedia.org email addresses from becoming Civi contacts (or at least opted out of all communication)

@MDemosWMF Can you prepare two CSVs for this, one each for Packages and Appeals, with three columns: Label, Value, Description. Label and Value would be the same.

@MRitch-WMF @JMando These extraneous supplemental addresses have been cleaned up.

Here's an upstream PR to make these importable.

What I've done is download the errors from Civi and uploaded to Sheets, then deleted the first four columns (which are about the import status and not needed).
Then I pasted the spreadsheet I sent you last week into another sheet in the same spreadsheet.
Then I added a LOOKUP formula to lookup the new cid in the other sheet (=LOOKUP(B2,Sheet1!A$2:B$351)), copy and pasted the resulting values into the same column (so I don't lose them when I delete the next column that the formula refers to). Then I deleted the original cids column, leaving only the corrected cids.
And here it is:
https://docs.google.com/spreadsheets/d/1iSXt1tUpJ_mcHNWMWAtgl-ubpjV5i4zvWo9ccY9STV4/edit?usp=sharing

@MDemosWMF I've also disabled the job that deletes trashed contacts after a year until we resolve this issue. So the problem won't get any worse, at least.

Thanks @SHust, maybe Civi fortnightly this Wednesday?

@SHust also do we need to think about this in Deduper as well? I'm guessing the same issue could happen there as well.

@MDemosWMF The question is more do we have a full list of the contact IDs we have sent to Faircom, so that we could check if any of those cids have been merged and send that information to them so they can merge them on their end.

Thanks @SHust, so is this specifically about making sure we are keeping the Paypal email as primary for Paypal recurring donors or would this be for all recurring donors? I would assume if we get a newer email address from a contact who has a non-Paypal recurring, we'd want to keep the newer email in that case.

@SHust That one looks like a case of the donor having two cids with two different email addresses and one with a US address based on a past donation (presumably via US IP, from their Paypal account or perhaps erroneously) and the other with a Canadian address. The US contact had country = US in Acoustic and Civi. I've now merged these two contacts into cid 61938254.

@SBurnett-WMF @SHust I think it would help to understand why DR needs to see if there is an active recurring when merging. In theory, it shouldn't matter which way you merge for recurring contributions. Some context on what we're trying to achieve (beyond just the merge screen) would help clarify the request.

@SHust or @SBurnett-WMF Could you clarify what the issue we are trying to improve here is? Naively, it shouldn't really matter which way we merge a pair of contacts as the final contact should have all the information. Is there some information that isn't being transferred correctly when we merge?

Thanks @krobinson, that's useful context.

@MDemosWMF Would you be able to just update the cids in the import errors file with a LOOKUP in Excel and then re-upload?

@krobinson The country is set based on the address we have in Civi for the donor with a fallback to a guess at their country based on past donations if they don't have an address with a country. So if their country is set in Civi, we don't check past donations at all (and if we do check past donations, I noticed that we aren't even selecting the most recent one, added T410924: Select most recent donation country when adding country to silverpop_missing_countries). This is documented here, but would it be better somewhere else?

1. We could look at this in the new year, but the difficulty is that we are adding a lot of complexity to these trigger-based queries. Maybe we need to put together a document of what the ideal flow would be in all these kinds of cases and then see what we can do.

@JMando or @MDemosWMF Do you know what kind of records we have of the list of cids we have sent to Faircom? If we wanted to send them a list of merges can we select those based on cid from data we have somewhere or would we need to get this from Faircom?

@MDemosWMF Sent you a slack with CSV. I'll say this one is done, but I still have the one from last month as a task to figure out how we want to handle sending all the merge updates to Faircom automatically.