Hey @HNordeenWMF... So to determine if any technical work is needed for this task, and when it might be needed, it'd be great to get clarity on two questions:
1 - In which specific campaigns or tests that occurred over this period did users see a flow that included an opt-in questio (via banners, links to pages on Donate wiki)?
2 - How soon might we send e-mails to users who actually opted out but didn't get correctly flagged in Civi?

Thanks so much in advance! :)

Hi @AndyRussG ! Using your timeline of when it was live 2022-06-15 and 2022-06-30

1. Banners were live in

• India (May 31-June 28)- Opt-in not required as default but can still be seen
• South Africa (May 24-June 21)- Opt-in not required as default but can still be seen
• Latin America (May 31-June 28) - Opt-in requried for all countires
• Bundle #1 ES, AT, LU, BE (Pre-test June 15) - Opt-in requried for all countires

2. The good thing is, there aren't email campaigns following the Bundle pre-test or South Africa campaign. And India and Latin America emails went out before this issue started

• India emails concluded June 23
• South Africa - no fundrasing emails
• Latin America - emails concluded May 27
• Bundle #1 - no fundraising emails

In T311974#8062689, @HNordeenWMF wrote:

Opt-in not required as default but can still be seen

OK hmmm I guess that does complicate it a bit for those campaigns. So, just to clarify, this means donors had the chance to select to opt in or opt out, but could still reach a Payments form without answering the question, correct?
And for the countries where it was required, this means all donors must have explicitly answered either yes or no, right?
Thanks again!!

I'll do my best to clarify but @Pcoombe might need to jump in and help explain. (We're talking about how opt-in requirements work!)

Opt-in not required as default but can still be seen
Not quite, Opt-in wasn't shown at all for the majority of donors, but it would be shown if somehow someone's country required it. When we target countries that have no opt-in requirement set as default, we still see a few data points of people that "needed opt-in" during these campaigns, maybe for reasons of VPN this is where I'm not sure. Its a super small number, I just checked in one India test it was around <1% of people "needed opt-in". Those people would've explicitly said yes or no to an opt-in question during the India campaign.

Opt-in requried for all countires
Correct, all donors coming in from the countries where opt-in is required have to answer yes or no to get to payments. There's always a small discrepancy between # of people who "needed opt-in" and # of donations for the same reasons as above I think.

Luckily, the banner team has had to deal with our flakiness on this parameter before :P - since then they have been tacking on a bit with the optIn value to the utm_source to make sure it gets persisted somewhere.

-- How many chose 'no' to opt-in and made a donation
select count(*) from contribution_tracking
where ts BETWEEN '20220615000000' AND '20220630235999'
and utm_source like '%optIn0%'
and contribution_id IS NOT NULL;

16533

-- What values they got in their communication table
select opt_in, count(*)
from contribution_tracking ct
inner join civicrm.civicrm_contribution c on c.id=ct.contribution_id
left join civicrm.civicrm_value_1_communication_4 n on n.entity_id=c.contact_id
where ts BETWEEN '20220615000000' AND '20220630235999'
and utm_source like '%optIn0%'
group by opt_in;
+--------+----------+
| opt_in | count(*) |
+--------+----------+
|   NULL |    14378 |
|      0 |     1271 |
|      1 |      884 |
+--------+----------+

It's not quite as simple as running an update statement with that join setting opt_in=0 though - we need to insert rows for 14,368 of those donors. Just looking at a few of the rows, it seems like we can use defaults for all but opt_in. The opt_in_source/medium/campaign stuff is not generally set by the donation import so we can leave those null.

Change 834413 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Opt-out users missed due to GatewayChooser bug

https://gerrit.wikimedia.org/r/834413

Change 834413 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Opt-out users missed due to GatewayChooser bug

https://gerrit.wikimedia.org/r/834413

Change 836305 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Corrected version of opt-in fix second step

https://gerrit.wikimedia.org/r/836305

Change 836305 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Corrected version of opt-in fix second step

https://gerrit.wikimedia.org/r/836305