@SHust, we're looking at this now.

Change 965537 had a related patch set uploaded (by Jgleeson; author: Jgleeson):

[mediawiki/extensions/DonationInterface@master] WIP: Handle Paypal not sending over donor email in GetExpressCheckoutDetails call

https://gerrit.wikimedia.org/r/965537

@SHust, thanks for flagging this. It looks like PayPal is not sending us donor emails for a small number of donations. We're not sure what is different about the affected donations, but we've got a fix in mind, so we're gonna get that working, backfill the older donations missing an email address and reach out to PayPal to find out what they're up to!

@jgleeson Thanks for the update! Does this mean they will automatically receive the 'ty receipt' after the emails reach Civi?

@jgleeson this is messier than we thought. I actually only see 25 rows for people where the EMAIL is missing on the Get..Details response, and those don't include a bunch of the CIDs listed above.

And I realized that PayPal recurring donors actually get into Civi via the listener, not via the front end. The front end sets up the new subscription with PayPal, sends a message to the pending queue, and then waits for paypal to charge the donor and send us two messages: 1 recurring_payment_profile_created and 1 recurring_payment.

In the case of CID 60832113 (contribution tracking ID 192710606 in the 20230921 log files), we see the EMAIL coming in fine on the Get..Details response, then being echoed in the last "Initiating cURL for donor xxx@yyy.zz" log line, so we can tell it was added to the gateway data array OK, and should have been sent along to the pending table. But the IPN logs show the recurring_payment_profile_created message and the recurring_payment messages came in with email blank.

So if the email got sent along to the pending table, why didn't the RecurringMessage actually merge in the email from the pending details? Seems to call self::mergePendingDetails for both of those txn_types in this switch statement: https://phabricator.wikimedia.org/diffusion/WFSP/browse/master/PaymentProviders/PayPal/RecurringMessage.php$32 .

I tried to see if it found the pending message, but I can't see "Searching for pending message with gateway" in the logs anywhere. I would have expected the logs for the jobs-paypal runner to be in fundraising-misc alongside e.g. the pending queue consumer since it's on the Civi box. I guess we haven't had paypal's log at debug level this whole time.

In T348760#9247897, @SHust wrote:

@jgleeson Thanks for the update! Does this mean they will automatically receive the 'ty receipt' after the emails reach Civii

These might have to be manually triggered, but I'll see if I can do that to as part of the task

Change 965537 abandoned by Jgleeson:

[mediawiki/extensions/DonationInterface@master] WIP: Handle Paypal not sending over donor email in GetExpressCheckoutDetails call

Reason:

This was more complicated than we thought

https://gerrit.wikimedia.org/r/965537

The latest is that we sent an email to PayPal on October 17th with some examples of the problem, and we're still waiting for a response. Email subject 'Payer's Email Address Missing in GetExpressCheckoutDetails API Responses [ Case # 12878050]'

@krobinson @Cstone I can see the email addresses for the donations affected from September are possibly being manually backfilled? However, it looks like donations are still coming in without emails, as we had three yesterday and 17 so far this month.

| Year-Month | emailless_paypal   |
+------------+--------------------+
| 2023-10    |                 17 |
| 2023-09    |                 36 |
| 2023-08    |                 10 |
| 2023-07    |                  4 |
| 2023-06    |                  3 |
| 2023-05    |                  3 |
| 2023-04    |                  1 |
+------------+--------------------+

I think they are being backfilled by the 2nd recurring donation, I put the ones affected in a group called 2023-10-19 Paypal no email, most have had the email added still ~50 without one I can keep working on that

Latest donation with no email was on Oct 29th, and already emailed Paypal team for investigation

The one time donations with missing emails have been back filled.

No new email missing record since Oct 29th, and still no email reply from paypal side yet, but they will keep the ticket open until we find a new record, since they can only do that record track within 2 weeks, and they are not working on our case yes, so all previous records are not able to be used.

status: waiting on paypal.