Deployed, and it looks like the css fixed it.
Thanks for the bug report, @jbolorinos-ctr !

Interesting, this looks like it's being caused by an automatic html-to-text conversion. In the HTML source it's like this:

...if they have a <a href="https://donate.wikimedia.org/wiki/Matching_Gifts">corporate matching gift program</a>.

@EMartin I can look up yesterday's examples too, but if you search in the Ingenico console now I believe you will see 73513909.1, 73514644.1, 73515520.2, and 73516052.3 out of status 600.

73485102.1 - orphan rectified failed because it was trying to warn us of low MinFraud queries and hit a codepath that doesn't work under Civi.
73484895.1 - looks weird in the orphan rectifier logs - rectified with result (emtpy array) and has no contrib ID.
73484088.1 - looks weird in the orphan rectifier logs - rectified with result (emtpy array) and has no contrib ID.
73516052.3 - orphan rectified at 17:39:59, stored as contrib ID 41561786
73481987.1 - looks weird in the orphan rectifier logs - rectified with result (emtpy array) and has no contrib ID.
73480663.1 - looks weird in the orphan rectifier logs - rectified with result (emtpy array) and has no contrib ID.
73515520.2 - orphan rectified at 17:39:08 UTC and stored with contrib ID 41561566
73514644.1 - was pushed through by orphan rectifier at 17:38:54 UTC and stored with contrib ID 41561515
73513909.1 - pushed through via the orphan rectifier at 17:38:27 UTC (maybe after you listed it) and is stored as contribution ID 41561415

OK, so most of these people got back to payments-wiki with a dead session. Not sure if that means our session storage is filling up or if it's just browser quirks.

OK, this is deployed

OK, this is up on staging (and looks good to me). I've sent a copy to @CCogdill_WMF and to @MBeat33 .

see also: T182148

We've changed the timing of the queue consumer and the thank you mailer so that they run at 3 minute intervals, with 2:30 of overlap. With this timing the queue delay has been staying generally under 2 minutes to civi import and under 5 minutes to TY mail send.

Ahh, while our fraud filters are one way things stop at 600, they're not the only way. If there's an error redirecting the user to our site, or if our API calls to capture / approve the payment fail, that can also leave the payment in limbo. I'll take a look at the logs to see if these donors all got back to our site and whether they are among the people affected by those sporadic API errors.

We didn't stop any of those for fraud - they're all listed with a near-zero risk score and validation_action=process. What made them appear as stopped for fraud to you?

We apply all of our in-house filters first, then skip the MinFraud check if we've already decided not to process the donation.

Hi! here's my request for Kerberos credentials for Hadoop access on stat100X and notebook100X. My username is ejegg.

Related discussion - as we import events from Silverpop, we may be turning temporary suppression into permanent opt-outs in Civi: T231254

Yep, we only export a given email address to one list or the other, never both.

Oh, even in October we were only getting new files on Fridays. It looks like we would need to subscribe to a different report in order to get daily files, and that would mean a bit of code changes to update the parser.

This looks like it's related to the small volume. I guess Adyen only sends us an audit file after a certain amount of donations have accumulated since the last one, but at least once per week. In October (when we had the France campaign going) we got 10 batch files delivered. I'll see if there are settings in the console to increase the frequency.

OK, so the new issue is not missing donors, but an unexpected number of donors marked as unsubscribed in the database. Maybe we should make a new ticket so as not to muddy the issue?
I wonder if these are related to the jobs @Eileenmcnaughton has been running over the past couple weeks to catch up on importing missing events (including unsubscribes) FROM silverpop.

OK, I see that EventLogging uses the same img.src fallback as the old pipeline beacon-sender, but that EventLogging also will skip sending if navigator.doNotTrack or window.doNotTrack are set. Do we know if that's enough to explain the difference? And if we have a good idea of what percentage of users on each UA are setting that, can we store more UA information so people can compensate for the dip at the end of the pipeline when querying the db?

@CCogdill_WMF those import screenshots show it was 3.27M on Nov 21st and 3.69M on Dec 1st. I'm sure it jumped quite a bit when we started putting people with latest_optin_response=NO on the suppression list.

The old pipeline is parsing logs of hits to /beacon/impression, sent via sendBeacon with a fallback of creating an img with that src if navigator.sendBeacon is false-y.
Does event-logging have a less robust fallback?
Or could it be the ordering of the events? I.e. adblockers deciding that one sendBeacon is acceptable but that the second should be blocked?

@KHaggard thanks for including the unsubscribes in that screenshot. It looks like a drop of 401,614 donors on the master list is matched by a rise of 426,685 donors on the unsubscribes list. Does that sound like a reasonable number of unsubscribes for 10 days of heavy email?

@KHaggard confirmed, the upload directory looks empty via SFTP.

Huh, yeah, I see some civicrm logs tagged with your client certificate for that day, but nothing right at that time. I'm trying to think why the database change log would have your name on it. Do you import donations via spreadsheet ever?

@KHaggard since today's upload was late too, this morning's file might still be sitting there waiting for processing.

OK, the timing of the upload does seem to be the issue. It was finishing before 9:00 till the morning of Nov 25, but on that day it was still uploading at 9:00. Yesterday and today it's been uploading late too.

The log shows that donation was edited twice recently by @RLewis - on 2019-11-12 22:32:46 she changed the appeal from 06annualfundraiser to E19_20, then on 2019-11-13 17:08:04 she changed it to E1920_Email

Patch set 32 includes all the improvements I can think to make short of throwing out the tree selector to replace it with something that would be clearer for simple country selection. Ready for more review, or suggestions on that alternate admin UI.

During the whole first week of the banner campaign last year we were raising between one and three million dollars a day, so it would be really stressful to lose our ability to diagnose and debug problems if that meant we were losing up to a yearly salary during a 4 hour outage. I second the request to move this to either before the banners start or at least a week (preferably two) weeks after they start.

Oops, never mind @MBeat33 , I just realized that the code @jgleeson is proposing does indeed include donations from all contacts with the same email.

@MBeat33 for the on-demand end-of-year summary email button that will appear on an individual contact record, did you want the email to only include donations on that contact record, or donations for all contacts that share the same email address?

All of the language pack XLS files I downloaded already had the value as 'Donate'. In the individual theme configurations for the 'iframe' and 'redirect' themes I had to change the value for each language from 'Pay' to 'Donate', and click 'Save and Publish' on each language individually for it to take effect.

OK, looks like the payments-wiki side would be pretty easy. Our base adaptor class has logic for switching accounts depending on transaction data (we needed it for WorldPay).

Regarding the text of this ticket: we only export one row to silverpop for each email address in our database, and we export it with the name address, and contact id of the most recently created contact record that has that email address.

OK, looks like the merged records for 24041039 had different versions of the email address (but which the email service treats as equivalent).

We can't see anything wrong with the data in the silverpop export table or in the Acoustic UI for 24041039

Deployed last night, and this morning's charge job was able to get through a payment for the donor with the overlong last name whose payment failed on the 13th.