@JSimmons I'm seeing a number of activity types:

@DBautista-WMF Is the request for Endowment highest donation date and Endowment last donation amount fields in Civi still relevant?

@SHust How would you feel about closing this one up? I think we've made some improvements in various ways and we are in better shape than we were. If you think we're at an acceptable speed baseline now (acknowledging that it could always be speedier), I think we should close this up and file a new phab if required in the future.

@Ejegg @MBeat33 Did this paymentswiki variant that allows higher donation ever get done (and if so have we replicated on combowiki)? Is it still something we want?

And of course the receipt itself will show the contact's primary email, not the email it is being sent to.

@AMJohnson This was a quick one, so I just did it rather than have it linger in the low priority column for a few years.

This has been implemented.

@NNgu-WMF I've added to the Acoustic spreadsheet, but was just waiting until all these new fields are filled to share.

@NNgu-WMF Yes, that's the one. It is being filled with data right now.

@SHust In this case we set their gmail address as ApplePay type based on their most recent donation earlier this year and then more recently they gave via ApplePay again with a different email, so we did replace that one. So I think this is indeed a case where it would be better to keep all emails.

Thanks @Dwisehaupt

@Dwisehaupt one more silverpop alter for this late addition:

ALTER TABLE silverpop_export
  ADD COLUMN foundation_recurring_latest_cancel_date DATETIME
  AFTER foundation_recurring_year_latest_donation_date;

@Dwisehaupt can you do this on silverpop:

ALTER TABLE silverpop_export_latest
  ADD COLUMN latest_cy_native_total DECIMAL(20, 2),
  ADD COLUMN latest_cy_count INT UNSIGNED;

Thanks @dmiranda, that makes sense to me. Two things stand out in the mobile screenshot that we should address here as well though, maybe in a separate patch: the space between the bottom of the top menu and the Hi PERSON is quite large and the top menu takes up a tonne of space. I wonder if we need those menu items at all on mobile? I think we could replace with a hamburger menu or we could just hide them entirely. Alternatively, the top menu for the donor portal is much better, already has a hamburger menu and matches the existing website (I think the EPC header is supposed to match the old website). Assuming we will in the not too distant future make EPC part of donor portal, I don't think we should spend a lot of time on this, so if the easiest answer is just hide the menu items, I think that's fine.

Also ignore every.org contact.

Another wrinkle, ideally we'd handle this for cases where the contact is automatically merged, but that can happen a few minutes later, so we'd need to think about how that might effect finance data (maybe we need to flag if a batch has been exported and queue up an adjustment in that case). This happened for a donation by cid 71644830 (a DAF), a duplicate individual was created at 18:55 and then was merged into the correct individual at 19:00 (and that individual had the DAF email record that would have triggered the update). We could also tackle this to some degree by improving matching to the individual instead of creating the duplicate in the first instance.

@Peter Yes, thanks! All of our runs seem to be around 7-8 minutes now, which is great.

@NNgu-WMF No, I think we need to make a clean break from those because they are fundamentally different from the new audience framework and having both will only cause confusion and queries that miss contacts or add more contacts. We definitely don't want to combine the two. The old monthly statuses are based on how long ago the last successful recurring donation was, the new framework looks at the status of the recurring.

So I think what happened is that Symfony mail was actually disabled in Civi, but it is a requirement of wmf-civicrm. Past headers from Failmail alerts show that it was sending through the same path as the other emails, not Symfony (and I have a vague memory that I noted something about the extension, probably that it was disabled). When we re-enabled wmf-civicrm, it re-enabled this requirement and now monolog uses Symfony mail.

This is also blocking our "Failmail alert" emails

Merged upstream

Just so we don't lose this, we want to add a setting for a list of contacts that we never want to update appeals for starting with: Humble Bundle, Anonymous, Foundation Source, Paypal Giving Fund

@NNgu-WMF is it just the lapsed status question that's a blocker here? We are doing a bunch of calculation currently for the old framework on every donation coming into Civi, so ideally we'd like to stop doing that sooner rather than later.

Upstream PR. No need for our own patch, because we don't have any of the changes that come before this yet.

@RLewis feel free to add something to my calendar for Wed or Thurs if you want to talk this through.

A few notes for tests:

@Peter or @pwangai For parallelizing these tests, can we make copies of the db and run each against a different copy? Or are we limited to one db?

Wow, thanks so much @Peter, that's a huge improvement with tmpfs.

@RLewis for DAFWTG, we don't retain the original appeal other than in our logs (for online gifts, we would have it in the tracking table, but not for DAF gifts). If we want to overwrite them but also retain the original, I suppose we could add a previous appeal field to the appeal change table.

Thanks!

@RLewis Sure, any testing you can do is great.

Still happening (62 of these today), so there's something else going on, will need investigation.

Thanks @ppenloglou let me know when it is deployed and we can close this up.

Here are the two Searchkits to find and set appeals for past donations before this was in effect. I've done the past updates just now (~55 in total), so assuming all is working well, we won't need these any more:
https://civicrm.wikimedia.org/civicrm/admin/search#/edit/9592
https://civicrm.wikimedia.org/civicrm/admin/search#/edit/9838

@RLewis Sounds good. Are any or all of these appeal codes also overwriteable for Direct Mail or DAF emails? Or are these just for relationship managers and we would only overwrite DAFWTG and White Mail for those? Ideally we'd just have one list of overwriteable appeals and not have to manage lists for each type of overwrite.

@RLewis Just want to remind you that the appeal codes need to be set for the DAF emails that went out last week (and the older ones from August too). We won't be able to set any appeals automatically until that's done.