@RLewis OK - the import will pick up the value in the Contribution Type column once the fix deployed - so if that can handle a mix in one file

@bsisolak yes - those 2 - our db is still tracking them - obviously they were 'processing' until very recently

Why would they be 'Failed' - we don't actually remove them from our tracking until we get a SUCCESS response - 2 of the 5 failed we are still checking on every cron run - ie the ones from July

@MBeat33 @krobinson we have deployed this from a technical point of view - but we added another non-technical step where you can do a final QA & approval before they are live. If you got to my record (11863262 ) and try to send me a failure message you will see the screen looks a bit different - this is specifically because I'm set to Spanish - which is a language with an unapproved version available (& no approved version). The languages in this state are

These contacts seem to have been fully deleted. In Silverpop there is a list of jobs in https://engage4.silverpop.com/ux/#/gdprjoblisting but it includes the 5 hung jobs (now showing as failed rather than 'processing' like last time) and a fistful of jobs for the last few days

The second one

So looking at the first one - he has 2 references in silverpop

@RLewis so it seems like the Endowment donations are in the main csv that gets imported? And they ARE being imported but the type is being ignored. From what you sent me it looks like the Contribution Type column has the data but it's being ignored - I' have a fix that will pay more attention to that column in the csv

It was added by mdemos-ctr (& then you fixed it to be Endowment)

I've loaded the series of patches ending in https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/635410 on staging. I then set the SMTP host to more than one server using

Thanks @Jgreen - very helpful. As a bonus I can see whether it went via SMTP or not as I applied your XMailer suggestion to the SMTP but it's not set on phpmail

Cool - thanks for confirming

@Dwisehaupt - are these the values we would need for config?

@Jgreen @Dwisehaupt - I have in my head that we want to use smtp - but I realised I need to confirm that

@Cstone this is now in 'done' - but did they get enabled on live? I did some patches to give us more control enabling - (ending in https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/634139 ) but those aren't merged - unless you did an install that I'm missing

I just enabled this with `drush vset match_on_import 1```

We can remove a civi-hack once completed is all good

@LeanneS this is live

@KHaggard there should hopefully be a new full output uploading.... It would be an additional file

I just replaced the view

OK - those queries ran - so check tonights update & see how it compares

& this would be the other query to implement @Ejegg's patch

I just ran this on on staging to reflect the change

Sounds good -

I'm not comfortable reviewing this patch due to the copy being in different languages so I have set it up (with a few patches) so I can generate tests on staging. Bonus - it permits us to deploy the updates & someone else (or us) can qa & enable individually if we want - I will forward emails on to @Cstone & @CDenes_WMF unless someone else wants to qa them

@KHaggard @bsisolak the opt out file should upload in the next hours & then daily. Filename is something like Optout-20201015131007.csv

Patch is merged through to deploy branch. Not deployed as yet - is any co-ordination needed?

@LeanneS did you test on like or staging - it seems OK for me on staging...

I just put up a concept patch - it might be enough depending on whether the file is just the same - it adds a checkbox

Also noting where the 'more data' is

@MBeat33 @KHaggard where is more info on these 5? There are perhaps co-incidentally 5 jobs that have been sent to silverpop that are still 'processing'

@LeanneS OK - I've put what I think is wanted onto staging as an update to 'Citibank Individuals' import

Also - this is the 'Citibank Individuals' import?

@LeanneS I just realised there is a question here - we have a citibank import & my initial assumption was this updates that - but it occurs to me that the

https://frmon.frdev.wikimedia.org/d/Pq1YNMviz/fundraising-overview?viewPanel=22&orgId=1&from=now-120d&to=now&refresh=1m

@jgleeson yeah - in fact that issue hit us last time too I think - but didn't realise & then it was really slow on live. This time I figured it out so it should be OK - the main thing is I test the update on staging & then deal with anything I spot

I also tried symfony 4 vs symfony 3 but locally could not measure a consistent difference

I did find a few memory leaks which I fixed in core - including one in our main code path & one in the dedupe code. But locally the speed difference was not significant. These are bundled with the 5.31 release along with other improvements - so we will see if it adds up to anything on prod

OK - I found a fix for this - https://lab.civicrm.org/dev/core/-/issues/2120

ALTER TABLE dev_civicrm.log_civicrm_address MODIFY log_id bigint(20) unsigned AUTO_INCREMENT

except it won't run on live...

Now I'm seeing

It's doing activity too

I've just deployed this on staging

To clarify - the fees not the donation amount would be off (in non-converted currency) - for the AUD & CAD if should be pretty close but more out for the 115 EUR , 41 GBP, 2 SGD & 1 PLN ones

@RLewis - you should be able to import now - per above we are thinking to skip the retrospective fix up on concern it is not worth the time

• the remaining ones will be in the civi update

I deployed one fix but I got more merged upstream which will be in the civi deploy

@RLewis we chatted about this on our fr-tech meeting - there are up to 600 donations with incorrect fee amounts - most less than $0.50 & it's not clear if some are OK in the mix - so we were thinking we would leave them rather than try to fix

@RLewis there are around 600 already-imported benevity contributions which *might* have inaccurate fee_amount values. The weird thing is there is just one of these with PLN currency, 4 with SGN and another 41 GBP - which are low numbers. The Benevity code somewhat assumes one currency per file - hence you can specify the currency for the import - how would these odd ones have come in?

Hmm - I think I'd like to pick up the fix question in tech talk - there's about 600 possibly affected donations

MariaDB [civicrm]> select sum(total_amount), count(c.id), sum(fee_amount), (sum(fee_amount) /sum(total_amount)) as total_fee_ratio, c.id, c.total_amount, c.fee_amount, fee_amount/total_amount as fee_ration, c.net_amount,

->        c.currency, original_amount, source, original_currency
-> FROM wmf_contribution_extra x
->   LEFT JOIN civicrm_contribution c on c.id = entity_id
-> WHere gateway = 'benevity' AND original_currency != 'USD'
-> 
-> AND fee_amount > 0 GROUP BY original_currency LIMIT 10;

+-------------------+-------------+-----------------+-----------------+----------+--------------+------------+------------+------------+----------+-----------------+-----------+-------------------+

+-------------------+-------------+-----------------+-----------------+----------+--------------+------------+------------+------------+----------+-----------------+-----------+-------------------+

+-------------------+-------------+-----------------+-----------------+----------+--------------+------------+------------+------------+----------+-----------------+-----------+-------------------+
6 rows in set (0.393 sec)

@mepps no there was an issue see https://phabricator.wikimedia.org/T264222#6507012

@RLewis I tracked it down - but it's a bit problematic. Basically we were converting the donation amount but not the fee amount - so for EUR it might look like

OK - I just deployed a fix