After a bit of code diving I came up with a fix that makes it checked by default if the contact about to be removed has data in any (non location) field and the contact being kept does not - this would apply to prospecting fields & also to fields like middle name or other custom fields

@NNichols should be ok again now

@DStrine I've upped this to 'unbreak now' as there is something actively broken. We discussed on IRC & it seemed like it didn't merit disturbing Jeff or Dallas after hours but ideally first thing when they start their day . I think the file to load should be safe enough to 'just do'

@Jgreen @Dwisehaupt - could one of you load this triggers file - under webroot

I added this https://phabricator.wikimedia.org/T236111 as a follow up on the preventative side

It looks like a custom field was deleted - 'Prior WMF Giving' - however, the triggers are still trying to update that data - causing the error. I need @Jgreen or @Dwisehaupt to help me reload them

Added a civi user id - just need the cert now

@NNichols is this ongoing? Just trying to rule out a load issue

Lol - I love the idea of deduping a dedupe phab task

@NNichols I just ran an update which ensured that any prospecting data was moved to the merged-to contact where there was prospecting data on the deleted contact but not the merged to contact. I think this is probably the majority of them & most of those remaining have valid prospecting data on both but I'll do some more queries

Related discussion https://phabricator.wikimedia.org/T221914

I added this to the sprint as we discussed during sprint planning I would bring in the phabs that tied in well with working on T235450

The issue of opting out is the subject of https://phabricator.wikimedia.org/T231254 - I'm going to focus this on more narrowly on resolving merge conflicts on dedupe data - I think if we have 2 contacts with the same email & for one it is on hold we should retain the email & retain on_hold automatically & not require user intervention

These are now all in. It's a bit hard at this point to determine if there was a problem or the server was just under load (per above) but let's see how we go next file

Looking at the first example the code correctly marked that person as opt_in = NO - which I believe is enough to stop them getting emails - that information is exported to Silverpop. You can see the 'NO' on the left

Currently the field you see on that form is treated as 'opt in' - ie that person was a 'NO' to opt_in but not 'opted out' - as described.

I just tried it again & it completed.

thanks @NNichols

@MBeat33 can this be closed?

I spent a while trying to replicate this & came to the conclusion that

1. most instances seemed to be cases where the lower contact id had been merged into the higher contact id (indicating the contacts had been 'flipped')
2. the checkboxes to carry across the data were not checked.

@LeanneS so I guess this is in now - when do you expect the next stripe file?

@NNichols I just tried to replicate this & failed. Is it the case that this ALWAYS happens or only intermittently? I note this was logged in August and there have been some relevant code changes since then but I'm assuming it was brought into the sprint because it's still causing problems?

@LeanneS she should have a civi email now but I assume you need @Jgreen to do the certificate & he'll need an email confirmation from Lisa I guess

@LeanneS it sounds like one or more row might have had an issue but we didn't narrow down which & what :-(

@LeanneS Ok - I tried with just one row - I wonder if it is something part way in? Did you try with the full file?

yay

This is the one I imported civicrm/contact/view?reset=1&cid=35768581

@LeanneS I just tried one row & it worked. Did it fail off the bat

OK I just deployed it - I'm not sure what time the import kicks off but next time it does it is included

@KHaggard just pinging you about co-ordinating deploying this - we are ready to go

@LeanneS this is done now

@DStrine @NNichols I still don't know how to replicate this :-(

@LeanneS this is still open but it's done isn't it?

@CCogdill_WMF - if you are ok to check your end I'll deploy this change tomorrow

It would need to be done by fr-tech I think - I don't think you can.

I just put up a very simple patch adding it at the end - from memory we don't need to co-ordinate too carefully if we add it at the end.

@CCogdill_WMF we have birth_date date (& in some cases deceased_date) - is birth date ok or do you need 'age'

The hash field is in core table so we wouldn't want to change the field - I think it's duplicated in the civicrm_campaign table but I think my reason for that I think was civicrm didn't support custom fields for mailings at the time.

@CCogdill_WMF I just managed to get past the last hurdle blocking data coming in - try doing this query

I also note that this would be more useful if we could dedupe by modified_date - the blocker is we don't have an index. I tested it & it was only a few minutes to add them

I've put up a patch for review & we have some options (@DStrine @Ejegg )

I've been testing the extension on staging with 'live tweaking' enabled. There is a lot of noise in the numbers and each time I do a bunch of tests it seems a bit slower. I wonder why? Maybe we should re-boot staging & see if it makes any difference?

civicrm_contact.is_opt_out shows up as no bulk emails user opt out. We fill in this field when users unsubscribe via Silverpop or DS & it affects our uploads to Silverpop (and we can't CiviMail them should we wish to)

@Ejegg this is what I use for the HAC project - however the structure of the extension is a bit incomplete - it was intended to permit the creation of smarty-template blocks that could be inserted into other templates.