Lol - I love the idea of deduping a dedupe phab task

@NNichols I just ran an update which ensured that any prospecting data was moved to the merged-to contact where there was prospecting data on the deleted contact but not the merged to contact. I think this is probably the majority of them & most of those remaining have valid prospecting data on both but I'll do some more queries

Related discussion https://phabricator.wikimedia.org/T221914

I added this to the sprint as we discussed during sprint planning I would bring in the phabs that tied in well with working on T235450

The issue of opting out is the subject of https://phabricator.wikimedia.org/T231254 - I'm going to focus this on more narrowly on resolving merge conflicts on dedupe data - I think if we have 2 contacts with the same email & for one it is on hold we should retain the email & retain on_hold automatically & not require user intervention

These are now all in. It's a bit hard at this point to determine if there was a problem or the server was just under load (per above) but let's see how we go next file

Looking at the first example the code correctly marked that person as opt_in = NO - which I believe is enough to stop them getting emails - that information is exported to Silverpop.

Currently the field you see on that form is treated as 'opt in' - ie that person was a 'NO' to opt_in but not 'opted out' - as described.

I just tried it again & it completed.

thanks @NNichols

@MBeat33 can this be closed?

I spent a while trying to replicate this & came to the conclusion that

1. most instances seemed to be cases where the lower contact id had been merged into the higher contact id (indicating the contacts had been 'flipped')
2. the checkboxes to carry across the data were not checked.

@LeanneS so I guess this is in now - when do you expect the next stripe file?

@NNichols I just tried to replicate this & failed. Is it the case that this ALWAYS happens or only intermittently? I note this was logged in August and there have been some relevant code changes since then but I'm assuming it was brought into the sprint because it's still causing problems?

@LeanneS she should have a civi email now but I assume you need @Jgreen to do the certificate & he'll need an email confirmation from Lisa I guess

@LeanneS it sounds like one or more row might have had an issue but we didn't narrow down which & what :-(

@LeanneS Ok - I tried with just one row - I wonder if it is something part way in? Did you try with the full file?

yay

This is the one I imported civicrm/contact/view?reset=1&cid=35768581

@LeanneS I just tried one row & it worked. Did it fail off the bat

OK I just deployed it - I'm not sure what time the import kicks off but next time it does it is included

@KHaggard just pinging you about co-ordinating deploying this - we are ready to go

@LeanneS this is done now

@DStrine @NNichols I still don't know how to replicate this :-(

@LeanneS this is still open but it's done isn't it?

@CCogdill_WMF - if you are ok to check your end I'll deploy this change tomorrow

It would need to be done by fr-tech I think - I don't think you can.

I just put up a very simple patch adding it at the end - from memory we don't need to co-ordinate too carefully if we add it at the end.

@CCogdill_WMF we have birth_date date (& in some cases deceased_date) - is birth date ok or do you need 'age'

The hash field is in core table so we wouldn't want to change the field - I think it's duplicated in the civicrm_campaign table but I think my reason for that I think was civicrm didn't support custom fields for mailings at the time.

@CCogdill_WMF I just managed to get past the last hurdle blocking data coming in - try doing this query

I also note that this would be more useful if we could dedupe by modified_date - the blocker is we don't have an index. I tested it & it was only a few minutes to add them

I've put up a patch for review & we have some options (@DStrine @Ejegg )

I've been testing the extension on staging with 'live tweaking' enabled. There is a lot of noise in the numbers and each time I do a bunch of tests it seems a bit slower. I wonder why? Maybe we should re-boot staging & see if it makes any difference?

civicrm_contact.is_opt_out shows up as no bulk emails user opt out. We fill in this field when users unsubscribe via Silverpop or DS & it affects our uploads to Silverpop (and we can't CiviMail them should we wish to)

@Ejegg this is what I use for the HAC project - however the structure of the extension is a bit incomplete - it was intended to permit the creation of smarty-template blocks that could be inserted into other templates.

@LeanneS OK to close?

@CCogdill_WMF let us know if this is closable

@Dwisehaupt I seem to be up & running on the new key!

@NNichols I removed spaces from the emails in civi - 1480 emails affected - but I don't think that's it.

@LeanneS this is now imported - the group is civicrm/group/search?reset=1&force=1&context=smog&gid=693 (2019-tsmart-gender_import ) - wee might choose to delete that group later

@CCogdill_WMF the extra gender data is now in so will be available for you from tomorrow. As I mentioned we are going to leave the extra fields in on our end

@MBeat33 ok to close?

@CCogdill_WMF I deployed a cut at this today but it's failing at the moment because of the firewall. I opened T232803 for that. Note that until that is fixed the whole job is failing & mailing data is not being updated. I think it will be resolved in the next day & there is no real impact but just letting you know

@Jgreen I keep thinking about this at times when you are not around - can you put the password in a text file in my home dir on staging?

This is now running & around 9000 are imported so far - the y are being put in the group 2019-tsmart-gender_import - although I doubt we need to keep that group long-term

good to hear - that was definitely confusing

@NNichols so 7k if you go right back but around $2k recently - but in many cases the data was also on the kept contact or has since been added toit

I've been testing this extension today

Just noting 89902 as a contact id to be updated for checking purposes when I try again tomorrow

I tested on staging & it updated gender for 10022 but my code to add them to a group (which I did want to do) was missing a beat so just need to merge that & try again)

@CCogdill_WMF we determined that it only takes 3 minutes longer at our end with these fields vs without so we are going to leave them in our nightly exports

Yes please - can you confirm it's still a thing today