Lol - I love the idea of deduping a dedupe phab task
@NNichols I just ran an update which ensured that any prospecting data was moved to the merged-to contact where there was prospecting data on the deleted contact but not the merged to contact. I think this is probably the majority of them & most of those remaining have valid prospecting data on both but I'll do some more queries
Related discussion https://phabricator.wikimedia.org/T221914
I added this to the sprint as we discussed during sprint planning I would bring in the phabs that tied in well with working on T235450
The issue of opting out is the subject of https://phabricator.wikimedia.org/T231254 - I'm going to focus this on more narrowly on resolving merge conflicts on dedupe data - I think if we have 2 contacts with the same email & for one it is on hold we should retain the email & retain on_hold automatically & not require user intervention
Wed, Oct 16
These are now all in. It's a bit hard at this point to determine if there was a problem or the server was just under load (per above) but let's see how we go next file
Tue, Oct 15
Looking at the first example the code correctly marked that person as opt_in = NO - which I believe is enough to stop them getting emails - that information is exported to Silverpop.
Currently the field you see on that form is treated as 'opt in' - ie that person was a 'NO' to opt_in but not 'opted out' - as described.
I just tried it again & it completed.
@MBeat33 can this be closed?
Mon, Oct 14
I spent a while trying to replicate this & came to the conclusion that
- most instances seemed to be cases where the lower contact id had been merged into the higher contact id (indicating the contacts had been 'flipped')
- the checkboxes to carry across the data were not checked.
@LeanneS so I guess this is in now - when do you expect the next stripe file?
@NNichols I just tried to replicate this & failed. Is it the case that this ALWAYS happens or only intermittently? I note this was logged in August and there have been some relevant code changes since then but I'm assuming it was brought into the sprint because it's still causing problems?
Wed, Oct 9
Mon, Oct 7
@LeanneS it sounds like one or more row might have had an issue but we didn't narrow down which & what :-(
Sat, Oct 5
@LeanneS Ok - I tried with just one row - I wonder if it is something part way in? Did you try with the full file?
Fri, Oct 4
This is the one I imported civicrm/contact/view?reset=1&cid=35768581
@LeanneS I just tried one row & it worked. Did it fail off the bat
Thu, Oct 3
OK I just deployed it - I'm not sure what time the import kicks off but next time it does it is included
@KHaggard just pinging you about co-ordinating deploying this - we are ready to go
@LeanneS this is done now
Wed, Oct 2
@LeanneS this is still open but it's done isn't it?
Tue, Oct 1
@CCogdill_WMF - if you are ok to check your end I'll deploy this change tomorrow
It would need to be done by fr-tech I think - I don't think you can.
I just put up a very simple patch adding it at the end - from memory we don't need to co-ordinate too carefully if we add it at the end.
@CCogdill_WMF we have birth_date date (& in some cases deceased_date) - is birth date ok or do you need 'age'
Mon, Sep 23
The hash field is in core table so we wouldn't want to change the field - I think it's duplicated in the civicrm_campaign table but I think my reason for that I think was civicrm didn't support custom fields for mailings at the time.
@CCogdill_WMF I just managed to get past the last hurdle blocking data coming in - try doing this query
Sat, Sep 21
I also note that this would be more useful if we could dedupe by modified_date - the blocker is we don't have an index. I tested it & it was only a few minutes to add them
I've been testing the extension on staging with 'live tweaking' enabled. There is a lot of noise in the numbers and each time I do a bunch of tests it seems a bit slower. I wonder why? Maybe we should re-boot staging & see if it makes any difference?
Fri, Sep 20
civicrm_contact.is_opt_out shows up as no bulk emails user opt out. We fill in this field when users unsubscribe via Silverpop or DS & it affects our uploads to Silverpop (and we can't CiviMail them should we wish to)
Thu, Sep 19
@Ejegg this is what I use for the HAC project - however the structure of the extension is a bit incomplete - it was intended to permit the creation of smarty-template blocks that could be inserted into other templates.
Sep 18 2019
@LeanneS OK to close?
@CCogdill_WMF let us know if this is closable
Sep 13 2019
@Dwisehaupt I seem to be up & running on the new key!
@NNichols I removed spaces from the emails in civi - 1480 emails affected - but I don't think that's it.
@LeanneS this is now imported - the group is civicrm/group/search?reset=1&force=1&context=smog&gid=693 (2019-tsmart-gender_import ) - wee might choose to delete that group later
@CCogdill_WMF the extra gender data is now in so will be available for you from tomorrow. As I mentioned we are going to leave the extra fields in on our end
@MBeat33 ok to close?
@CCogdill_WMF I deployed a cut at this today but it's failing at the moment because of the firewall. I opened T232803 for that. Note that until that is fixed the whole job is failing & mailing data is not being updated. I think it will be resolved in the next day & there is no real impact but just letting you know
@Jgreen I keep thinking about this at times when you are not around - can you put the password in a text file in my home dir on staging?
Sep 12 2019
This is now running & around 9000 are imported so far - the y are being put in the group 2019-tsmart-gender_import - although I doubt we need to keep that group long-term
good to hear - that was definitely confusing
@NNichols so 7k if you go right back but around $2k recently - but in many cases the data was also on the kept contact or has since been added toit
I've been testing this extension today
Just noting 89902 as a contact id to be updated for checking purposes when I try again tomorrow
I tested on staging & it updated gender for 10022 but my code to add them to a group (which I did want to do) was missing a beat so just need to merge that & try again)
Sep 11 2019
@CCogdill_WMF we determined that it only takes 3 minutes longer at our end with these fields vs without so we are going to leave them in our nightly exports
Yes please - can you confirm it's still a thing today