@LeanneS I don't see the prospecting fields being missing when I look - cid 601342

I note that for some contacts their 'old' address is a subset of their new one - maybe having both at the point of import is fine as it gives us a good view on the data but we could look at a script at the end to remove any addresses that are a subset of the primary address

@LeanneS I imported the first 5 rows into this group - civicrm/group/search?force=1&context=smog&gid=669 do you want to check it out?

@LeanneS how urgent is this (trying to channel David - I think he would say 'when do you need it by')

sigh

@CCogdill_WMF the new fields should show up in about 3 hours (I set it going) - I guess you'll be asleep then :-)

@Jgreen we have made a minor update to the triggers but I need you to reload them now - the change is per https://gerrit.wikimedia.org/r/#/c/wikimedia/fundraising/crm/+/528594/1/sites/all/modules/wmf_civicrm/scripts/triggers.mysql

from Brian

this is rolled back atm

OK - written up here https://www.mediawiki.org/wiki/Fundraising_tech/CiviCRM#Data_transfer_between_CiviCRM_and_Watson_Campaign_Automation_(aka_Silverpop,_IBM_)

you can add it here

@NNichols have you added this yet ? Please do

Also note that @Ejegg has rolled back a different patch that went out this week so we will see what the output from that is

@MNoorWMF @CCogdill_WMF all the referenced fields are new fields per the deployment earlier this week. They don't seem to be outputting a different format but were they definitely added at the silverpop end?

@MBeat33 we pushed out a minor permissions change - see if they can access it now

OK - so the number of rows is double there - ie

OK I'm getting good results on staging even updating 150k contacts at once - so struggling to explain the hang. Still I have got some explain results to dig into running it on a large batch

Actually I think I have a patch that would amend them - just pushed it up

@MBeat33 it looks like we need to fix the deduper to work with lower permissions & I should do that. In the meantime I'm OK to give them the Donor Services Management role (which is what you have) until we sort this out if you think that's OK

OK - reversing this out of the deploy -

Back in doing for the backfill script

Yes, that was included

I included this in the deploy just now - seemed ok from looking & from comment above

OK - we have had one donation come in since they were re-enabled & it worked (short the date_of_largest_donation field which is currently excluded) - I'll work on a backfill now

@Jgreen on live there is now a trigger file sites/all/modules/wmf_civicrm/scripts/contribution_triggers.mysql which has the triggers minus the ones I think most likely to be problematic - if you can reload that

@CCogdill_WMF we are now ready to deploy the addition of

The most likely problem point would be the date of latest largest field - perhaps a contact with many donations of the same amount caused the issue ? I didn't hit it in my testing but that felt like the least robust piece in the puzzle

@LeanneS sigh - the options for the values have changed again

@LeanneS the 'full_address' is just street_address+supplemental_address+ supplemental_address_2+city + etc isn't it?

Do we want this investigation task still - I feel like we have sub-tasks now

@DStrine should we bump this out of the sprint - it's really the phab I created to track things once we've started off the background task so I would not expect to start it until next sprint

@LeanneS there are a couple of columns that I don't see mapped in your image

the script finished

So far I've found locally I'm missing

• option values for PG Stage
• University Affiliation
• Family Composition
• Occupation

I'm thinking here to use an extension for this & to convert our 'pre-checks' (ie. our throttle logic) to a hook. Undecided as yet whether just to track through settings like in other places or to investigate Civi's batch mechanism - will have a discussion with Tim