Note that our link to download this data will expire in a couple of days - will evaluate this data in upcoming Sprint N

@Eileenmcnaughton I was eyeing this one up but I wanted to check whether you'd already planned to move on to it after the segmentation stuff.

Just noting that this phab task is connected and is the prep needed for the re-import phab.

I've uploaded the files from Data Axle to prod under /srv/org.wikimedia.civicrm-files/data-axle & am writing notes incase someone else chooses to pick this up

There are 2 zip files - I'm calling them 505 & 037 as the last 3 digits of each. 037 is larger. I have focussed on the smaller one (505) and created a blank data mapping for it - see the image below ( note that that my attempt to create an import template went wonky - but a data mapping is good enough in the first instanc)e. I have saved the full files along with files with a reduced amount of data (2 lines & 100 lines).

The goal would be to successfully import these through the UI, saving the data mapping in the process, and then script it to do the full amount. (The smaller files duplicate rows in the full file but it's OK to import a contact more than once as it should update it the same way each time)

In order to map in through the UI we need to work with @MDemosWMF
& @NNichols to ensure all the custom fields we need exist - per T336890 to map to. It would probably be better to re-use target smart fields & add new labels if that is possible in some cases. We may need to add new fields to the acoustic export

In previous years we decided NOT to map address fields, birthday or name fields as these were unchanged from data we sent out (except for in some cases some formatting was worse).

@Eileenmcnaughton this year we did do national change of address and DOB updates so I think we would want those imported. However, we do not need to import email since we didn't get new data for that and I don't believe we would need to update name fields either.

As far as the existing Targetsmart fields I believe when we talked earlier we realized we couldn't reuse these because the Data Axle coding was different. For example the Net Worth ranges are different but use the same letter coding. There was caution of overwriting that old data as well but perhaps archiving it somewhere?

Would it cause any issues to create all new fields? I think there was interest in creating a new tab for this data on the records for ease of viewing, but let us know your thoughts on that.

It sounds like @NNichols has experience with custom fields and I'm happy to assist with that if you need us to create those. We will just want to confirm where on the record these should go (Prospect tab or New tab).

@KHaggard has been looped in for awareness on the Acoustic side.

@ERoden-WMF returns next week so we can always have a larger conversation then if needed!

Just for clarification, Joseph Mando sent me a Delve list to suppress from our email sends. We don't have any room in the calendar for the next few months to import any new fields into Acoustic, so please hold on kicking off any nightly import changes that would affect Acoustic. Thanks!

@AKanji-WMF - just checking on this - I thought the idea was to not have this in our sprint this time to give space for snoozing

Thanks @Eileenmcnaughton ! Will move it out.

Change 992259 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Add managed template for data axle import

https://gerrit.wikimedia.org/r/992259

Change 992259 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Add managed template for data axle import

https://gerrit.wikimedia.org/r/992259

Change 994275 had a related patch set uploaded (by Damilare Adedoyin; author: Eileen):

[wikimedia/fundraising/crm@deployment] Add managed template for data axle import

https://gerrit.wikimedia.org/r/994275

Change 994275 abandoned by Damilare Adedoyin:

[wikimedia/fundraising/crm@deployment] Add managed template for data axle import

Reason:

https://gerrit.wikimedia.org/r/994275

@Eileenmcnaughton I looked into why you couldn't find the Single: Y/N field in the files and I think wires were crossed at some point and we received the slightly different field of 'CE_Selected_Individual_Marital_Status_Code.' I can update this in the doc linked above, but in Civi it will be labeled Marital Status and the options are: S /M / U (Single/Married/Unknown). Is there a way to update that in the prospect fields? @ERoden-WMF there must have been a miscommunication somewhere, but I think this is close enough to what we were looking for.

OK - that seems to have worked - https://civicrm.wikimedia.org/civicrm/contact/view?reset=1&cid=19 - I just need to kick off the big import now - which I will do soon

It is running https://civicrm.wikimedia.org/civicrm/queue/monitor?name=user_job_1101

Looks like the import is still chugging along slowly

Thanks for the update @Eileenmcnaughton and creating those 3 side phabs for the other fields we want to work on a bit more.

@MDemosWMF I think we are around 1/3 in - the first import fully but only about 1/4 of the second

Change 1006585 had a related patch set uploaded (by Eileen; author: Eileen):

[wikimedia/fundraising/crm@master] Increase the number of contacts that import per queue item

https://gerrit.wikimedia.org/r/1006585

@MDemosWMF these are imported now & can be found in the group here https://civicrm.wikimedia.org/civicrm/group/search?force=1&context=smog&gid=1942

There are 2 contacts who didn't import because some formatting in them messed up their csv rows

https://civicrm.wikimedia.org/civicrm/search#/display/Import_1114/Import_1114?_status=ERROR

I wasn't going to worry about that.

I think this is done now

Change #1006585 abandoned by Eileen:

[wikimedia/fundraising/crm@master] Increase the number of contacts that import per queue item

Reason:

https://gerrit.wikimedia.org/r/1006585