Updating here that we've looped @Eileenmcnaughton in on our email thread with Chariot regarding this. They told us that dashboard APIs are going live later this month, which is what we'll need to get the data out of Chariot into Civi. We will have to wait until then to investigate further.

@Lars That's great! I will keep an eye out when I process this week.

@Elbar53 you can relay the information to Caitlin/Evelyn and see what they say as far as urgency.

Agreed! I think matching the zip would help as it seems that is more commonly included in the donor info. @Lars

@Lars Just want to jump in on this and say since the gift data is recorded on the DAF when present that record gets pulled into our list to send TYLs to. So we would want the contact information there to send acknowledgements to by email or mailing address.

@Lars It looks like we have the unique match rule for fidelity and Engage Organization import (below), but for Benevity we have the "first match" rule like you mentioned. I guess I am slightly confused on the difference between the two?

@Lars I would say save the existing data as secondary if it is different than what is already on the record. This would likely be our most up to date information if it was sent in with the gift, but the address/email history is still there. I agree with sharing from the Individual to the DAF so if one is updated with a new primary then both contacts are.

I will go ahead and import the ones above!

@Lars I don't believe I have seen it before, or if I have it has been a long time. I think this error when it shows up for not creating any record at all is more of a hinderance because we want it to create the record if one doesn't exist. If it has the same details as something already in Civi the import should dedupe based on the dedupe rules in place so I don't see it as having much use right now.

@Lars Here is the error report for a recent Chariot Organization import: https://civicrm.wikimedia.org/civicrm/search#/display/Import_4084/Import_4084?_status=ERROR

@Lars That's great, thanks for figuring this out!

@Eileenmcnaughton The fields in the file are all that I have available to me for the data entry. I'm not sure if more information lives somewhere else in the Paypal console.

@Lars Tried again - here is the error and some screenshots to show formatting:

@RLewis Yes, I would think if you are looking for a tally of incoming gift totals you would need to include the estimated incoming amount on the activity. I would say estimated because there are times where the tracker shows an amount the donor said they were giving but we receive a different amount. This is not super often, but just to be aware they can sometimes not match up. May not cause too much of an issue.

@Lars Very strange - I tried again and the same errors. Here is the page if you are able to take a look: https://civicrm.wikimedia.org/civicrm/import/contact?_qf_Preview_display=true&qfKey=CRMImportController7y2QRVY3fJNZ11BWuRaL6zUfnxFla32N0B5Z2tHs5c9iU_456

@Lars I don't have the error on hand at the moment, but basically it says there was no match found for that Organization name. Then I have to go in and create the record by hand before I can continue to import those gifts. This is using import template ID 3685. I will have more to import later in the week so I can get screenshots then.

@Eileenmcnaughton We enter the papyal grants using batch data entry weekly. All files are saved here that are exported from Paypal and have the donor/donation information if that helps! https://drive.google.com/drive/folders/1wMdv7SWqyN09iWNJEGY-fVfu-CGtlTHB?usp=drive_link

@Lars Yes, here are the current two we get the most volume through currently:

Thank you!

@Lars The same thing is happening with a new NCOA file. I also tried to open it as a google sheet and export as a .csv but it's not working for me. The file is here: https://drive.google.com/file/d/1XY1WPeBuVFTjuBHOJj0v8pDlfPipKdq3/view?usp=drive_link. Any ideas why it's registering as an invalid date format?

@Lars Yes please do! I am not sure how that occurred but noticed it as well. We definitely want 'White Mail' and not the one without a space.

@Eileenmcnaughton @Lars I don't know anything about the 'is major gift' but we are setting the fund info upon import of check gifts.

Ok I think I got these in! Imports should be complete now. @Lars @MRitch-WMF

@Lars Right, if it is an hour wait vs. causing other donation processing queue deadlocks perhaps we just roll with this unless it gets worse. @Elbar53 Does that sounds ok to you?

@Lars Noting that @Elbar53 had an import that was paused today for quite a while, ~1 hr, but it did end up going through on its own. Wondering if it is related to this? Noting here in case.

@Lars Just imported that last large file, but there are 77 that came back with no matching CID - can you help find the correct CID and get these into Civi? https://civicrm.wikimedia.org/civicrm/search#/display/Import_3917/Import_3917?_status=ERROR

Thanks!

@Eileenmcnaughton Sorry I accidentally shared the wrong version of the file. Here is the correct one: https://drive.google.com/file/d/1KORv4UJlT8szYClqh8JGwv9nz0l1APLH/view?usp=drive_link

The other two updated files are imported!

@Lars Could you upload the Individual WMF file saved here to Civi so I can import the updated activities? https://docs.google.com/spreadsheets/d/1qdtzaIcii1JufTzV9g0WpNuM1dG8gnCI/edit?usp=drive_link&ouid=110785378642940298706&rtpof=true&sd=true

@Lars Thanks for sorting this out! I am out the rest of the week, so if we need this in sooner I won't be able to but can put it on my to-dos for next week.

@Lars Was wondering if in the meantime we would be able to create two import templates for linked payers we are getting majority of gifts through currently: Groundswell and Daffy. Or is that too much work? Depending on the lift this could help cover our needs between now and getting the double soft credit feature for imports. Otherwise we are having to add that banking institution to each contribution manually since I don't think adding a soft credit/type is available as a mass update field in Civi (if we were to try and add the soft credit in bulk after import).

@Lars Agreed, I think creating the address for the individual and then sharing that with the DAF in an automated way would be a big help. This also applied to our batch data entry work. Not only if this is information is coming in through an import, but also if there is that DAF relationship link for an Org and individual that already exist in Civi for it to auto apply to the DAF if there is no address/email info there.

@Lars Ah I think I understand, yes it would be ideal to automate this if we can! Perhaps alongside the 2026 work.

@Lars good to know about the second soft credit difficulty level. It's great having that feature in batch data entry and was hoping there could be something similar for imports where we can map two different soft credits and soft credit types to the contribution for importing DAFs. We are hoping to utilized DAFpay more especially in 2026 so it would be great to have this on a roadmap for an early 2026 timeframe if we can. Let me know how that sounds. I think we are planning to use this payment method in Direct Mail down the road as well and we will need this capability if we are going to be able to scale on this payment method.

@Lars Thanks this is a big help! Will note that link to import future appeal/package codes.

Thank you @Lars ! These are now in Civi.

@Lars Here are the two files. All of the appeal codes are already in Civi, but I created a file as an example in case it is helpful for setup.

The full list @JMando would be able to weigh in on. I believe they have a static universe of CIDs they are working with, but not all are pulled in for each mailing. So not sure if it makes sense to run the whole universe to check for merges at a certain frequency or if we would rather run the specific lists for each mailing as we plan who is being pulled in.

@Lars Woops thanks! I have not done that before, but if you send the steps I could try it out.

@Lars I'm not sure I understand what you mean. Faircom creates the full list of CIDs as far as those that are chosen to mail to for each campaign. So if you wanted to look at those ahead of time they would need to send them over so they could be updated if needed with a new CID before finalizing.

@Lars @Anil1956 The only outstanding piece is re-importing those 305 errors with the correct CIDs. Not sure if that was done on your end or can you provide a file with that data to re-import?

@Lars I tried to update the date in the file and reimport with the YYYY-MM-DD format and correct CIDs, but when I do the import the updates I made to the date format in the file are not reflected. I tried to save multiple times and reimport, but it is not showing that I changed it to YYYY-MM-DD in the file. Any ideas?

@Lars Thanks! It looks like those are a part of the Faircom using old CID issue. I will try to find the new CIDs myself since there are so few and reimport.

@Lars I just tried to adjust again and it is still coming back with the error:

@Lars Got it, thanks!

@Lars I got it! Thanks! @Elbar53 @BJordan-WMF This is now implemented in the Navigation menu for batch data entry

@RLewis @DBautista-WMF for visibility - they may be a better help for some of these questions

Thanks @Lars! @Elbar53 FYI!

@Lars Not super urgent but in the edit form for Organizations the order of the fields is a bit off. I think the address type and primary selection should be at the top and then street address field after that, etc.

@Lars I think seeing the name of the linked record will be helpful because then we can better see if a new record will need to be created for the soft credit or not (or at least if we need to search and see if there is already a record but needs to be linked). I think that combined with the auto soft crediting would solve for our needs. I'm not sure how heavy of a lift that auto soft crediting is so if we need something to solve for easily adding the soft credit within batch data entry I think the View contact link you mentioned would still help to have a quick way to go straight to that contact and grab the CID to add to the soft credit field.

Great! Thank @Lars !

@Lars I usually go to the import templates page. Is that ok to continue doing?

@RLewis What's your take on this from MG perspective? Lars and I are trying to automate some gift data processes. I would think we could share both ways if one doesn't have an address but the other does (Org <> Individual)?

@Lars that makes sense I suppose since we would want it to be reflected accurately. I wouldn't want to mess up any geocoding that happens in Civi. I thought at first maybe there was an error with the coding, but if that is the accurate zipcode/city we should be ok. I don't think this comes up very often either from what I understand.

@Lars Just want to confirm if the record already exists in Civi we are able to make edits via a pop-up window as well, not only for entering a new record.

@Lars Thank you! Glad you were able to find out what it was.

@Lars I just tried again and it's all back! The mapping and everything. I wonder why it didn't show for that period of time..

@Eileenmcnaughton I'm not sure we will use this after all since we are now working on switching to Gr4vy

@Lars thanks for looking into it!

@Lars Agreed, I think the mapping issue is more concerning. I put the file up on the drive: https://drive.google.com/file/d/1jGfRa1mQAAkaODuI1YRjmUjoClIS9pDj/view?usp=drive_link

@Lars I just saved it in excel.

@Lars That part was fixed with the download, but now I have moved to the mapping page and it is completely blank. All of the saved mapping is gone - do you know why that would've happened? Is there a way to get that back?

Strange! It was a file from Engage that was .xls then I saved it as a .csv for import (maybe there was an error in saving on my end?) I will try that fix, thanks!

@Lars As far as totals it would be good to see the import net total and fee total as it is in Civi so we can cross check that everything imported correctly. Like you mentioned row count to see the number of transactions imported as well.

Just noting @ERoden-WMF flagged there may be an easier way to implement this using Metabase. I think @RLewis and @AFowler-WMF were going to submit the request for @IAckerman-WMF 's team to explore this in asana. Not sure where that stands currently though.

Yes, I've started using it seems to be working well!

Thanks Lars! fyi @BJordan-WMF

@Lars No big rush at all. @Eileenmcnaughton requested I throw this phab in now, but not top priority.

@Lars Yes that's right:

Thank you! @BJordan-WMF will try this out and let you know if anything comes up.

@RLewis let me know if that's alright with you and you can always pass any tickets my way if needed!

Got it, thanks! @Eileenmcnaughton

Thanks @AKanji-WMF ! They also offered to have their technical folks join a meeting with our team to answer any questions. I can loop them in once that is on the calendar.

@Lars Great! Thanks for that info.

Thanks @Lars ! I want to check quickly that the same is also true for the postmark date field?

@Lars @Eileenmcnaughton Yes, would be great to get clarification on what is recommended. We want to be careful not to cause any duplicates as well. Sometimes that would occur after the deadlock hit and we re-ran the import.