Page MenuHomePhabricator

Duplicates after batch data entry
Closed, ResolvedPublic

Description

After doing a batch data entry import on 12/18 I saw that most of the gifts in the file dated 12/16 -12/17 were duplicated in Civi. There is a searchkit here showing the duplicates: https://civicrm.wikimedia.org/civicrm/admin/search#/edit/3973

We want to remove these as soon as we can so the data is accurate, so let me know when you are done looking at them. I do know when I did the import it gave me the 'database deadlock, return to home' notice, so I submitted the batch again. I have done this before and it doesn't always duplicate the gifts when I do this. Sometimes it does, sometimes it doesn't. I'm not sure what it could be and want to make sure we are not causing the duplicates. It seems to be something going on with the Civi side.

Here is the import file as well: https://docs.google.com/spreadsheets/d/1Ko7T1wan6t7-yPnbfKE5jpAPNHGoJ1K8/edit?usp=drive_link&ouid=110785378642940298706&rtpof=true&sd=true

Event Timeline

Hi @MDemosWMF thanks for raising this - please go ahead and delete the duplicates, flagging for @Eileenmcnaughton to have a look.

@MDemosWMF - we did used to have strict rules in place to prevent duplication & we relaxed them over time due to all the exceptions - that is probably what has changed

"it gave me the 'database deadlock, return to home' notice, so I submitted the batch again" - did that happen when you clicked submit?

@Eileenmcnaughton Yep, after I clicked submit it does the twirl triangle like it's processing and then goes to another screen that says 'database deadlock, return to home.'

Maybe it's finding a middle ground - is there a way for it to show which gifts have gone in before the DB deadlock hit? maybe on that DB deadlock screen? Right now we would have to create a search to try and see if anything went through and that can take time.

@MDemosWMF where are we at with this one - do we still need to do anything?

@BJordan-WMF Can you please weigh in on how the batch data entry imports have been going and if you notice these duplicates after import still occurring?

@MDemosWMF I do believe this is still a presenting issue. I have been splitting large batches into two parts to avoid deadlocks, but I had one last week that resulted in duplicates.

I think the issue here is not knowing what gifts made it into Civi without doing a separate search after hitting the deadlock - could it operate like the mapping imports where it shows which gifts imported directly on that screen? More visibility into what has occurred with the import on the deadlock screen would be helpful.

Eileenmcnaughton renamed this task from Duplicates after import to Duplicates after batch data entry.Jul 8 2025, 11:10 PM

@MDemosWMF Ok to close this one since we've switched to the new imports?

@Lars Yes, I think this should be resolved - we will let you know if it pops up again!

Lars claimed this task.