Page MenuHomePhabricator

Make Silverpop export leaner
Open, MediumPublic4 Estimated Story Points


I'm interested in seeing us make the Civi to IBM export leaner. It currently exports over 18 million rows of data every night. Every Big English, the export job ends up failing a few times, I think partly because the file is just so big. It also takes a long time to run the import on the IBM side, during which time no other data jobs can run (like sending a mailing).

The solution: in the nightly export from Civi, only export rows with values that have been updated in the last 48 hours. That will bring the row count down from over 18 million to what must be less than a few hundred thousand.

Additionally, this will address a problem we have: we can't perform normal data hygiene and purge undeliverable addresses from IBM, because the Civi job re-adds them to IBM each day. This prevents us from getting accurate list counts which impairs our projections.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 17 2017, 8:03 PM
CCogdill_WMF triaged this task as Medium priority.Aug 17 2017, 8:04 PM
CCogdill_WMF added a subscriber: MNoorWMF.
DStrine set the point value for this task to 2.

@CCogdill_WMF there may have been some miscommunication but we don't think we are updating the undeliverable in civi. We can't remember it this was deliberate or not. Do you remember any conversations on this?

It would be good if we could mark their contact records as such somehow,
but I think we decided not to unsubscribe undeliverables. The reason for
this is that sometimes an undeliverable is someone who has soft-bounced too
many times. This could have happened because we were blocked at their ISP.
If we are able to remove that block in a future fundraiser, we don't want
them permanently unsubscribed.

Hi @CCogdill_WMF we talked about this & we thought the emails should not get re-exported to civi if the data has been updated from silverpop to Civi. However, from what I can tell we are not yet updating bounced emails back to civi & that is based on my recollection of a conversation - plse update on

Hey there, sorry I'm just coming back to this task now! I just modified the task description to make the nouns clearer--I think you all thought I was talking about the opposite end of the process. @Eileenmcnaughton does this clarify my intention?

New DoD:
Let's just try to only export those updated in the last 48 hours. The email team would like to see how this affects their workflow. We will create new tasks after this for other ideas.

DStrine changed the point value for this task from 2 to 4.

Ccing @KHaggard so she can follow changes on this task.