Page MenuHomePhabricator

Monitor the return of RML import in December
Closed, ResolvedPublic

Description

In previous years we have turned off this job during BE (with intermittent success at turning it back on) - creating a phab to make sure we turn it on next sprint

@EYener @KHaggard @MBeat33 this just is failing at lot at the moment & has not historically run during peak flow @Ejegg

Related Objects

StatusSubtypeAssignedTask
ResolvedNone

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 1 2020, 11:52 PM

Hi @Eileenmcnaughton!! Could you give us a bit more context on this? We have 3 RML programs running at the moment for en6c countries so I'd like to know the implications of turning off this RML import. Thanks!

DStrine added a subscriber: DStrine.Dec 2 2020, 4:50 PM

This is the RML data flowing from Acoustic to Civi. Acoustic still has all these records but we are not creating new records in civi from them at the moment. That typically has not interfered with your workflows when emailing these groups. I don't imagine this is a blocker for any of your campaigns at the moment but let's discuss if you have more questions.

Ejegg added a comment.Dec 2 2020, 4:50 PM

@MSuijkerbuijk_WMF This would just delay bringing the RML emails from Acoustic into Civi (creating email-only contacts) for the next couple of weeks.

The job that brings those in needs to write to some of the same database tables as the job which brings in donations from payments-wiki, and right now the contention means that the RML import job is failing almost every time it runs (and sometimes preventing a few donations from getting into Civi).

Do your RML workflows generally involve working with the email-only contact records in Civi right after they sign up?

@Ejegg I'm afraid I can't answer that question, I don't know how the RML data connects with Civi.
We do have specific query criteria in the RML programs referring to last_donation_date to exclude recent donors, but that should exist on Acoustic, right?
@KHaggard or @MNoorWMF do you know if stopping the RML data import would affect the current RML programs running and how that connects with Civi?

EYener added a comment.Dec 2 2020, 5:58 PM

@DStrine Any chance we can chat about this during Civi Fortnightly? I'd also like to learn more / hear some details.

@EYener I can't make that call, late for me. So could you record it? Or update me after?

A bunch of us had a conversation about this and it should not be blocking any email sends in the near future. This export to civi is really just "nice to have" for record keeping. This can be turned off for now. However I will make another cleanup task around the rml import.

DStrine triaged this task as Medium priority.Dec 3 2020, 7:21 PM

This is 'done' (off) but lets not close this until it's on again

Hi all, just to chime in here, this is actually causing workflow issues for DS.

We are unable to unsubscribe donors who are minors or who signed up by mistake as they are not in Civi and we get an error that says 'information not found' when plugging their address into any of the unsub links.

We also can't snooze for 90 days donors those who reply on email 1 and who want to donate via check, who send us a sob story and explain they are unable to this year, those who clicked the link but already donated and just did it to close the banner... This means they get an email 2 regardless, and that is just bad for donor engagement.

If we are unable to do any admin on contacts that are not in Civi, then for us the best workaround for us is not to send email 2s until they are in Civi, as it makes our life really difficult. Unless we can have an unsub link that speaks to acoustic and not Civi.

Thoughts are welcome, but I would really like us to look at this as it's putting us between a rock and a hard place.

Ejegg added a comment.Dec 8 2020, 3:30 PM

Looks like we pulled in 147,978 contact over today's run. Should be caught up after tomorrow's!

Great news, thank you @Ejegg

DStrine renamed this task from Turn off import of RML during peak load to Monitor the return of RML import in December.Dec 8 2020, 8:14 PM
Eileenmcnaughton added a parent task: Restricted Task.Dec 14 2020, 10:48 PM

There are only 6900 contacts that we have not retrieved. However we seem to be unable to retrieve them - my suspicion is they are on the Silverpop suppression list - a question I raised over on https://phabricator.wikimedia.org/T269296 with Brian.

In terms of this phab I think we are resolved though

Change 649474 had a related patch set uploaded (by Eileen; owner: Eileen):
[wikimedia/fundraising/crm@master] Enotice fix

https://gerrit.wikimedia.org/r/649474

Change 649474 merged by jenkins-bot:
[wikimedia/fundraising/crm@master] Enotice fix

https://gerrit.wikimedia.org/r/649474

SHust added a subscriber: SHust.Dec 18 2020, 6:16 PM
DStrine closed this task as Resolved.Jan 5 2021, 8:08 PM