Page MenuHomePhabricator
Create Task
Maniphest T349358

Acoustic suppressed emails 2023-10-19
Closed, ResolvedPublic
Actions

Description

~40k emails came in on omnimail_recipient_process_unsubscribes which was causing the timeout failmails (taking 35 minutes to finish instead of ~1)

These all had status Suppressed which is not the user generated unsubscribe status (that is opt-out). Talking to @KHaggard we think it was from a list that was accidentally set to suppressed.

Since there were still suppressed unsubscribes coming in I turned off the omnimail_recipient_process_unsubscribes job at 00:11 UTC on 2023-10-20.

We then turned off the silverpop export on the acoustic side and fr-tech side as we don't want these accidental unsubscribes to get back to the master suppression list.

We need to:

  • Figure out if its possible to clear out the unsubscribes that are coming in from acoustic (not sure how this works or if its possible)
  • Remove all the accidental unsubscribes from civi
  • Turn the silverpop export and unsubscribe jobs back on

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Revert the rest of the erroneous unsubscribeswikimedia/fundraising/crmmaster+25 -18
Break up table create and batch runwikimedia/fundraising/crmmaster+5 -1
Remove Supressed from unsubscribeswikimedia/fundraising/crmmaster+2 -2
Test reverting a batch of 100 unsubscribeswikimedia/fundraising/crmmaster+45 -0
Customize query in gerrit

Event Timeline

Cstone created this task.Oct 20 2023, 4:24 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2023, 4:24 AM
Ejegg added subscribers: bsisolak, Ejegg.Oct 20 2023, 1:51 PM

@bsisolak I'm not sure if you're monitoring Phab, but we're wondering why Acoustic seems to have started sending 'Suppressed' events when we suppress donors via our scheduling process. Our Civi integration is treating those as unsubscribes, which is NOT what we want. Have you seen this with other clients? And is there a way to distinguish the 'suppressed from one send' events from 'suppressed forever because they unsubscribed' events?

bsisolak added a comment.Oct 20 2023, 3:05 PM

This was related to something @KHaggard sent me an email on, and that a large suppression list was added at the time of send.

You have been getting suppressed events from Acoustic for all time, it's been this way for time immemorial. I suspect you have never noticed as the numbers have been small.

• KHaggard triaged this task as High priority.Oct 20 2023, 3:52 PM

I'll also add that the reason why this is the first time we're seeing this is that we usually don't rely much on suppression lists because we handle everything with queries, however, I always thought of suppression lists as a possibility to use if we have rush jobs and need to suppress a group of people quickly. I didn't realize that if we deployed an email and the suppression was used that Civi would pick it up and translate it as unsubscribes. Definitely don't want going forward, as we may need the flexibility to suppress people for certain sends and not other sends.

I'm around today if there's any other information needed. Hoping we can resolve before emails need to start going out again on Monday. Due to the time constraint, I'm setting this task as high priority.

bsisolak added a comment.Oct 20 2023, 4:25 PM

More context. There are three types of suppressions (System, Org, Mailing). System suppressions are domains that are suppressed on all of Acoustic as they are not real, spam trap, generic garbage. Org level are the Master Suppression List, and Mailings level is when someone added a Suppression List at the time of send.

Because you update the Master Suppression List every night, you have perhaps never seen this before.

These should not be marked unsubscribes of course. This is similar to the issue years ago where you marked Reply Mail Blocks as unsubscribes. So larger question, what else to you mark as unsubscribes?

gerritbot added a comment.Oct 20 2023, 5:13 PM

Change 967481 had a related patch set uploaded (by Cstone; author: Cstone):

[wikimedia/fundraising/crm@master] Remove Supressed from unsubscribes

https://gerrit.wikimedia.org/r/967481

gerritbot added a project: Patch-For-Review.Oct 20 2023, 5:13 PM
Cstone added a comment.Oct 20 2023, 5:15 PM

@bsisolak Currently 'Opt Out', 'Reply Abuse', 'Suppressed' are all marked unsubsrcibes, the above patch is removing Suppressed from that list

gerritbot added a comment.Oct 20 2023, 6:54 PM

Change 967511 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Test reverting a batch of 100 unsubscribes

https://gerrit.wikimedia.org/r/967511

AKanji-WMF moved this task from Triage to Chaos Crew Backlog on the Fundraising-Backlog board.Oct 20 2023, 8:01 PM
AKanji-WMF added a project: Fundraising Tech - Chaos Crew.
gerritbot added a comment.Oct 20 2023, 8:07 PM

Change 967511 merged by Ejegg:

[wikimedia/fundraising/crm@master] Test reverting a batch of 100 unsubscribes

https://gerrit.wikimedia.org/r/967511

gerritbot added a comment.Oct 20 2023, 8:10 PM

Change 967481 merged by Ejegg:

[wikimedia/fundraising/crm@master] Remove Supressed from unsubscribes

https://gerrit.wikimedia.org/r/967481

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2023, 8:29 PM
gerritbot added a comment.Oct 20 2023, 9:13 PM

Change 967533 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Break up table create and batch run

https://gerrit.wikimedia.org/r/967533

gerritbot added a project: Patch-For-Review.Oct 20 2023, 9:13 PM
gerritbot added a comment.Oct 20 2023, 9:30 PM

Change 967533 merged by Ejegg:

[wikimedia/fundraising/crm@master] Break up table create and batch run

https://gerrit.wikimedia.org/r/967533

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2023, 9:30 PM
gerritbot added a comment.Oct 20 2023, 9:59 PM

Change 967537 had a related patch set uploaded (by Ejegg; author: Ejegg):

[wikimedia/fundraising/crm@master] Revert the rest of the erroneous unsubscribes

https://gerrit.wikimedia.org/r/967537

gerritbot added a project: Patch-For-Review.Oct 20 2023, 9:59 PM
gerritbot added a comment.Oct 20 2023, 10:36 PM

Change 967537 merged by jenkins-bot:

[wikimedia/fundraising/crm@master] Revert the rest of the erroneous unsubscribes

https://gerrit.wikimedia.org/r/967537

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2023, 11:10 PM
Cstone added a comment.Oct 23 2023, 7:05 PM

Update for what happened on Friday:
Elliott created a temp table with all the affected emails, we created a script to undo the unsubscribes based on the suppressed status. We ran that then turned all the jobs back on.

Cstone moved this task from Backlog to Done on the Fundraising Tech - Chaos Crew board.Oct 23 2023, 7:05 PM
XenoRyet closed this task as Resolved.Oct 23 2023, 7:39 PM
XenoRyet set Final Story Points to 8.
Dwisehaupt moved this task from Done to Completed 20231010 - 20231024 on the Fundraising Tech - Chaos Crew board.Oct 26 2023, 11:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits