Wed, Nov 13
Cool suggestion! I like it.
Tue, Nov 12
Sat, Nov 9
Here are the API details!
Fri, Nov 8
Fwiw, I don't think the problem is the filtering scan, I think it's the timing of when this stripe donation got added to Civi. If I'm reading their contribution record right, their contribution was Enqueued At Timestamp Oct 24, 2019 16:10, which was 2 days after the email went out and 7 days after they donated.
Thanks for opening the ticket! If there's an API call to IBM to remove
people on the suppression list, that would be our best bet, though I don't
know if that exists. Should I ask our consultants?
Wed, Nov 6
Ah, I see. Here's the current login page:
You don't have to actually reset the IBM id, IIRC, only the campaign
automation login. Let me know if you need us to check with our consultant
Thanks, Elliott! We didn't realize this was a requirement with the system.
We learned our lesson ;)
Mon, Nov 4
Wonderful, thank you all for the help!
Okay, I saved the pw in a file called silverpop.txt as suggested.
Fri, Nov 1
I just reset the password on the account so Katie could get in and try to re-run the file imports, but now y'all need to reset the pw again on your side. I'm going to update the task title to reflect that.
Thu, Oct 31
Okay, then we need to find a way to use the silverpop API to remove people
from the suppression list when they opt back in.
That looks right to me! I want to note that from a silverpop perspective, scenario 2 would require an extra step of us removing a contact from the Suppression list. I think that has to be done manually, though maybe there's an API call for it.
Thu, Oct 24
I think my preference is option 2. I know we were concerned about people
who opted in NO and some point and changed their response to YES in the
future. It seems easier to me to set up an annual workflow where we remove
anyone from the unsubscribes list who has changed their response to YES.
Does that seem sensible?
Wed, Oct 23
So awesome! Thanks, everyone :)
Oct 16 2019
Huh, I remember us discussing that latest_optin_response field but I
thought we settled on opting these people out. I think that's what we need
to do--turn the opt out on the banner into a hard no.
Oct 8 2019
Definitely, the expected behavior is if someone opts-out of bulk email
while donating, they should be marked as opted-out in Civi and added to the
unsubscribes file that gets sent to IBM each night.
Oct 4 2019
Thank you all so much.
Oct 3 2019
Yeah, you'll need a CiviCRM cert to access the dash (
https://dash.frdev.wikimedia.org) at the least. Sorry, I take for granted
that we just set this up for fundraising people :)
Cool, we must've missed the cutoff time. Thanks, Katie!
Never manually add if it's data we're importing from Civi :) You should see
it when setting up the import. If you don't, just hit run on today's import
and we'll try again tomorrow.
I'm optimistic we caught it in time :) otherwise we'll try again tomorrow.
Hey Eileen, my mistake, I thought it had been deployed! We had already put
the import on pause last night so we could import the new data today. Is it
possible to squeeze it into today's import?
Oct 1 2019
Ooh awesome, sounds like hardly any lag at all, even in the event of a
Thanks so much for this! Just one last question--how often will this be
updated? Daily? Will there ever be a lag?
This is ultimately up to Michael, but I'm wondering if it'll be easier on
DS to send these in a few isolated batches so their team knows when to
expect them. Like 4 batchs of 10k, one per week?
We can use birth_date.
It would be great if she has access to all the same databases that I do,
but these are the ones on my mind:
Sep 24 2019
Okay, from meeting with Elliott, it sounds like the multi-currency
formatting is not a concern. I'm down with Michael's suggestion! Noting as
well that Michael confirmed it's good to remove endowment donations.
And @MBeat33 's suggestion above is fine for me, though I am skeptical as to how much sense it makes to support the annual totals in multiple currencies. Is that easy enough to do, or should we scrap the annual total?
Yes, omit endowment donations @Ejegg!
Sep 23 2019
Ah yessss this is so awesome, Eileen!! Would be good to rename that hash
field if possible to utm_source or mailing_id or something so it's easier
to find and join across tables, but I think this is exactly what I wanted
:D And just in time for our new analysts to use.
Sep 18 2019
Thanks, Eileen! We're going to set up a new recurring import tomorrow. I
think you can close this on your end.
Sep 11 2019
Thanks, Eileen! Let us know when that new data gets added. We didn't set up
those new fields to import on a recurring basis because we didn't want to
slow down import runtime, so we'll need to do another manual import to add
Sep 6 2019
Oh sorry! If we're talking exact matches, yes, I think they should be opted
out until the donor tells us otherwise. My bad 😊
Great, thanks @Eileenmcnaughton! I just told IBM to skip today's import so
I think the file should be around for Katie and me to import later. We'll
update this task with how it goes!
@Eileenmcnaughton let me know if I should pull in Trilogy for help!
Hm, I would disagree here. I might expect a donor would unsubscribe their
old email address knowing that they only check it for certain purposes and
only want to receive emails from us at a different address. I think we
should let donors specify communications preferences on a per-email basis.
Sep 4 2019
Sounds good! Thank you :)
It runs daily at 9am UTC.
Okay! Would be great to do this on the Friday morning UTC export since we
aren't sending emails on Friday, just in case something goes amiss with the
Aug 31 2019
Thanks, everyone, for jumping on a very unexpected task. I appreciate all
the follow-up and feedback.
Aug 29 2019
@MNoorWMF is checking into the opt-in language.
Aug 21 2019
Aug 20 2019
Thanks for scoping this out, @XenoRyet, seems like a pretty thorough
approach to me!
Aug 13 2019
Thanks so much for the quick turnaround, Eileen! I was able to disable the
automated import before the file was uploaded this morning and remapped the
fields. Everything seemed to import perfectly and I have successfully run
queries on the newly imported data. Seems like everything is working. I'm
happy to resolve this task! Thank you again.
Aug 12 2019
Hey all, I'm reopening this task as for some reason, we're only getting fields for 2018, 2019, and 2020. The whole reason we need these fields is to create donor segments based on *past* giving behavior, so we need fields showing donations going back at least the last five years (2014-2017 also need to be included).
Jul 30 2019
Spec doc for the preference center here: https://docs.google.com/document/d/1wKtGxJn06bs6OGAFV7DKUfVw2salKdU_NAUn5SMGks0/edit#
Jul 26 2019
Hi there! Just checking if there is an ETA on getting this data into silverpop? We are getting lots of pressure from higher up to provide our full segmentation strategy, and I had really been hoping to be able to work with the data in silverpop in order to create that schedule, so we don't have to duplicate work. If we don't see this getting done in the next week, please let me know and I will start querying in Civi to fully flesh out our segmentation strategy.
Jul 25 2019
Hmm... looks like I missed this two years ago :D
Jul 23 2019
IBM just likes to name its list types confusing (IMO) things. The two list
types we send to are Queries and Contact Lists. Not sure if it would help
if I walked someone through the UI to see what I mean, but happy to do that.
Jul 19 2019
Cool, thank you! That way we just disable the current import and set up a
new one mapping the new fields. Give me a 24 hours' heads up so we can
disable the active import (if it runs we lose the file on FTP and can't
It's easiest to add the fields once they appear in the export file. Are
they there already, just not populated?
Yep, we'd need to forget in each DB. And yes, the UI is spooky! But it
Here it is! https://www.mozilla.org/en-US/newsletter/existing/8813b665-cfa8-495b-ada5-9c0a9c011522/. Feel free to play with my preferences if you want--I'm opted into this list with multiple email addresses 🤓
I believe that a manual deletion in IBM and the forget me process are
different. The reason is that just in case this record exists anywhere else
in IBM, like on a different list, it will remain in IBM's logs. I think we
need to do the Forget Me if were going to do this properly.
Jul 18 2019
Thanks so much for the quick turnaround, Jeff! Katie is iiiiiin.
Thanks, Katie! Just a +1 here to prioritize this. Katie is our email stats
person (among other things!). We're running our Japan campaign which has
multiple tests per week, and need to be able to stay on top of stats. It
makes the campaigns way more labor intensive for Moska if she has to run
stats as well.
Jul 17 2019
Okay, looks like I missed something big here! @NNichols can I get some more context on why it's important to combine endowment gifts with annual fund gifts in these "latest donation" fields? These fields are some of the most important ones we use in our fundraising appeals. I'm worried that combining this data will get us in some legal trouble because we'll end up pulling them into our annual appeal emails and may suggest that an endowment gift actually went to WMF.
Oh huh I didn't realize that our totals were including endowment gifts. Can
we make it so that the non-endowment fields in the silverpop export are
just that--non endowment? We might get ourselves into some legal trouble if
our copy suggests that someone's last gift was to WMF when it was really to
the endowment... Why have these fields cross over when we can have two
Thanks for asking! It would be great to have some of those endowment
fields, too. These would be great to start with:
Okay, starting a thread with David+Eileen+Trilogy.
Got it, thank you!
Awesome! Are the stats from the one test we ran backfilled, or is that test a dud?
Hey all, apologies for not adding feedback sooner! Your suggestion was
interesting, Eileen, but felt like it's achieving something similar to the
calendar year totals we're adding. Maybe I'm not seeing what problem we're
trying to solve for here since I don't typically use the Civi UI? Let me
know if there's something for me to look at here.
@Eileenmcnaughton awesome! Do you have a projection on how long that will take? We've been waiting to be able to use this data to map out our Q2 schedule. Could we possibly have it run through Big English data before other stuff?
Hey @Eileenmcnaughton did this get deployed during the maintenance on
Hi @Eileenmcnaughton, thanks for investigating! I asked Brian and Eric at
Trilogy and they confirmed the API can't do this for us :( Their response,
Jul 9 2019
I can get the column headers for you super easily from IBM. If you actually
want to see the SQL that builds it, that's a different story. Should I get
you the column headers?
Awesome! You think we might be able to get new data in by EOW?
@DStrine is this about figuring out the different email preferences in Civi? i.e. On hold or Do Not Mail? Or is this about something else?
Hey all! I see something happened with this task last week. Should I be checking for data?
Jul 8 2019
Ah, duplicating my comment from T223935 here:
Last week, we performed the process I outlined above in silverpop and purged anyone from our Master Suppression List (MSL) who had been opted-out by the unsubscribes import. That removed 1.25M records. Then, the nightly import imported 936k new rows, which suggests that we successfully removed about 310k records from the MSL that were not supposed to be there. This feels like a big win!
Cool, when would it make sense to discuss this further? Happy to outline
our expectations and see what might work within the constraints of our
Jul 5 2019
Jul 3 2019
Would it be possible to time this change with T170972 so we only have to remap the fields once?
Jun 28 2019
Hmm I guess in a totally ideal world, we would keep a record of all engagements, but I could work with just the latest.
We sent a big bulk email blast on Thursday, so yeah, that seems entirely
Shouldn't be related to the opt-out pages. We aren't sending traffic there
until that tracking task is done.
Awesome! Am I right in thinking we should have seen decreases in... both export files last night? Or just the Main Database import? @KHaggard can you advise how the IBM imports looked from last night?
Jun 27 2019
We met with Katherine this week! We have a follow-up meeting in a couple weeks and should finalize copy at that point. Putting a reminder on my calendar to update y;all the week of July 15 :)
Jun 26 2019
Sounds great, thank you! :)
Jun 25 2019
Hmm, I guess I could do all of this with what we have in IBM. I could create a copy of the main database in silverpop and then purge all the people who have opted out using silverpop forms from it. I will then purge the remaining list from the master suppression list, and then the nightly unsubscribe import will re-import anyone we missed. Whew! @MNoorWMF does that all make sense to you? Would like to sanity check the method.
Assuming 'the variant of the page' = utm_source, yes! In other words, we
need to track our 3 main utms:
A related question has been bumping around in my mind: which contacts
opted out and are no longer? I.e. were there *on hold* people who were once
opted out and should be opted back in? I don't actually know if these cases
exist, but wondered if combining those into this request would allow us to
accomplish two goals at once.
So does that mean a bunch of opt-ins have been added to the unsubscribe
list? When an email gets added to IBM's master suppression list (which is
where the unsubscribes import points), it stays there, even if the next
night's file doesn't include that email. Can I get a list of people who
have been erroneously added to the suppression list? I think I can purge a
whole csv from the list as a bulk job.