Wed, Sep 18
Thanks, Eileen! We're going to set up a new recurring import tomorrow. I
think you can close this on your end.
Wed, Sep 11
Thanks, Eileen! Let us know when that new data gets added. We didn't set up
those new fields to import on a recurring basis because we didn't want to
slow down import runtime, so we'll need to do another manual import to add
Fri, Sep 6
Oh sorry! If we're talking exact matches, yes, I think they should be opted
out until the donor tells us otherwise. My bad 😊
Great, thanks @Eileenmcnaughton! I just told IBM to skip today's import so
I think the file should be around for Katie and me to import later. We'll
update this task with how it goes!
@Eileenmcnaughton let me know if I should pull in Trilogy for help!
Hm, I would disagree here. I might expect a donor would unsubscribe their
old email address knowing that they only check it for certain purposes and
only want to receive emails from us at a different address. I think we
should let donors specify communications preferences on a per-email basis.
Wed, Sep 4
Sounds good! Thank you :)
It runs daily at 9am UTC.
Okay! Would be great to do this on the Friday morning UTC export since we
aren't sending emails on Friday, just in case something goes amiss with the
Sat, Aug 31
Thanks, everyone, for jumping on a very unexpected task. I appreciate all
the follow-up and feedback.
Thu, Aug 29
@MNoorWMF is checking into the opt-in language.
Aug 21 2019
Aug 20 2019
Thanks for scoping this out, @XenoRyet, seems like a pretty thorough
approach to me!
Aug 13 2019
Thanks so much for the quick turnaround, Eileen! I was able to disable the
automated import before the file was uploaded this morning and remapped the
fields. Everything seemed to import perfectly and I have successfully run
queries on the newly imported data. Seems like everything is working. I'm
happy to resolve this task! Thank you again.
Aug 12 2019
Hey all, I'm reopening this task as for some reason, we're only getting fields for 2018, 2019, and 2020. The whole reason we need these fields is to create donor segments based on *past* giving behavior, so we need fields showing donations going back at least the last five years (2014-2017 also need to be included).
Jul 30 2019
Spec doc for the preference center here: https://docs.google.com/document/d/1wKtGxJn06bs6OGAFV7DKUfVw2salKdU_NAUn5SMGks0/edit#
Jul 26 2019
Hi there! Just checking if there is an ETA on getting this data into silverpop? We are getting lots of pressure from higher up to provide our full segmentation strategy, and I had really been hoping to be able to work with the data in silverpop in order to create that schedule, so we don't have to duplicate work. If we don't see this getting done in the next week, please let me know and I will start querying in Civi to fully flesh out our segmentation strategy.
Jul 25 2019
Hmm... looks like I missed this two years ago :D
Jul 23 2019
IBM just likes to name its list types confusing (IMO) things. The two list
types we send to are Queries and Contact Lists. Not sure if it would help
if I walked someone through the UI to see what I mean, but happy to do that.
Jul 19 2019
Cool, thank you! That way we just disable the current import and set up a
new one mapping the new fields. Give me a 24 hours' heads up so we can
disable the active import (if it runs we lose the file on FTP and can't
It's easiest to add the fields once they appear in the export file. Are
they there already, just not populated?
Yep, we'd need to forget in each DB. And yes, the UI is spooky! But it
Here it is! https://www.mozilla.org/en-US/newsletter/existing/8813b665-cfa8-495b-ada5-9c0a9c011522/. Feel free to play with my preferences if you want--I'm opted into this list with multiple email addresses 🤓
I believe that a manual deletion in IBM and the forget me process are
different. The reason is that just in case this record exists anywhere else
in IBM, like on a different list, it will remain in IBM's logs. I think we
need to do the Forget Me if were going to do this properly.
Jul 18 2019
Thanks so much for the quick turnaround, Jeff! Katie is iiiiiin.
Thanks, Katie! Just a +1 here to prioritize this. Katie is our email stats
person (among other things!). We're running our Japan campaign which has
multiple tests per week, and need to be able to stay on top of stats. It
makes the campaigns way more labor intensive for Moska if she has to run
stats as well.
Jul 17 2019
Okay, looks like I missed something big here! @NNichols can I get some more context on why it's important to combine endowment gifts with annual fund gifts in these "latest donation" fields? These fields are some of the most important ones we use in our fundraising appeals. I'm worried that combining this data will get us in some legal trouble because we'll end up pulling them into our annual appeal emails and may suggest that an endowment gift actually went to WMF.
Oh huh I didn't realize that our totals were including endowment gifts. Can
we make it so that the non-endowment fields in the silverpop export are
just that--non endowment? We might get ourselves into some legal trouble if
our copy suggests that someone's last gift was to WMF when it was really to
the endowment... Why have these fields cross over when we can have two
Thanks for asking! It would be great to have some of those endowment
fields, too. These would be great to start with:
Okay, starting a thread with David+Eileen+Trilogy.
Got it, thank you!
Awesome! Are the stats from the one test we ran backfilled, or is that test a dud?
Hey all, apologies for not adding feedback sooner! Your suggestion was
interesting, Eileen, but felt like it's achieving something similar to the
calendar year totals we're adding. Maybe I'm not seeing what problem we're
trying to solve for here since I don't typically use the Civi UI? Let me
know if there's something for me to look at here.
@Eileenmcnaughton awesome! Do you have a projection on how long that will take? We've been waiting to be able to use this data to map out our Q2 schedule. Could we possibly have it run through Big English data before other stuff?
Hey @Eileenmcnaughton did this get deployed during the maintenance on
Hi @Eileenmcnaughton, thanks for investigating! I asked Brian and Eric at
Trilogy and they confirmed the API can't do this for us :( Their response,
Jul 9 2019
I can get the column headers for you super easily from IBM. If you actually
want to see the SQL that builds it, that's a different story. Should I get
you the column headers?
Awesome! You think we might be able to get new data in by EOW?
@DStrine is this about figuring out the different email preferences in Civi? i.e. On hold or Do Not Mail? Or is this about something else?
Hey all! I see something happened with this task last week. Should I be checking for data?
Jul 8 2019
Ah, duplicating my comment from T223935 here:
Last week, we performed the process I outlined above in silverpop and purged anyone from our Master Suppression List (MSL) who had been opted-out by the unsubscribes import. That removed 1.25M records. Then, the nightly import imported 936k new rows, which suggests that we successfully removed about 310k records from the MSL that were not supposed to be there. This feels like a big win!
Cool, when would it make sense to discuss this further? Happy to outline
our expectations and see what might work within the constraints of our
Jul 5 2019
Jul 3 2019
Would it be possible to time this change with T170972 so we only have to remap the fields once?
Jun 28 2019
Hmm I guess in a totally ideal world, we would keep a record of all engagements, but I could work with just the latest.
We sent a big bulk email blast on Thursday, so yeah, that seems entirely
Shouldn't be related to the opt-out pages. We aren't sending traffic there
until that tracking task is done.
Awesome! Am I right in thinking we should have seen decreases in... both export files last night? Or just the Main Database import? @KHaggard can you advise how the IBM imports looked from last night?
Jun 27 2019
We met with Katherine this week! We have a follow-up meeting in a couple weeks and should finalize copy at that point. Putting a reminder on my calendar to update y;all the week of July 15 :)
Jun 26 2019
Sounds great, thank you! :)
Jun 25 2019
Hmm, I guess I could do all of this with what we have in IBM. I could create a copy of the main database in silverpop and then purge all the people who have opted out using silverpop forms from it. I will then purge the remaining list from the master suppression list, and then the nightly unsubscribe import will re-import anyone we missed. Whew! @MNoorWMF does that all make sense to you? Would like to sanity check the method.
Assuming 'the variant of the page' = utm_source, yes! In other words, we
need to track our 3 main utms:
A related question has been bumping around in my mind: which contacts
opted out and are no longer? I.e. were there *on hold* people who were once
opted out and should be opted back in? I don't actually know if these cases
exist, but wondered if combining those into this request would allow us to
accomplish two goals at once.
So does that mean a bunch of opt-ins have been added to the unsubscribe
list? When an email gets added to IBM's master suppression list (which is
where the unsubscribes import points), it stays there, even if the next
night's file doesn't include that email. Can I get a list of people who
have been erroneously added to the suppression list? I think I can purge a
whole csv from the list as a bulk job.
Jun 24 2019
Thanks for looking at this @DStrine. The question of hits was only one part of this ask. The Unbreak now portion was this:
Jun 14 2019
Jun 13 2019
I think this is extraneous now. Closing :)
Hey all, update on copy. We're meeting with Katherine in two weeks to draft new copy for the annual TY email and will get this email together after that meeting. I will update this task by Friday, June 28.
Jun 11 2019
Yeah, I think so. Thank you so much!
cc @KHaggard, I'm going to tag you in a trello card with some next steps but here is the context if you're curious.
Sigh... so this definitely seems to be a field mapping issue on the IBM side. My apologies! I think I know how the problem originated: the field that civi passes over in the export is called opted_in which is the same name as an IBM system field. When we set up new imports (i.e. if it breaks and we need to kick the job, or if we add a new column and need to remap fields), IBM defaults to overwriting whatever data is in this field in place of their system data. There is a manual way to override this, but it is easy to miss and with more people joining the team, the likelihood this mistake repeats itself is high.
Jun 6 2019
Er, weird! I sent this reply via email yesterday morning but it looks like it didn't post:
May 31 2019
Hey @XenoRyet, yeah, it's pretty easy to rerun the queries. Looks like the numbers haven't really changed.
May 28 2019
@MNoorWMF ccing you so you can help me implement this when ready! :)
Talked with David about this last week. He confirmed this auto-email will come from Civi, just need to get fr-tech the copy. We will get a draft to you by Friday, June 7!
May 22 2019
Doh! Embarrassed I missed that and grateful for your keen eyes, @MBeat33.
May 21 2019
Awesome @Ejegg!! Thank you!
May 15 2019
Okay great, thanks!
Good catch, Sam! +1 to adding validation.
Thanks for working on this! cc @MBeat33
May 10 2019
Awesome! It's in trello for email 1. Thanks again!
Per your suggestion @Pcoombe, I'd love to test this in Spain! It's ideal timing because we need big lists sizes for landing page tests and we have the room to just cut the esES list in half.
May 9 2019
Cool, seems fine to me!
May 8 2019
I think we should go with whatever behavior the dedupe process uses--I think it makes the old email secondary and the new one primary? But I'm not sure.
Oh okay. As long as Civi logs once that they are RML, it doesn’t matter if
it gets overwritten, right?
Ah, I don’t think I’m relying on the silverpop group—we automatically add
data into some database fields when someone opts in for RML and that data
stays there (unless it gets overwritten by a new sign up). Totally okay to
Sorry, which bit? It seemed like the numbers you were coming up with above
made sense but I don't really understand the process of running commands on
group IDs. I don' think it matters to me if their group ID is reassigned,
but maybe I'm missing something?
May 2 2019
@TSkaff confirmed that Monday works for the banner team!
May 1 2019
Redundant (see T222220)
Cool, that all sounds great! Is there any chance we can also satisfy this
ask in T220645?
Remind me--will this be affecting payments or only Civi? If only Civi, emails can run whenever.
Apr 30 2019
I think we should drop the on_hold for mail block cases. Mail blocks mean
the user’s email client blocked us, either because the user marked our send
domain as undesirable, because the email client blacklisted our IP, or
because the email client requires whitelisting. It isn’t always indicative
of a user preference.