Page MenuHomePhabricator

Engagement emails: technical investigation
Closed, ResolvedPublic

Description

As the team is in the very early stages of this project, we want to get some initial technical thoughts on what is easy and what is difficult around sending emails. We know that Mediawiki already sends emails in a few ways, and we should get a sense of whether we would be changing an existing service, adding a new one, or something else.

Here are the very basic existing requirements around emails, which will change in the future, as the project becomes better defined:

  • Include rich content.
  • Be sent based on rules that incorporate welcome survey responses, edit history, and time.
  • Allow people to unsubscribe - see T215743 and T215672
  • Allow us to track opens and clicks.
  • Allow us to track when they are replied to
  • Have a noreply@ from address

For some examples of emails that we might want to send, see these slides (images to published on wiki during January 2019).

Completing this specific task means writing down thoughts either here or in some other document.

A note on tracking: we're not sure yet what sort of tracking we will want to do. It's critical that anything we track complies with our privacy policy and the needs of our users, while still allowing us the data to improve the feature. We will learn more by talking to other teams in the foundation.

  • Other standard metrics that are tracked by ESPs
    • Sent - Number of emails that moved through the Email Service Provider’s sending mail server
    • Delivered - Number of Sent emails that were not rejected by a receiving server
    • Bounced - Sent emails that were rejected. There are two types of bounces:
      • (a) Hard bounces are permanently rejected emails (e.g., due to invalid email address or the sender’s server has been blocked);
      • (b) Soft bounces are temporarily rejected emails (e.g., due to recipient mailbox being full, server being down, message exceeding size limits. Repeated soft bounces → hard bounces.)
    • Marked as Spam/Abuse rate - Number of recipients who reported the email as spam, divided by delivered emails

Event Timeline

MMiller_WMF renamed this task from Engagement Emails: technical investigation to Engagement emails: technical investigation.Jan 17 2019, 9:03 PM
RHo updated the task description. (Show Details)Jan 18 2019, 1:07 PM

Here are the very basic existing requirements around emails, which will change in the future, as the project becomes better defined:

Include rich content.

Sure, we can do HTML email, with the caveat as Roan as mentioned several times that doing this in a device and email agnostic way is quite difficult. It would probably make sense to pick a handful of clients (Gmail web interface on desktop, Gmail app interface on Android/iOS). The more simple the layout, the more likely it is that it will function well across viewport sizes, devices, and email clients.

Be sent based on rules that incorporate welcome survey responses, edit history, and time.

This sounds fine.

Allow people to unsubscribe somehow.

We could provide a link to "Unsubscribe me from these emails" at the bottom of the email. How we want to implement that is a product question. We could do any of the following:

  • Provide a preference or set of preferences in the notifications section of prefs (Special:Preferences#mw-prefsection-echo) where users can opt-in to the emails (are we also doing Echo notices?) for each of the engagement emails separately, or for all of them together
  • Provide a link for "Unsubscribe" at the bottom.
    • The unsubscribe button could immediately unsubscribe the user from just that engagement email type or for all engagement emails. The user could be directed to a page where we inform them how to re-subscribe, or we could take them to Special:Notifications and provide a notice that they are unsubscribed.
    • The button could also just drop the user into the Notifications section of the preferences, where they'd then have to figure out what to do next on their own.

Allow us to track opens and clicks.

Email marketing platforms generate links per email, such that if abc@blah.com is sent an engagement email with a link to https://cs.wikipedia.org/wiki/Special:UserHomepage, that links appears to them as https://sketchyemailmarketingplatformcompany.com/link/adfg2f3f23, and then that email marketing platform knows that a visit to adfg2f3f23 means that abc@blah.com clicked the link tailored to them.

Similarly, opens can be tracked by embedding an image; when the email is opened (Gmail web UI by default will load images, while other email clients usually ask you what you want to do) the server hosting the image can associated a load of that image with a mail open. Like the link tracking, this can be done on a per user basis.

The big question is if it's important to have the open/click info associated with an individual user (with all the privacy and legal implications of that) or whether it would suffice for us to know e.g. of the 100 engagement emails sent in the last week, 20 of them were opened; of the 20 opened, there were 5 clicks on the call to action button, 4 clicks on the unsubscribe button, etc.

Trying to keep track of clicks/opens in a general way (rather than a per user basis) will be a lot less engineering time.

In both cases, I imagine we'd need a hidden Special page (or two pages) for handling link clicks and loading the image to detect opens.

Allow us to track when they are replied to
Have a noreply@ from address

Based on scanning T202329, it seems like if we set the reply-to to something unique for our project (like noreply-engagement-cs), then we can track this.

@RHo -- what is important about having a noreply address? Is it just that the name of the address itself deters people from replying into a black hole? What if we were able to direct replies to somewhere useful instead of a black hole?

@RHo -- what is important about having a noreply address? Is it just that the name of the address itself deters people from replying into a black hole? What if we were able to direct replies to somewhere useful instead of a black hole?

Using emails to post on wiki? That would be clever.

RHo added a comment.Jan 23 2019, 3:15 PM

@RHo -- what is important about having a noreply address? Is it just that the name of the address itself deters people from replying into a black hole? What if we were able to direct replies to somewhere useful instead of a black hole?

Yes exactly, it's important so that people are not replying into a black hole as currently occurs. I added it as a preference over setting a specific reply-able email address since thinking about scaleability, it would reduce the added overhead of including a facility for someone to reply to these replies. But we can defer this if it is easier to set a specific proper reply email address and we are not concerned about volume for now.

Be sent based on rules that incorporate welcome survey responses, edit history, and time.

"time" as in "24h after their last interaction with the wiki" is perhaps the only thing that we don't have proper infrastructure for at the moment. We could run a job every hour or so and do it there. We also have to keep track of who should receive and has received the emails. We could (ab)use the preference system one more time or build something specifically for this purpose.

Echo sends daily or weekly digests of notifications when requested (it's a user preference), using a cron job that runs every day at 00:00 UTC.

As for open tracking, I'm happy to hear that image loading works in desktop GMail. As Kosta says, we could create a hidden special page that serves empty (or 1x1px) images and logs open events.

Click tracking could be done with a hidden special page that logs the click and then redirects to the intended target, or for links that go directly to our own special page (the newcomer dashboard), just with a URL parameter.

As for unsubscribing, our current notification emails don't contain an immediate unsubscribe link, but instead link to the user's notification preferences. If we're going to add an immediate unsubscribe link to these engagement emails, I think we should also consider adding them to notification emails. In both cases, they could be implemented by linking to a special page, but that would only work if the user is logged in. If the user is not logged in, we either have to build a URL-token-based auto-login feature (which would need security review and likely not pass muster), or put a signed token in the link URL that's trusted for unsubscribing the user but nothing else.

I think the most challenging/annoying pieces of this are going to be:

  • Obtaining the various pieces of data that appear in the mock-ups of some of the emails (like the total number of pageviews in the last week/month)
  • Setting up a cron job (or something else) and other infrastructure that we need so that we can schedule emails to be sent at a certain time
  • Writing HTML+CSS for these emails that works within the narrow-ish set of features supported by email clients; as Kosta said, we should pick a handful of popular clients and aim at those, but it's still going to be frustrating from time to time
  • Getting a noreply address set up (requires coordination with SRE; the existing sender address is wiki@)
RHo updated the task description. (Show Details)Feb 10 2019, 11:24 PM
RHo added a comment.Feb 10 2019, 11:27 PM

As for unsubscribing, our current notification emails don't contain an immediate unsubscribe link, but instead link to the user's notification preferences. If we're going to add an immediate unsubscribe link to these engagement emails, I think we should also consider adding them to notification emails. In both cases, they could be implemented by linking to a special page, but that would only work if the user is logged in. If the user is not logged in, we either have to build a URL-token-based auto-login feature (which would need security review and likely not pass muster), or put a signed token in the link URL that's trusted for unsubscribing the user but nothing else.

Hi @Catrope + @MMiller_WMF - linking to this task T215743 as IANAL but think having an ability to unsubscribe without being logged in will likely be required.

MMiller_WMF closed this task as Resolved.Feb 27 2019, 5:13 PM

This task is complete because the engineers have weighed in, and the salient points have been captured in the tasks around the actual coding.