Page MenuHomePhabricator

GSoC/Outreachy proposal for Implementing HTML E-Mail support in MediaWiki
Closed, DeclinedPublic

Description

Project

Possible Mentors
Synopsis

HTML emails have a lot of advantages over plain text emails; being responsive to screen size and supporting fonts other than monospace (difficult to read for dyslexics) are some of the many pointed out by Tim Starling. In the same email, another serious issue he points out is that plain-text does not work well with bidirectional languages, since plain-text just follows visual order. In addition, certain clients fail to correctly identify links in plaintext, leading to serious inconvenience for the users.

Given the fact that MediaWiki has a very large and diverse user base, these issues really ought to be taken care of - there is no reason to continue using only plain-text emails given the benefits of HTML emails.

As of now, MediaWiki has support only for plain text email. A few extensions (such as Echo) have their own mechanism for sending out HTML emails, but core as such does not, which means any mails that core sends (such as change notification emails) cannot be sent as HTML Emails. This project plans to implement this functionality in MediaWiki's core.

Prior Art

MediaWiki’s core already has a provision for sending HTML email, as long as one is being provided to it. Some extensions using it are:

  • Extension:Echo - This extension already has HTML email support implemented. Once core has support for this, the extensions can use it instead of writing their own.
  • Extension:EmailPage - This extension sends rendered HTML pages to email addresses using PHP Mailer. This is different from what we want to do, as we do not want to be email wiki pages, we wish to convert a given text email into an html email.
  • Patches - Some patches have been submitted trying to implement support for HTML email. We should probably be working with these and extending them.
Phases​

Note: Testing and review will be done as and when the phases are being developed.

  • Phase I : The MVP
    • Feed text coming from wfMessage as is, i.e., assume the text given has no HTML tags in it.
    • Implement very basic plain text to HTML conversions:
      • Use nl2br to convert newlines to <br/> tags
      • Include links in <a> tags. Account for links being in brackets (()), in quotes (‘’ or “”) and any other format that we find people tend to mention links in plain text.
    • In order to reduce redundancy and allowing extensions to reuse/alter their existing HTML email support, a new global $wgCoreHTMLEmail could be added to allow Wiki Administrators to send plain text emails if they want to (send plain text email if $wgCoreHTMLEmail is set to false, and send multipart email if it is true) , besides $wgAllowHTMLEmail, which decides whether the Wiki allows anybody (core or extension) to send an HTML Email. To explain the different cases more clearly,
      • if ($wgAllowHTMLEmail == false) only plain text emails go
      • if ($wgAllowHTMLEmail == true and $wgCoreHTMLEmail == false) Anybody *except core* can send an HTML email.
      • if ($wgAllowHTMLEmail == true and $wgCoreHTMLEmail == true) Everybody can and core *will* send an HTML email.
    • The server will send emails to the client depending upon user preferences:
      • The user will be given a “Receive plain text email only” option in Special:Preferences. When this is ticked, the server will send only plain text emails to that specific user.
      • When the above option is unticked, the server will send a multipart email (both HTML and plain text) when both $wgAllowHTMLEmail and $wgCoreHTMLEmail are set to true.
  • Phase II : Preliminary styling
    • Integrate templates from Zurb Foundation, decide on which ones to keep, which ones to leave.
    • Include branding of any kind that might be required
      • Add the official MediaWiki logo at the beginning of the mail.
      • A default sign-off message, perhaps.
    • Ask community for design suggestions, something like a wish-list
  • Phase III : Wish-list and translations

    Some alternative implementations might exist that achieve what I am trying to in a better way. I will take some time to look around for those. For now, my proposed solution for translations is as follows:
    • When generating a key to be passed to wfMessage(), it will be assumed that if the key does not end with -html, the text is wikitext/plaintext. If the caller of the function wishes to include html in the text, the key must be appended by -html.
      • When the key sent ends with -html, we can run wfMessage($key)->exists() on it, and if it does not exist, we can use the key without the ending -html, assuming that version exists. This will be done so that even if an html version of the text exists and the caller wants to use the plaintext version, they have an option to do so. This will require discussion with the i18n team. Allowing arbitrary HTML in the messages will have security concerns that will need to be looked into and taken care of.
    • I will try to implement the necessary features that come up on the wish-list, and any other features that may be very useful, if time permits.
  • Phase IV : Wrap-up
    • Robust testing, to see if any possibility of a security breach has not been accounted for.
    • Create/Edit the required documentation pages for this functionality.
    • Announce the introduction of the functionality on wikitech-l mailing list.

Deliverables

  • The MVP: Ability to be able to send plain text as an HTML email with minimal conversions of plaintext to HTML. (Phase I)
  • Tests and Patches: They will be submitted on gerrit, and will be available for review and testing to the community.
  • Documentation: Explaining in detail how the functionality is to be used, in terms of variables, functions and classes.
Timeline
Time PeriodTask(s)
Apr 22 - May 23Community Bonding Period. Go through existing art and decide structure of the source code.
Weeks 1, 2, 3 ( May 23 - Jun 13 )Work on and finish the MVP (inclusive of the tests) and finalise decisions with regard to the structure of the functionality (methods, classes, global variables, etc). Begin to gather a community wishlist. (Phase I)
Week 4 ( Jun 14 - Jun 20 )Discussing and integrating templates agreed upon, and gathering community review on them. (Phase II)
Week 5 ( Jun 21 - Jun 27 )Mid-term Evaluation Make sure MVP is complete, functioning and deployed. Check that Weekly reports are all up to date. Continue with work on Phase II.
Week 6, 7 ( Jun 28 - Jul 12 )Get community review and design team suggestions on branding of the email, and look at any other possibly better templates. (Phase II)
Weeks 8, 9 ( Jul 13 - Jul 26 )Work on translation of content, discuss the best solutions for our scenario with the i18n team. Security loop-holes created by allowing HTML in the text provided will be worked on fixed. (Phase III)
Weeks 10, 11 ( Jul 27 - Aug 9 )Robust testing, security review, work on refactoring the code to make it better and more efficient. (Phase IV)
Week 12 ( Aug 10 - Aug 16 )(Attempt to) Implement any additional features deemed necessary/desirable according to the community wish-list and reviews. Fix minor issues, file bugs. (Phase IV)
Week 13 ( Aug 17 - Aug 23 )Pencils down Make sure functionality is complete, functioning and deployed. Check that weekly reports are up to date. Make sure any known bugs are either solved or filed. Send mail on wikitech-l announcing the functionality. (Phase IV)
Timeline Details
Week 0 (Community Bonding Period)
  • Go through the existing art for implementing HTML support for Emails, and discuss whether these should be built upon or a different approach would be needed.
  • Decide whether implementing only one function would do the job, or a class with internal functions would be needed.
  • Research on templates other than those provided by the Zurb Foundation, and whether they can and should be used instead.
  • Look for other extensions that have their own implementation for HTML Emails and investigate their workflow for the same, and discuss how they are to be adapted to, if at all.
Week 1
  • Implement a function transformContentToHtml (probably as a static method of a class, so HtmlEmail::transformContentToHtml or EmailNotification::transformContentToHtml) that accepts a string with the plain-text version of the body of the email and returns an associative array with both html and text fields that can then be sent to UserMailer::send.
    • Basic operations would be performed on the text, such as nl2br and putting links into <a> tags
    • Example:
HtmlEmail::transformContentToHtml("Dear User,\n\nWelcome to https://www.mediawiki.org"`);

would return

Dear User,<br />
<br />
Welcome to <a href="https://www.mediawiki.org">https://www.mediawiki.org</a>

Week 2

  • Change transformContentToHtml to accept an i18n key instead of a string directly.
  • Implement the $wgCoreHTMLEmail and add checks for it in transformContentToHtml.
  • Investigate changing Special:Preferences to add a user preference for receiving only plain-text versions of email.

Week 3

  • Add the user preference to Special:Preferences
  • Add some preliminary unit tests in order to complete an MVP:
    • Test that the transformContentToHtml gives the correct output for an obvious input
    • Test the auto-linking part of the function by attempting to feed in complex URLs
    • Check that different combinations of $wgCoreHTMLEmail and $wgAllowHTMLEmail give the expected output

Week 4

  • Look into the Zurb Foundation templates, decide which ones are relevant and whether any changes are required to the existing templates
  • Select the appropriate templates and make any required modifications
  • Investigate how the HTML templates will be put into the repository and then used by the code (plain HTML files? includeing PHP files with variables? Some basic templating engine?)

Week 5

  • Mid-Term Evaluation to-dos :
    • Check whether weekly reports are up to date
    • Make sure MVP is complete, functioning and deployed
    • Any known bugs in MVP are either solved or filed
  • Implement the chosen templating mechanism into the transform method, including passing of the template name (or some other specification) in the call to the method
  • Update tests to include the checking the templating system
  • Look into the benefits of caching template processing to increase efficiency

Week 6

  • Look into branding, start a wish-list sort of thing for the community (and the design team) to suggest/request inclusion of certain things: logos, some kind of default signature
  • Look for other options for templates, consider whether implementing and maintaining a custom set of templates for Wikimedia is necessary/required/desired

Week 7

  • Include elements required from branding and update tests as consensus is reached on them
  • Finalise reviews on whether implementing and maintaining a custom set of templates for Wikimedia is necessary/required/desired
  • Put basic architecture to implement the custom set of templates if necessary/required.

Week 8

  • Start discussions on better i18n and a security review
  • Discuss methods of including HTML in messages without breaking any existing systems (one of the possible disadvantages mentioned in the original task) with the i18n team
  • Simultaneously discuss security issues with the current implementation as well as with the proposed i18n implementation. Simple htmlentities or possibly something full-fledged like HTMLPurifier

Week 9

  • Implement agreed-upon i18n solutions
  • Fix any identified security loopholes, add any advanced sanitization that was agreed upon.

Week 10

  • Add more robust tests (since the feature-set will be near completion)
  • Check if there are any incompatibilities with the way this has been implemented in core and the way extensions want it
  • Refactoring code to make it more readable or to improve efficiency, if the need is felt (readability and efficiency will be ensured as the code is written, but it may so happen that this is not heeded as much attention at all times)

Week 11

  • Start documenting the new functionality on mediawiki.org
  • Attempt to modify extensions that are currently sending HTML email
  • Implement caching of template processing if its importance is agreed upon

Week 12

  • Buffer week. Complete any pending tasks or attempt to implement any extra features that were suggested during any of the reviews.
  • File bugs found, attempt to fix most of them.

Week 13

  • Pencils Down to-dos:
    • Make sure functionality is complete, functioning and deployed
    • Check that weekly reports are up to date
    • Make sure any known bugs are either solved or filed
    • Code submission to Google
  • Complete documentation of the functionality.
  • Announce new functionality on wikitech-l and mediawiki-l.

Profile

Name: Smriti Singh
Email: smritis.31095@gmail.com
IRC handle (freenode): galorefitz
Internet Presence : MediaWiki user page
Location:
I'll be visiting 2 places this summer, India (+05:30 GMT), and Kuwait (+03:00 GMT)

Typical Working Hours:

  1. May-July: 6-8 hours a day, between 6:30 a.m - 8:30 p.m (GMT) (at least 40 hours a week)
  2. August : 5-6 hours a day, between 3:30 a.m - 11:30 p.m (GMT) (at least 40 hours a week)
Communication

I will be submitting reports weekly on a phabricator task, tracking the progress of the project, on Phabricator. I will be available on #wikimedia-dev and #wikimedia-tech on IRC (freenode) during my working hours, so I can be reached there. I will also be responding on the relevant phabricator tasks. The source code will be pushed to Gerrit, and will be accessible, viewable and reviewable there.

Contributions

Microtasks: T31856 T48109 T62434
My contributions to the community can be viewed here and here. I haven't contributed to any other FOSS organisations yet, but plan to in the future.

About Me

Education: In progress. I am a Computer Science Student (currently in my third year), studying at the International Institute of Information Technology (IIIT), Hyderabad, India.

How did you hear about this program? From senior year students in college.

Why MediaWiki? I started contributing to MediaWiki in May, 2015. The people here were quick to respond, very encouraging, and amazingly helpful. Contributing, seeing my work make a difference to the community, felt great. Not many communities have all of the above, and this is what encouraged me to choose MediaWiki.

Why this project? HTML emails have a lot of advantages over plain text emails; being responsive to screen size, supporting fonts other than monospace (difficult to read for dyslexics), supports bidirectional languages, unlike plaintext, which only supports visual order. Certain clients fail to correctly identify links in plaintext, leading to serious inconvenience for the users. Given the fact that MediaWiki has a very large and diverse user base, these issues really ought to be taken care of - there is no reason to continue using only plain-text emails given the benefits of HTML emails.

Additional Information (as mandated by Gnome)

Do you meet the eligibility requirements outlined here ? - Yes.

Preferred pronoun - she

Prior Commitments -
May-July, 2016 : None.
11th August, 2016 - 23rd August, 2016 : College, will take approximately 25 hours a week

Course details
From 11th August, 2016 - 23rd August, 2016 : My college has a system of electives for every semester, and so I am not sure right now what courses I’ll be taking and how many credits they’ll be. The total number of credits, though is 20 (normal, full-time course load is of 20-24 credits). For more information, please refer to https://www.iiit.ac.in/academics/curriculum/undergraduate/BTech-CSE (Year IV, Semester I)

Related Objects

Event Timeline

Galorefitz updated the task description. (Show Details)Mar 20 2016, 11:35 PM

@01tonythomas When I was submitting my proposal to Google, they asked me for a proposal tag. Which tag will this project come under, operations or quality assurance?

Galorefitz updated the task description. (Show Details)Mar 21 2016, 8:25 AM

@01tonythomas When I was submitting my proposal to Google, they asked me for a proposal tag. Which tag will this project come under, operations or quality assurance?

Oops. Thank you for bringing this one up. I have added few more tags 'notifications', 'general' and 'web'. So probably 'notifications' ?

Oops. Thank you for bringing this one up. I have added few more tags 'notifications', 'general' and 'web'. So probably 'notifications' ?

@01tonythomas yeah, I guess so.
@Parent5446 @01tonythomas any suggestions, clarifications, or comments on the proposal?

IMPORTANT: The deadline for submitting your proposal for Outreachy'12 falls in less than 24 hours at Mar 22 2016, 07:00 pm UTC. . Please make sure that you have a copy of your proposal in the application system at outreachy.gnome.org beforehand, and make sure it has answers to the specific questions in https://wiki.gnome.org/Outreachy#Application_Form. You are encouraged to apply for both GSoC and Outreachy, in case you are eligible.
IMPORTANT: The deadline for submitting your proposal to Google Summer of Code 2016 application system at GSoC application system falls in roughly 24 hours at Mar 25 2016, 19:00 UTC. Please make sure that you have a pdf copy of your proposal in the application system beforehand, to avoid last minute confusions. Remember to relate your Phabricator task and associate 2 mentors in the proposal description, so that it gets easy for review. Past the deadline, you should only make changes limited to fixing typos, or incorporating feedback's. Good Luck, and check out the micro-tasks!

Since this proposal is valid for both GSoC and Outreachy, we are considering it for GSoC only, as per our rules at https://www.mediawiki.org/wiki/Google_Summer_of_Code/Admins#Proposal_submission_phase

Hi @Galorefitz, your proposal looks good. Please elaborate the timeline a bit more though. With very clear, bulleted goals for each week or fortnight.

Galorefitz renamed this task from GSoC/Outreachy Proposal for Implementing HTML E-Mail support in MediaWiki to GSoC Proposal for Implementing HTML E-Mail support in MediaWiki.Mar 24 2016, 6:57 PM
Galorefitz updated the task description. (Show Details)Mar 24 2016, 7:03 PM
Sumit added a comment.Mar 25 2016, 4:18 PM

@Galorefitz a gentle reminder to submit your proposal as a final pdf at https://summerofcode.withgoogle.com before the application deadline, i.e, 25th March 17:00 UTC.

Galorefitz renamed this task from GSoC Proposal for Implementing HTML E-Mail support in MediaWiki to GSoC/Outreachy proposal for Implementing HTML E-Mail support in MediaWiki.Mar 25 2016, 6:25 PM
Galorefitz updated the task description. (Show Details)

@Galorefitz a gentle reminder to submit your proposal as a final pdf at https://summerofcode.withgoogle.com before the application deadline, i.e, 25th March 17:00 UTC.

Done, thank you for the reminder! :)

Galorefitz updated the task description. (Show Details)Mar 25 2016, 6:28 PM
Galorefitz added a subscriber: Qgil.

And in light of

Just a side comment about

Since this proposal is valid for both GSoC and Outreachy, we are considering it for GSoC only, as per our rules at https://www.mediawiki.org/wiki/Google_Summer_of_Code/Admins#Proposal_submission_phase

I think the idea still is that candidates eligible to both programs should submit their applications to both, knowing that in the case of acceptance in both the priority is to get a slot in GSoC, not Outreachy. The idea to apply in both is still valid, because we should avoid a scenario where a student is not getting a GSoC project (i.e. because we get 5 slots and there are 7 good candidates) and then she is not getting an Outreachy project either, because we advised her not to apply.

You're right, my mistake. I was counting on the fact that we usually get the slots we ask for in GSoC.

and

Just a side comment about

Since this proposal is valid for both GSoC and Outreachy, we are considering it for GSoC only, as per our rules at https://www.mediawiki.org/wiki/Google_Summer_of_Code/Admins#Proposal_submission_phase

I think the idea still is that candidates eligible to both programs should submit their applications to both, knowing that in the case of acceptance in both the priority is to get a slot in GSoC, not Outreachy. The idea to apply in both is still valid, because we should avoid a scenario where a student is not getting a GSoC project (i.e. because we get 5 slots and there are 7 good candidates) and then she is not getting an Outreachy project either, because we advised her not to apply.

Edited the page to reflect the idea to apply to both. With respect to the current rounds, all our candidates who are eligible for both programs have applied for both. So the remark above I think applies to considering the proposals for GSoC during evaluation for this round.

I am adding the tag back. :)

Aashaka removed a subscriber: Aashaka.Mar 25 2016, 6:57 PM
Galorefitz updated the task description. (Show Details)Apr 5 2016, 2:41 PM
Sumit added a comment.Apr 22 2016, 7:50 PM

Congratulations @Galorefitz for getting selected for this project in GSoC 2016! Wish you a good luck with it. You can start discussing ideas and get to speed with the project as the Community Bonding period has started.

Welcome to Google-Summer-of-Code (2016) and to the Community Bonding period! Happy to have you here, and this should be crucial time to create important decisions regarding how your project should take shape during this two month internship period. You can find information about Community bonding period, from our Life of successful doc here. To make sure everything go as per planned, please follow the instructions in T133647 and create the 'Community Bonding Evaluation for $project' task as a subtask of your proposal task. Please note that all further tasks you create for evaluation and GSoC organization purpose should be subtasks of your proposal, and not the parent task - lets reduce the notification count. In case you are stuck, feel free to comment below T133647 or open up a conpherence task with the mentors and org admins. You can find example tasks in the task description of T133647

Niharika removed a subscriber: Niharika.Apr 26 2016, 5:04 PM

@Galorefitz : Please create the blocking community bonding period tasks and others as per my request above :)

Once you have made that, please add it as a blocker to T133647 ( You can edit T133647 and add your community bonding evaluation task as a blocker ).

Qgil removed a subscriber: Qgil.May 2 2016, 8:13 AM
Ljonka added a subscriber: Ljonka.Jun 17 2016, 10:59 AM
Osnard added a subscriber: Osnard.Jun 22 2016, 12:50 PM
01tonythomas closed this task as Declined.Sep 14 2016, 6:41 AM

End of project timeline. Closing it down. Thank you for all the efforts.

Outcome of this 2016 round projects at https://www.mediawiki.org/wiki/Google_Summer_of_Code_past_projects#2016