Page MenuHomePhabricator

Encrypt emails with GnuPG (GPG)
Open, LowPublic

Description

There should be a way to encrypt emails sent by MediaWiki. (See recent Facebook announcement of a similar feature.) The user should be able to provide a GPG public key and MediaWiki would encrypt the email with that key, and possibly sign it with some key that belongs to the operator, to prove that the email indeed originated from the wiki.

There are several features that could benefit:

  • messages from other users via Special:Emailuser (probably the highest value)
  • system notfications (for password reset links this provides some added security against a powerful opponent; for watchlist notifications it improves privacy)
  • Echo notifications - not sure they are used for anything right now that's particularly private, but they might in the future. Also important for private wikis, especially if something like T90067 gets implemented, but page names might be considered private information as well.

MVP

  • handle a single GPG key by providing a text box in user preferences where the user can paste their key
  • provide basic instructions, probably by linking to something like (1) (2) (3)
  • verify that the key is valid and not a private key
  • add a checkbox to enable/disable encryption
  • encrypt user->user mails and Echo mails

(Some further ideas that are not in scope for the MVP: expose the key via API, allow setting up a sitewide key and use that to sign, refactor email handling so there is an appropriately high-level hook that all mails got thtough (so we can encrypt non-Echo system mails), do the encryption on client side, do not double-encrypt already encrypted messages, send a test email with a verification link and only enable if the user could click on it (Facebook does this, and it seems like a good idea to prevent accidents), detailed documentation about setting up email clients or using Mailvelope-style extensions, keyserver support, keybase.io integration.)

Implementation plan:

  • figure out whether the PECL gnupg library is usable. The Debian package was last updated in 2013, but it's supposed to use GPGME which in turn is a wrapper that makes calls to the gpg executable, so the version of the PECL library should not matter that much. For Wikimedia it would need to be recompiled with HHVM though.
  • create a standalone library (which can be reused by the extensions consuming the GPG keys) for one more level of wrapping, either around the PECL library or the command-line gpg tool directly. Unfortunately both of those require using a keyring. We probably don't want to leave around public keys (which might be secret) in /var/www/.gnupg/pubring.gpg of random machines; we also don't want to break other extensions using GPG by permanently changing the home directory. So we need to do something like: create temp dir -> set GNUPGHOME environment variable to temp dir -> run GPG commands -> reset GNUPGHOME and delete temp dir (use a scoped callback or something similar to make sure this happens). This kind of sucks but should be doable.
  • create Extension:GPGMail (name bikeshedding welcome)
  • add the textarea and the checkbox to user preferences
  • on save, call gnupg_import + gnupg-keyinfo to verify the key (the second might not be necessary)
  • add the User objects of the sender and the target user as extra parameters to the EmailUser and EmailUserCC hooks, which are right now too low-level to be of any use
  • add a similar hook to Echo; maybe in EchoNotificationController::formatNotification
  • use the hooks and encrypt the mails.

(Ended up creating a new pair of hooks instead that affect every outgoing email.)

Status

Current version is suitable as MVP for small wikis but not for Wikimedia as there are multiple problems with storing the key in user options:

  • they are verified every time they are loaded, but key verification is expensive and should only be done on change
  • they are included into every page and a typical key is around 1K
  • management interface is poor (e.g. should display key metadata)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
In T12453#1393106, @Tgr wrote:

Do we use TLS for sending those emails? That would be the lower hanging fruit since it effects much more users. (According to Facebook's post about STARTTLS almost all receiving servers support it nowadays.)

That's T101451

That's T101451

That one seems to be about Wikimedia mailing lists, not the mails sent by MediaWiki itself.

I was thinking about how public key management should work. Should that be an extension of its own, or partof the GPG mailer extension? I can think of a few other use cases for public keys (proof of identity, client-side encryption of messages, interaction with SSH-login-based systems like git or OpenStack) but none of that seems very plausible.

I wonder if it makes sense to do the encryption in the browser; that way the plaintext of the message never passes through the Wikimedia servers. openpgpjs is not nearly as mature as one would like a PGP library to be, but apparently it has went through some audits at least...

I'm taking this as a personal goal for Q1. The key management part has been split to T107642.

Tgr raised the priority of this task from Low to Medium.Aug 1 2015, 2:11 AM

There are definitely pitfalls with doing crypto in js. I'll try to hunt
down a paper that summarizes the issues.

They said, I like the concept. Our forcing https will ensure the message
can't be intercepted by an eavesdropper, but it's a nice property for us to
not be able to decrypt it either.

This assumes we're honest and don't add a site script that sends another
copy of the email scraped from the text box. But unless we subvert our own
deployment process, there would be an audit trail of the script being added
to the site, so maybe not too crazy to assume that.

Tgr renamed this task from Cipher/encrypt email with GnuPG (GPG) to Encrypt emails with GnuPG (GPG).Aug 11 2015, 2:02 AM
Tgr updated the task description. (Show Details)

A site script can at least be detected by the person who is being eavesdropped on, with server-side encryption in comes down to trust. It's always nice if users can trust but verify. Also, getting malicious code on our app servers unnoticed is probably easier for an attacker than to make those servers send malicious code to the client. (And if they can send malicious javascript, that would defeat the encryption either way.)

Whether there is a JS OpenPGP library that's as battle-tested as the GNU one is the tough question, but Google is apparently working on one; that sounds promising.

Anomie pointed me to SecurePoll_GpgCrypt as an example of using GPG via shell commands.

Change 241542 had a related patch set uploaded (by Gergő Tisza):
[WIP] Add user parameter to EmailUser and EmailUserCC

https://gerrit.wikimedia.org/r/241542

There are two widespread formats for encrypting email: traditional (also called PGP/inline; just replace the text of the email with the GPG encrypted (ASCII-armored) version of the text) and PGP/MIME which is more complex and needs to be done by feeding the final email body (after multipart separators and attachments are added) to mimegpg. PGP/MIME is better specified and offers more robust protection, but has less client support (link, other link).

For now I'm going for traditional as that seems to have more support, and also easier to integrate. For PGP/MIME EmailUser is too early, the right place to hook in would probably be AlternateUserMailer where the body is already MIME-formatted, but that hook does not allow changing the text.

Change 241573 had a related patch set uploaded (by Gergő Tisza):
Add a FormatEchoEmail hook for final changes to email content

https://gerrit.wikimedia.org/r/241573

Change 242000 had a related patch set uploaded (by Gergő Tisza):
MVP

https://gerrit.wikimedia.org/r/242000

Change 242001 had a related patch set uploaded (by Gergő Tisza):
Encrypt Echo notifications via PGP/inline

https://gerrit.wikimedia.org/r/242001

Change 242008 had a related patch set uploaded (by Gergő Tisza):
Add gpgmail role

https://gerrit.wikimedia.org/r/242008

Change 242791 had a related patch set uploaded (by Gergő Tisza):
Add UserMailerTransformContent and UserMailerSplitTo hooks

https://gerrit.wikimedia.org/r/242791

In T12453#1679650, @Tgr wrote:

Not necessarily useful, since that's talking about signing rather than encryption. Anything not in green in the chart may mean "doesn't work" as far as encryption goes.

There's also no indication as to how old that data is, but given the "Last-Modified: Sat, 06 Feb 2010 00:54:09 GMT" header served with the page it's at least 5.5 years old.

other link

I'm not seeing any real client analysis there, just assertion.

For PGP/MIME EmailUser is too early, the right place to hook in would probably be AlternateUserMailer where the body is already MIME-formatted, but that hook does not allow changing the text.

Your new UserMailerTransformContent hook could possibly handle it if it could also change the headers. Or just throw in a UserMailerTransformMessage hook.

Your new UserMailerTransformContent hook could possibly handle it if it could also change the headers. Or just throw in a UserMailerTransformMessage hook.

PGP/MIME processing needs to happen late in the process. You do the normal MIME transformation, encrypt the whole result (MIME headers + encoded, possibly multipart-formatted body) and than use that as the body of a new message with some special MIME headers and parts.

That means the hook call needs to happen after MIME encoding is done, and needs to be reimplemented in all alternate mailers otherwise encryption might be silently omitted when mail is sent via the AlternateUserMailer hook. (I tried to write the extension in such a way that any problem results in no mail at all rather than an unencrypted mail. If you specifically request encryption and we send cleartext anyway, that seems potentially dangerous.) For alternate mailers that will typically mean replacing a single SomeMailer::send($headers, $body) command with low-level messing around since they do not normally allow that level of control.

Also I would avoid having a homegrown PGP/MIME implementation if possible, but I haven't found any good alternative. mimegpg seems to be the most mature tool but you need to install a whole mail server to get it. There is a Python script and a PHP implementation but neither looks inviting.

So while I think both formats should be supported eventually PGP/MIME looks complicated enough that I would rather focus initially on inline PGP working well, unless you think it's just too bad a format to be used at all. (I would appreciate input on that as I don't know much about email encryption in practice; but it seems to be widely used without anyone really being angry about it. And I believe the concerns about manipulating the boundaries only apply to clearsigned messages.)

I for one would be happy with your simple solution.

In T12453#1695440, @Tgr wrote:

That means the hook call needs to happen after MIME encoding is done,

Or the hook would do the MIME encoding of the $body array itself, and manipulate the headers such that what it returns in $body comes out correct. Although that's probably not the best way of doing things.

and needs to be reimplemented in all alternate mailers otherwise encryption might be silently omitted when mail is sent via the AlternateUserMailer hook.

How so? AlternateUserMailer gets $headers, $to, $from, $subject, and $body, where $headers have the MIME headers already and $body is the MIME-encoded body. Run UserMailerTransformMessage with the same parameters just before AlternateUserMailer.

Also I would avoid having a homegrown PGP/MIME implementation if possible,

NIH is a problem, but this is taking it too far in the opposite direction.

  • Remove the existing Content-* headers from $headers and prepend them (and a blank line) to $body.
  • Encrypt $body like you're already doing, no change there.
  • Plug $encryptedBody into the MIME container below, that's your output $body.
    • $boundary is easy, just generate random strings of 30-40 alphanumeric characters and prefix "=_" until strpos( $encryptedBody, $boundary ) === false. Chances are you'll get it in the first try, since the ASCII-armoring is base64 which doesn't use underscores at all.
    • You could add some Content-Description and Content-Disposition headers if you want, like the one library you link to, but it's probably not necessary.
Maybe a preamble here, like "This is an OpenPGP/MIME encrypted message (RFC 4880 and 3156)" in the lib you link.

--$boundary
Content-Type: application/pgp-encrypted

Version: 1

--$boundary
Content-Type: application/octet-stream

$encryptedBody

--$boundary--
  • Set the new Content-Type header, multipart/encrypted; protocol="application/pgp-encrypted"; boundary="$boundary"

So while I think both formats should be supported eventually PGP/MIME looks complicated enough that I would rather focus initially on inline PGP working well, unless you think it's just too bad a format to be used at all. (I would appreciate input on that as I don't know much about email encryption in practice; but it seems to be widely used without anyone really being angry about it.

It's just nice to be compliant with the RFCs. Ask @RobLa about that ;)

And I believe the concerns about manipulating the boundaries only apply to clearsigned messages.)

Which concerns are those?

How so? AlternateUserMailer gets $headers, $to, $from, $subject, and $body, where $headers have the MIME headers already and $body is the MIME-encoded body. Run UserMailerTransformMessage with the same parameters just before AlternateUserMailer.

Uh, you are right. I must have confused it with the UserMailerChangeReturnPath hook. I'll give it a try then.

NIH is a problem, but this is taking it too far in the opposite direction.

Well, implementing PGP/MIME is not hard in itself (that's the shortest RFC not involving pigeons that I have seen) but the real hard work is catching all the possible errors that come from mail clients interpreting the RFC slightly differently. But that point is moot since there doesn't seem to be any good external tool to rely on.

Which concerns are those?

I read something about that, but now I can only find this which does not seem to be any less problematic for encryption. I probably misunderstood.

Added PGP/MIME. Also tested both with Mailvelope, the results are somewhat mixed:

  • inline PGP works well with plaintext emails
  • Mailvelope cannot open HTML emails with inline encryption (Error: Unknown ASCII armor type)
  • PGP/MIME with plaintext results in the MIME headers appearing in the body cleartext
  • PGP/MIME with HTML works nicely

FWIW the email sources are in P2151.

Testing with mutt:

  • PGP/MIME with plaintext is detected and automatically decrypts.
  • PGP/MIME with html is detected, and automatically decrypts before sending to the configured HTML viewer when viewing the HTML part.
  • Inline with plaintext isn't detected, but can be decrypted with the right command.
  • Inline with HTML isn't detected. Messing with the commands can get it to decrypt the plaintext part, but the HTML viewer is given the encrypted text to display.

Change 242791 merged by jenkins-bot:
Add UserMailerTransformX and UserMailerSplitTo hooks

https://gerrit.wikimedia.org/r/242791

Change 242001 abandoned by Gergő Tisza:
Encrypt Echo notifications via PGP/inline

Reason:
Not needed, ended up using different hooks.

https://gerrit.wikimedia.org/r/242001

Change 241542 abandoned by Gergő Tisza:
Do not reuse text parameter between EmailUser and EmailUserCC

Reason:
Not needed, I'm using I4c3a018110173c3b5d52a753fdcbec397b590ced instead.

I think the patch is valid and can be reopened if someone has a use case for it.

https://gerrit.wikimedia.org/r/241542

Change 241573 abandoned by Gergő Tisza:
Add a FormatEchoEmail hook for final changes to email content

Reason:
Not needed, I used I4c3a018110173c3b5d52a753fdcbec397b590ced instead.

https://gerrit.wikimedia.org/r/241573

Tgr changed the task status from Open to Stalled.Oct 8 2015, 10:16 PM
Tgr updated the task description. (Show Details)

What should we do if the PGP key expires?

I see three main options:
a. Encrypt the message with the outdated key anyway
b. Treat it as not having any key set and send the email in plain text
c. Make it a hard error and do not allow sending any mail

I'm inclined to go for (a), adding a preamble stating that the key used is expired and they should update it on their profile ASAP.

Aklapper changed the task status from Stalled to Open.Mar 22 2022, 1:08 AM
Aklapper lowered the priority of this task from Medium to Low.

(The previous comments don't explain who or what (task?) exactly this task is stalled on ("If a report is waiting for further input (e.g. from its reporter or a third party) and can currently not be acted on"). Hence resetting task status, as tasks should not be stalled (and then potentially forgotten) for years for unclear reasons.)

Tgr removed Tgr as the assignee of this task.Aug 23 2022, 6:03 AM

Not stalled per se, but would need more work. The extension works but isn't production-ready, would have to be improved and taken through the extension deployment process.