Page MenuHomePhabricator

Random codes in mail contents, when using sparkpost
Closed, ResolvedPublic

Description

It seems that some mails contain duplicated header, which result in random codes at the beginning of mail content.

Event Timeline

Zoglun created this task.Jan 25 2019, 2:26 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 25 2019, 2:26 AM

This error might related to the sparkpost tracking function, which is enabled by default. It add codes/small gif to each mail to get engagement tracking ability, and change each url link to get click statics.

If this is the case. May I pleased to request 3 more options to disable the tracking function?
Here is some github discussion about tracking enable/disable: https://github.com/laravel/ideas/issues/31

$wgSparkpostOpen_tracking = true/false;
$wgSparkpostClick_tracking = true/false;
$wgSparkpostTransactional = true/false;

This is possible but I can't reproduce since I don't have the necessary resources to. But I'll make an attempt to solve the problem but I have a few questions;

  • What should be the default for these 3 configurable vars, false? Since we we don't want these tracking infos into the emails?
  • Could the names be $wgSparkpostOpenTracking, $wgSparkpostClickTracking and $wgSparkpostTransactional?

Let me know your thoughts!

Hi D3r1ck01,
Default these three vars to true would be better, as the default tracking behavior of SparkPost.
$wgSparkpostOpenTracking, $wgSparkpostClickTracking and $wgSparkpostTransactional are better. That's good!

Here is the error mail content for reference.

--=_53414f2242c834c162d46e1c5bbe1840
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset=UTF-8

=E6=9C=89=E4=BA=BA=E5=9C=A8=E2=80=AA=E5=B0=91=E5=A5=B3=E5=89=8D=E7=BA=BF:20=
19=E5=B9=B4=E5=86=AC=E5=AD=A3=E6=B4=BB=E5=8A=A8=E2=80=9C=E5=BC=82=E6=9E=84=
=E4=BD=93=E2=80=9D=E2=80=AC=E4=B8=AD=E5=8A=A0=E5=85=A5=E4=BA=86=E2=80=AA=E5=
=B0=91=E5=A5=B3=E5=89=8D=E7=BA=BF:=E4=BE=A6=E5=AF=9F=E8=80=85=E2=80=AC=E7=
=9A=84=E9=93=BE=E6=8E=A5=E3=80=82



=E6=9F=A5=E7=9C=8B=E9=A1=B5=E9=9D=A2=EF=BC=9A <http://post.spmailtechn.com/f/a/l_90IdYb9v4BwvrB9vjXAw~~/AADaJQA~/RgReMWXNP0QdaHR0cHM6Ly96aC5tb2VnaXJsLm9yZy8lRTUlQj1XA3NwY0IKAA3N4E5cD_1jpVIPbWVyY3Vzc3JAcXEuY29tWAQAAAAA
0%91%E5%A5%B3%E5%89%8D%E7%BA%BF:2019%E5%B9%B4%E5%86%AC%E5%AD%A3%E6%B4%BB%E5=
%8A%A8%E2%80%9C%E5%BC%82%E6%9E%84%E4%BD%93%E2%80%9D?markasread=3D2269040>

=E6=89=80=E6=9C=89=E9=93=BE=E5=85=A5=E9=A1=B5=E9=9D=A2=EF=BC=9A <http://post.spmailtechn.com/f/a/0LS_Va90cAWqQAwPdMbsrA~~/AADaJQA~/RgReMWXNP0QLaHR0cHM6Ly96aD1XA3NwY0IKAA3N4E5cD_1jpVIPbWVyY3Vzc3JAcXEuY29tWAQAAAAA
=2Emoegirl.org/Special:%E9%93%BE%E5%85%A5%E9%A1%B5%E9%9D%A2/%E5%B0%91%E5%A5=
%B3%E5%89%8D%E7%BA%BF:%E4%BE%A6%E5%AF%9F%E8%80%85>

=E6=9F=A5=E7=9C=8B=E6=9B=B4=E6=94=B9=EF=BC=9A <http://post.spmailtechn.com/f/a/3tBPkRoVXjPLqd_ydymq0w~~/AADaJQA~/RgReMWXNP0QdaHR0cHM6Ly96aC5tb2VnaXJsLm9yZy9pbmRleD1XA3NwY0IKAA3N4E5cD_1jpVIPbWVyY3Vzc3JAcXEuY29tWAQAAAAA
=2Ephp?title=3D%E5%B0%91%E5%A5%B3%E5%89%8D%E7%BA%BF:2019%E5%B9%B4%E5%86%AC%=
E5%AD%A3%E6%B4%BB%E5%8A%A8%E2%80%9C%E5%BC%82%E6%9E%84%E4%BD%93%E2%80%9D&dif=
f=3D2321872&oldid=3Dprev>

--

=E8=A6=81=E6=8E=A7=E5=88=B6=E6=88=91=E4=BB=AC=E5=90=91=E6=82=A8=E5=8F=91=E9=
=80=81=E7=9A=84=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=EF=BC=8C=E8=AF=B7=E6=A3=
=80=E6=9F=A5=E6=82=A8=E7=9A=84=E5=8F=82=E6=95=B0=E8=AE=BE=E7=BD=AE=EF=BC=9A
http://post.spmailtechn.com/f/a/DiPGFmjlw9Dv1vfdaefM3Q~~/AADaJQA~/RgReMWXNP0RMaHR0cHM6Ly96aC5tb2VnaXJsLm9yZy9TcGVjaWFsOiVFNSU4RiU4MiVFNiU5NSVCMCVFOCVBRSVCRSVFNyVCRCVBRSNtdy1wcmVmPVcDc3BjQgoADc3gTlwP_WOlUg9tZXJjdXNzckBxcS5jb21YBAAAAAA~
section-echo
--=_53414f2242c834c162d46e1c5bbe1840
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html; charset=UTF-8

<html><head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DUTF-8" /=
>
        <meta name=3D"viewport" content=3D"width=3Ddevice-width, initial-sc=
ale=3D1.0" />
<style>
@media only screen and (max-width: 480px){
table[id=3D"email-container"]{max-width:600px !important; width:100% !im=
portant;}
}
</style>
<base href=3D"http://post.spmailtechn.com/f/a/U7iKuUdw3HLP2-XJyjIhkg~~/AADaJQA~/RgReMWXNP0QXaHR0cHM6Ly96aC5tb2VnaXJsLm9yZyJXA3NwY0IKAA3N4E5cD_1jpVIPbWVyY3Vzc3JAcXEuY29tWAQAAAAA>
</head><body>
<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0" width=3D"100%" alig=
n=3D"center" lang=3D"zh-Hans" dir=3D"ltr">
<tr>
<td bgcolor=3D"#EAECF0"><center>
<br /><br />
<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0" width=3D"600" id=
=3D"email-container">
<tr>
<td bgcolor=3D"#FFFFFF" width=3D"5%">&nbsp;</td>
<td bgcolor=3D"#FFFFFF" width=3D"10%">&nbsp;</td>
<td bgcolor=3D"#FFFFFF" width=3D"80%" style=3D"line-height:40px;">&nbsp=
;</td>
<td bgcolor=3D"#FFFFFF" width=3D"5%">&nbsp;</td>
</tr><tr>
<td bgcolor=3D"#FFFFFF" rowspan=3D"2">&nbsp;</td>
<td bgcolor=3D"#FFFFFF" align=3D"center" valign=3D"top" rowspan=3D"2"><=
img src=3D"http://post.spmailtechn.com/f/a/kEWIsVZ6cg2e4kQbN7SO6Q~~/AADaJQA~/RgReMWXNP0RBaHR0cHM6Ly96aC5tb2VnaXJsLm9yZy9sb2FkLnBocD9tb2R1bGVzPTNEZXh0LmVjaG8uZW1haWxpY29ucyZhbT1XA3NwY0IKAA3N4E5cD_1jpVIPbWVyY3Vzc3JAcXEuY29tWAQAAAAA
p;image=3Dlinked&amp;lang=3Dzh-hans&amp;format=3Drasterized" alt=3D"" heigh=
t=3D"30" width=3D"30"></td>
<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-family: Arial, Hel=
vetica, sans-serif; font-size:13px; line-height:20px; color:#72777D;">=E6=
=9C=89=E4=BA=BA=E5=9C=A8<strong>=E2=80=AA=E5=B0=91=E5=A5=B3=E5=89=8D=E7=BA=
=BF:2019=E5=B9=B4=E5=86=AC=E5=AD=A3=E6=B4=BB=E5=8A=A8=E2=80=9C=E5=BC=82=E6=
=9E=84=E4=BD=93=E2=80=9D=E2=80=AC</strong>=E4=B8=AD=E5=8A=A0=E5=85=A5=E4=BA=
=86<strong>=E2=80=AA=E5=B0=91=E5=A5=B3=E5=89=8D=E7=BA=BF:=E4=BE=A6=E5=AF=9F=
=E8=80=85=E2=80=AC</strong>=E7=9A=84=E9=93=BE=E6=8E=A5=E3=80=82</td>
<td bgcolor=3D"#FFFFFF" rowspan=3D"2">&nbsp;</td>
</tr><tr>
<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-family: Arial, Hel=
vetica, sans-serif; line-height: 20px; font-weight: 600;">
<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0">
<tr>
<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-family: Arial, =
Helvetica, sans-serif; padding-top: 8px; font-size:13px; font-weight: bold;=
 color: #54595D;">
=09
</td>
</tr>
</table>
<table cellspacing=3D"0" cellpadding=3D"0" border=3D"0">
<tr>
<td bgcolor=3D"#FFFFFF" align=3D"left" style=3D"font-family: Arial, =
Helvetica, sans-serif; font-size:14px; padding-top: 25px;">
<a href=3D"http://post.spmailtechn.com/f/a/zWEeWQTTMA8M8zE69APM2A~~/AADaJQA~/RgReMWXNP0Q5aHR0cHM6Ly96aC5tb2VnaXJsLm9yZy8lRTUlQjAlOTElRTUlQTUlQjMlRTUlODklOEQlRTclQkE9VwNzcGNCCgANzeBOXA_9Y6VSD21lcmN1c3NyQHFxLmNvbVgEAAAAAA~~
%BF:2019%E5%B9%B4%E5%86%AC%E5%AD%A3%E6%B4%BB%E5%8A%A8%E2%80%9C%E5%BC%82%E6%=
9E%84%E4%BD%93%E2%80%9D?markasread=3D2269040" style=3D"cursor:pointer; text=
-align:center; text-decoration:none; padding:.45em 0.6em .45em; color:#FFF;=
 background:#36C; font-family: Arial, Helvetica, sans-serif;font-size: 13px=
;">=E6=9F=A5=E7=9C=8B=E9=A1=B5=E9=9D=A2</a>&nbsp;&nbsp;<a href=3D"http://post.spmailtechn.com/f/a/ODa2aWmJSltkEj9OzmKqug~~/AADaJQA~/RgReMWXNP0QKaHR0cHM6Ly96PVcDc3BjQgoADc3gTlwP_WOlUg9tZXJjdXNzckBxcS5jb21YBAAAAAA~
h.moegirl.org/Special:%E9%93%BE%E5%85%A5%E9%A1%B5%E9%9D%A2/%E5%B0%91%E5%A5%=
B3%E5%89%8D%E7%BA%BF:%E4%BE%A6%E5%AF%9F%E8%80%85" style=3D"text-decoration:=
 none;font-size: 10px;font-family: Arial, Helvetica, sans-serif; color: #72=
777D;">=E6=89=80=E6=9C=89=E9=93=BE=E5=85=A5=E9=A1=B5=E9=9D=A2</a>&nbsp;&nbs=
p;<a href=3D"http://post.spmailtechn.com/f/a/3NseWqNh9DUT1ZXJJhXlnA~~/AADaJQA~/RgReMWXNP0Q_aHR0cHM6Ly96aC5tb2VnaXJsLm9yZy9pbmRleC5waHA_dGl0bGU9M0QlRTUlQjAlOTElRTUlQTUlQjMlRTU9VwNzcGNCCgANzeBOXA_9Y6VSD21lcmN1c3NyQHFxLmNvbVgEAAAAAA~~
%89%8D%E7%BA%BF:2019%E5%B9%B4%E5%86%AC%E5%AD%A3%E6%B4%BB%E5%8A%A8%E2%80%9C%=
E5%BC%82%E6%9E%84%E4%BD%93%E2%80%9D&amp;diff=3D2321872&amp;oldid=3Dprev" st=
yle=3D"text-decoration: none;font-size: 10px;font-family: Arial, Helvetica,=
 sans-serif; color: #72777D;">=E6=9F=A5=E7=9C=8B=E6=9B=B4=E6=94=B9</a>
</td>
</tr>
</table>
</td>
</tr><tr>
<td bgcolor=3D"#FFFFFF">&nbsp;</td>
<td bgcolor=3D"#FFFFFF">&nbsp;</td>
<td bgcolor=3D"#FFFFFF" style=3D"line-height:40px;">&nbsp;</td>
<td bgcolor=3D"#FFFFFF">&nbsp;</td>
</tr><tr>
<td>&nbsp;</td>
<td>&nbsp;</td>
<td align=3D"left" style=3D"font-family: Arial, Helvetica, sans-serif; =
font-size:10px; line-height:13px; color:#72777D; padding:10px 20px;"><br />
=E8=A6=81=E6=8E=A7=E5=88=B6=E6=88=91=E4=BB=AC=E7=BB=99=E6=82=A8=E5=8F=
=91=E9=80=81=E7=9A=84=E7=94=B5=E5=AD=90=E9=82=AE=E4=BB=B6=EF=BC=8C=E8=AF=B7=
<a href=3D"http://post.spmailtechn.com/f/a/bFcqY2rqCncQTtvdkxYZBg~~/AADaJQA~/RgReMWXNP0RBaHR0cHM6Ly96aC5tb2VnaXJsLm9yZy9TcGVjaWFsOiVFNSU4RiU4MiVFNiU5NSVCMCVFOCVBRSVCRSVFNyVCRD1XA3NwY0IKAA3N4E5cD_1jpVIPbWVyY3Vzc3JAcXEuY29tWAQAAAAA
%AE#mw-prefsection-echo" style=3D"text-decoration: none; color: #36C;">=E6=
=A3=80=E6=9F=A5=E6=82=A8=E7=9A=84=E5=8F=82=E6=95=B0=E8=AE=BE=E7=BD=AE</a>=
=E3=80=82
<br /><br />
</td>
<td>&nbsp;</td>
</tr><tr>
<td colspan=3D"4">&nbsp;</td>
</tr>
</table>
<br><br></center>
</td>
</tr>
</table>
</body></html>
--=_53414f2242c834c162d46e1c5bbe1840--

Change 487009 had a related patch set uploaded (by D3r1ck01; owner: Derick Alangi):
[mediawiki/extensions/SparkPost@master] Provide means to avoid SparkPost's tracking option

https://gerrit.wikimedia.org/r/487009

D3r1ck01 triaged this task as Normal priority.Jan 29 2019, 8:25 AM
D3r1ck01 added a project: User-D3r1ck01.
D3r1ck01 moved this task from Backlog to Doing [WIP] on the User-D3r1ck01 board.

Change 487009 merged by jenkins-bot:
[mediawiki/extensions/SparkPost@master] Provide means to avoid SparkPost's tracking option

https://gerrit.wikimedia.org/r/487009

@Zoglun, I've added that feature into the extension. Please test it and let me know if it's working as expected so we can resolve this task or if you run into any issues, let me know and I'll try to fix ASAP. Thanks!

Also, could you let me know if this extension is already running on a wiki instance that is used by many users? So that I can pay more attention and respond faster to issues raised. :)

@Zoglun, I've added that feature into the extension. Please test it and let me know if it's working as expected so we can resolve this task or if you run into any issues, let me know and I'll try to fix ASAP. Thanks!
Also, could you let me know if this extension is already running on a wiki instance that is used by many users? So that I can pay more attention and respond faster to issues raised. :)

Cool! I will test it out ASAP.

The extension SparkPost is used by Moegirlpedia, about 600 users use mail function each day, and roughly 700 mails were sent by this extensions per day. There are more wikis planing to use it, as long as the garbled email problem solved.

Oh nice! Looking forward to your feedback then about this new patch! Thanks for the update!

Zoglun added a comment.EditedJan 29 2019, 10:37 AM

With SparkPost commit 4f241dc6c585ae6303a675f09fb8d3f761c60d5c & MW 1.31,

$wgSparkpostClickTracking & $wgSparkpostOpenTracking dosen't seems to work.

We still get converted link address like: http://post.spmailtechn.com/f/a/XXXXXXXXXXXXXXXXXXXXXXXXX when both of them set to false.


The error email were successfully reproduced. article edit/change notification email got their Content-Transfer-Encoding changed from Content-Transfer-Encoding: quoted-printable to Content-Transfer-Encoding: 7bit.

Therefore all the Chinese/Japanese/Korean words become random code due to this error. We are still trying to figure out what cause this encoding change.

Thanks for the feedback @Zoglun, and I'll investigate and fix this. Also, does it mean $wgSparkpostTransactional works fine? I'll really wish to plead for the use of your API Key just for the purpose of development & testing. Could you send to me privately via email? (alangi.derick@wikipedia.de) so I'm sure that things are in good shape before I ask you to test? thanks!

Change 487014 had a related patch set uploaded (by D3r1ck01; owner: Derick Alangi):
[mediawiki/extensions/SparkPost@master] Follow-up on I81a2b8315ba6c2b267fa4b241ce7bab05724c2bc

https://gerrit.wikimedia.org/r/487014

Ohhh, also, I think I know what is wrong, I'm making a quick follow up patch and merge while you test and also while I wait for the API Key to be using for testing for upcoming patches. Thanks!

Oh, I just realized that I am not sure what this transactional option mean in SparkPost. There were not much information about this option. I don't know whether it work or not. Sorry for that.

I have sent an API code to your email address.

Ohhh, also, I think I know what is wrong, I'm making a quick follow up patch and merge while you test and also while I wait for the API Key to be using for testing for upcoming patches. Thanks!

Made a follow up patch: Not (exactly) certain about the patch as I've not tested yet but you can try let's see how it goes! Let me know your feedback. Thanks!

I've received the email you sent, thanks!

Change 487014 merged by jenkins-bot:
[mediawiki/extensions/SparkPost@master] Follow-up on I81a2b8315ba6c2b267fa4b241ce7bab05724c2bc

https://gerrit.wikimedia.org/r/487014

I03157d57249c942e7a1bffc953aeca8226e2db9f Does not disable link tracking convert in our test.

Change 487015 had a related patch set uploaded (by D3r1ck01; owner: Derick Alangi):
[mediawiki/extensions/SparkPost@master] Follow up on I03157d57249c942e7a1bffc953aeca8226e2db9f

https://gerrit.wikimedia.org/r/487015

Change 487015 merged by jenkins-bot:
[mediawiki/extensions/SparkPost@master] Follow up on I03157d57249c942e7a1bffc953aeca8226e2db9f

https://gerrit.wikimedia.org/r/487015

@Zoglun, I was able to track down the issue. I've made another patch to fix it. I've merged it, let me know if that works now. Thanks :)

Still no effect. :)

Please, just in case, could you pull on master? I'm pretty sure things should be find now? Just double check let's see. :)

Zoglun added a comment.EditedJan 29 2019, 11:56 AM

Did you get normal link address without convert? If you do get url without convert, then there might be problem with php cache?


git branch

  • master

git show
commit 3decd5b5d241311d5e90714a3dbdf2ca195c0732
Author: Derick Alangi <alangiderick@gmail.com>
Date: Tue Jan 29 12:32:02 2019 +0100

Follow up on I03157d57249c942e7a1bffc953aeca8226e2db9f

Bug: T214664
Change-Id: Id889536d8a0d9002c3be42e40b8c856bd2183620

diff --git a/extension.json b/extension.json
index 9703d91..c8c9b88 100644

  • a/extension.json

+++ b/extension.json
@@ -26,9 +26,9 @@

"load_composer_autoloader": true,
"config": {
        "SparkPostAPIKey": "",
  • "SparkpostClickTracking": true,
  • "SparkpostOpenTracking": true,
  • "SparkpostTransactional": true

+ "SparkPostClickTracking": true,
+ "SparkPostOpenTracking": true,
+ "SparkPostTransactional": true

},

mailed-by: spmailtechn.com
(0 minutes ago)
test http://post.spmailtechn.com/f/a/hPwOM...........................

So I guess this could now be an issue with the format, let me fix this and make another follow up :), I guess the follow up will fix it. Hang on :)

Change 487026 had a related patch set uploaded (by D3r1ck01; owner: Derick Alangi):
[mediawiki/extensions/SparkPost@master] Follow up on Id889536d8a0d900

https://gerrit.wikimedia.org/r/487026

Change 487026 merged by jenkins-bot:
[mediawiki/extensions/SparkPost@master] Follow up on Id889536d8a0d900

https://gerrit.wikimedia.org/r/487026

Please could you test again, I've made another patch to fix this issue. Looking forward to your feedback :)

Zoglun closed this task as Resolved.EditedJan 29 2019, 12:45 PM

Finally WORKED!!!

It turns out to be my mistake, I wrote the wgSparkPostClickTracking as wgSparkpostClickTracking, which miss the capital P in it! (Didn't notice your change in 3decd5b5d241311d5e90714a3dbdf2ca195c0732 .) I am sorry for such entry level mistake, sincerely.

Another good news, per our test case, the garbled Chinese/Japanese/Korean email problem disappear when ClickTracking disabled. Yeah~

I have marked this task as resolved for now. If there were further production test with garbled content, we will re-open this task.

@D3r1ck01 Thank you!

Okay cool! Nice to hear that, meaning I can revert the last patch as it's just same as previous but using a different approach. I'll prefer to use the previous rather than the last patch. Let me revert the latest patch :)

Nice to hear garbled problem has gone away, nice!

D3r1ck01 added a comment.EditedJan 29 2019, 12:55 PM

You're welcome anytime! Ops, sorry I didn't update you about the change in names to the config variables, used "P" instead of "p". Sorry, I should have told you!

So for benefit of doubt, I've reverted the latest change per this patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/SparkPost/+/487029. Please could you update your master and make sure everything is still working before we can finally finally say this has been close and resolved for good. The patch before the lastest uses some nice approach to set the options with a call to ->setOptions() rather using the options key in the post request.

Let me know if things didn't break so I can be at peace with myself :)

Interesting. Your last change before revert actually made something work.
with vriables:

$wgSparkPostOpenTracking = false;
$wgSparkPostClickTracking = false;
$wgSparkPostTransactional = true;

Change-Id: Ib801b7e2428524ffba467a081bcb666c4cfb7da5

test http://post.spmailtechn.com/f/a/Hb7FkeF..............

Before revert:

Change-Id: I08c4c48449ab35663872f1762483ad743958489b

test https://ja.moeaaa.org/ccc

Hmmm.... So are you suggesting we maintain the last change before revert or things are fine now?

The extension work fine before, but not after the revert.

I like the nice approach of setOptions, not sure why it doesn’t function as expected.

Okay, let me revert the revert then :D

@Zoglun, revert has been reverted. Let me know if everything is fine now. I'll like to bump this version of the extension to 1.1 too :). Confirm if everything is fine let me do that in an upcoming patch!

The newest version work well. ;)

Perfect perfect perfect! Do you have an IRC handle so we can chat there, phabricator is kinda slow but it works though, just wondering if you're available on IRC.

I'll be bumping this version of the extension to 1.1 now! :)