Page MenuHomePhabricator

Outlook/Microsoft bounced all? daily-article-l deliveries for Sept. 2
Closed, ResolvedPublic

Description

Screenshot 2021-09-01 at 22-33-26 Mailman3 - Grafana.png (660×3 px, 80 KB)

Got 1k+ bounces, nearly all from addresses using Outlook/Microsoft. I randomly checked some of the messages, they had errors like Decoding of header X-Spam-Report failed and 550 5.6.0 CAT.InvalidContent.Exception: InvalidCharsetException, Character set name (unknown-8bit) is invalid or not installed.

I saved 2 bounces in my home directory on lists1001 as invalid-charset.pck and invalid-spam.pck. sudo mailman-wrapper qfile <file>.pck can be used to dump/print the contents.

Event Timeline

fgiunchedi triaged this task as Medium priority.Sep 2 2021, 8:07 AM

I took a quick look at /root/invalid-spam.pck (needs to be readable by user list) and indeed the X-Spam-Report header is marked as encoded with unknown-8bit

X-Spam-Report: =3D?unknown-8bit?q?Spam_detection_software=3D2C_running_on_t=
he_system_=3D22lists1001=3D2Ewikimedia=3D2Eorg=3D22=3D2C?=3D
 =3D?unknown-8bit?q?_has_NOT_identified_this_incoming_email_as_spam=3D2E__T=
he_original?=3D
 =3D?unknown-8bit?q?_message_has_been_attached_to_this_so_you_can_view_it_o=
r_label?=3D
 =3D?unknown-8bit?q?_similar_future_email=3D2E__If_you_have_any_questions=
=3D2C_see?=3D
 =3D?unknown-8bit?q?_the_administrator_of_that_system_for_details=3D2E?=3D
 =3D?unknown-8bit?q?_?=3D
...

Other bits of the message like Subject are utf8 as expected

Subject: [Daily article] =3D?utf-8?q?September_2=3D3A_Indian_roller?=3D

It seems to me exchange is failing to accept the message with the unknown encoding, though other MTAs have no problem with it (?), this begs the question of why other daily-article-l submissions were fine e.g. two days ago. One possibility is that the user-agent and/or encoding to send the message has been changed?

Disclaimer: I'm looking at this in a clinic duty capacity, I won't have time next week to keep digging/investigating

Thanks for taking a look :) I don't really understand why spamassasin added the X-Spam-Report header in the first place, AIUI it's only supposed to do that if the score is higher than 4, but this one had X-Spam-Score: 2.1 (++).

Looking at the logs, this message got processed twice:

root@lists1001:/var/log# grep "61305a56.1c69fb81.ab59.12d9@mx.google.com" mail.info
Sep  2 05:00:12 lists1001 spamd[6552]: spamd: checking message <61305a56.1c69fb81.ab59.12d9@mx.google.com> for nonexistent:112
Sep  2 05:00:16 lists1001 spamd[6552]: spamd: result: . 2 - DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,LOTS_OF_MONEY,MONEY_NOHTML,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS scantime=3.6,size=6171,user=nonexistent,uid=112,required_score=4.0,rhost=127.0.0.1,raddr=127.0.0.1,rport=52758,mid=<61305a56.1c69fb81.ab59.12d9@mx.google.com>,autolearn=disabled
Sep  2 05:07:40 lists1001 spamd[12806]: spamd: checking message <61305a56.1c69fb81.ab59.12d9@mx.google.com> for nonexistent:112
Sep  2 05:07:44 lists1001 spamd[12806]: spamd: result: . -1 - DKIM_INVALID,DKIM_SIGNED,LOTS_OF_MONEY,MAILING_LIST_MULTI,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,UNPARSEABLE_RELAY scantime=3.7,size=52139,user=nonexistent,uid=112,required_score=4.0,rhost=127.0.0.1,raddr=127.0.0.1,rport=33836,mid=<61305a56.1c69fb81.ab59.12d9@mx.google.com>,autolearn=disabled

However, the day before's message only came in once:

root@lists1001:/var/log# zgrep "612f08d5.1c69fb81.eb5cc.37d8@mx.google.com" mail.info*
mail.info:Sep  1 05:00:12 lists1001 spamd[442]: spamd: checking message <612f08d5.1c69fb81.eb5cc.37d8@mx.google.com> for nonexistent:112
mail.info:Sep  1 05:00:15 lists1001 spamd[442]: spamd: result: . 0 - DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS scantime=3.6,size=5951,user=nonexistent,uid=112,required_score=4.0,rhost=127.0.0.1,raddr=127.0.0.1,rport=38880,mid=<612f08d5.1c69fb81.eb5cc.37d8@mx.google.com>,autolearn=disabled

I think we can wait again tonight to see if this happens again or if it was a freak incident. Looking in bounce.log I don't see this having affected any other lists.

One possibility is that the user-agent and/or encoding to send the message has been changed?

Just to rule this out I checked with MZ who said they made no changes to the script that sends out these emails.

Thank you all for investigating this issue. Confirming that I have not touched this script in months, honestly I'd forgotten all about it again. The source code is here: https://github.com/mzmcbride/daily-article.

Thanks for taking a look :) I don't really understand why spamassasin added the X-Spam-Report header in the first place, AIUI it's only supposed to do that if the score is higher than 4, but this one had X-Spam-Score: 2.1 (++).

Looking at the logs, this message got processed twice:

Bizarre indeed

[...]

I think we can wait again tonight to see if this happens again or if it was a freak incident. Looking in bounce.log I don't see this having affected any other lists.

Agreed, and indeed it doesn't seem to have happened today

One possibility is that the user-agent and/or encoding to send the message has been changed?

Just to rule this out I checked with MZ who said they made no changes to the script that sends out these emails.

Thank you and @MZMcBride, good to know we can exclude that!

Legoktm claimed this task.

And...hasn't happened since. Closing as resolved as just a fluke.