This usually can be done by the the list admins, I have access but rather leave turning the spamassassin or other knobs to the admins themselves.

Pinging @Aklapper & @Platonides FYI, as list admins mentioned on https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/

@A_smart_kitten: https://meta.wikimedia.org/wiki/Mailing_lists/Administration#Spam_filters mentions an X-Spam-Score header. I don't see any such header in the recent emails, so what do you expect list admins to do? It also talks about some ""Header filters" settings". Where to find them in the Postorious UI?

Perhaps only allow submissions from subscribers

That's already the case, isn't it?

and require all subscriptions to be approved?

I don't have time for that. Plus it completely destroys communication if you need a human to be around and awake to push some button to allow someone else to speak.

In T386559#10555125, @Aklapper wrote:

@A_smart_kitten: https://meta.wikimedia.org/wiki/Mailing_lists/Administration#Spam_filters mentions an X-Spam-Score header. I don't see any such header in the recent emails, so what do you expect list admins to do? It also talks about some ""Header filters" settings". Where to find them in the Postorious UI?

It should be a tab in there:

If X-Spam-Score is not being passed from exim4/postfix, that could explain it. Could be caused by the recent refactors of mail infra. I'll ping @jhathaway

In T386559#10555125, @Aklapper wrote:

@A_smart_kitten: https://meta.wikimedia.org/wiki/Mailing_lists/Administration#Spam_filters mentions an X-Spam-Score header. I don't see any such header in the recent emails, so what do you expect list admins to do? It also talks about some ""Header filters" settings". Where to find them in the Postorious UI?

I’m not personally sure, to be honest I was just pinging you as a list admin so that you were aware of this task :)

In T386559#10555205, @Ladsgroup wrote:

It should be a tab in there:

And it is, thanks. I was blind yesterday. And https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/header-matches/ has an x-spam-score entry. But that setting is not triggered without the mail header. :)

If X-Spam-Score is not being passed from exim4/postfix, that could explain it.

Maybe the mails are not passing through spamassasin?

https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/LNEA5MZV6IUKCTZPRBJ2BP2WXYHA4TA6/ seems to be another message that is missing an x-spam-score header and is also obvious spam. Where do we look/who do we ask to find out what is happening with Spamassassin?

Change #1248895 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] mailman: disable web posting

https://gerrit.wikimedia.org/r/1248895

@bd808 based on the User-Agent, User-Agent: HyperKitty on https://lists.wikimedia.org/, and looking in the logs, this message appears to have been posted from the web UI. Messages posted from the web UI are sent via SMTP to exim4, but the source IP is localhost, so our exim4 config skips spam checking. As briefly discussed, I think the only users of the web UI for posting are spammers, https://gitlab.com/mailman/hyperkitty/-/issues/264, so let's disable it.

In T386559#11683041, @jhathaway wrote:

As briefly discussed, I think the only users of the web UI for posting are spammers […]

FWIW this is not true, I have used the web UI each time I posted a message to wikitech-l.

In T386559#11683076, @A_smart_kitten wrote:

In T386559#11683041, @jhathaway wrote:

As briefly discussed, I think the only users of the web UI for posting are spammers […]

FWIW this is not true, I have used the web UI each time I posted a message to wikitech-l.

Thanks @A_smart_kitten, disabling is the easiest action, but there are probably other viable options. One unfortunate aspect of the feature, that does not seem configurable, is than anyone can post, and if they are not a subscriber they are added as one. So for spam messages, we are just adding bogus subscribers.

(To be clear, personally, I could probably try and configure my email setup to make it easier to send emails directly to wikitech-l if I needed to -- I just wanted to point out that it isn't only spammers using the web UI :))

In T386559#11683374, @A_smart_kitten wrote:

(To be clear, personally, I could probably try and configure my email setup to make it easier to send emails directly to wikitech-l if I needed to -- I just wanted to point out that it isn't only spammers using the web UI :))

Are you using the web UI because you are reading message threads via the hyperkitty archives and then deciding to respond? Generally interacting with a Mailman managed mailing list via SMTP is no different than participating in a threaded email discussion via any other SMTP backend. New threads start by addressing an email to the list's address. Threaded replies start by replying to the list address from a local message contained in the thread. There are a few fiddly details that your local mail user agent needs to attend to to make it possible for the message to be seen as a continuation of a thread (In-Reply-To:, References: headers), but that should be built-in with any reasonably full featured MUA implementation.

In T386559#11683842, @bd808 wrote:

Are you using the web UI because you are reading message threads via the hyperkitty archives and then deciding to respond?

From what I recall, the reasons why I've used the web UI may be one or both of:

• some of my personal email setup/implementation has been a bit of a nightmare (one that I'm trying to improve!), to the point where (e.g.) I'm not actually sure whether - if I replied to a wikitech-l email through my own email setup - my email would actually end up successfully getting through to the mailing list itself.
• I may have not wanted to risk accidentally revealing any personal information about myself, in case any would've been automatically added (either to the email body or to its headers) if I were to send a message/reply to wikitech-l from my own mail client.

if /message/new is the correct route, here is the count of usage from 03-05:

$ zgrep -E $'\tPOST\t.*/message/new\t' /var/log/apache2/lists.wikimedia.org-access.log.1 | sed -E 's#.*/list/([^@]*).*#\1#' | sort | uniq -c
     11 abstract-wikipedia
      1 biblio-es-l
      1 education
      1 mediawiki-api
      1 moderators-nl
      1 wikipedia-ne

I think most of this is spam, at least all the ones with sender jaririj15@gmail.com on abstract-wikipedia are spam from localhost

In T386559#11683912, @jhathaway wrote:

if /message/new is the correct route [...]

FWIW, if I'm understanding correctly -- from the <!-- Reply form --> in the source of this Hyperkitty page, it seems like /hyperkitty/list/[mailing-list]/message/[id]/reply might be a route that's used for replies via the web UI.

In T386559#11683933, @A_smart_kitten wrote:

In T386559#11683912, @jhathaway wrote:

if /message/new is the correct route [...]

FWIW, if I'm understanding correctly -- from the <!-- Reply form --> in the source of this Hyperkitty page, it seems like /hyperkitty/list/[mailing-list]/message/[id]/reply might be a route that's used for replies via the web UI.

thanks @A_smart_kitten, that route seems less active, perhaps mostly humans:

$ zgrep -E $'\tPOST\t.*/message/[^/]*/reply\t' /var/log/apache2/lists.wikimedia.org-access.log.1 | sed -E 's#.*/list/([^@]*).*#\1#' | sort | uniq -c
      1 mediawiki-api

Change #1249415 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] mailman: send web posting through Spamassassin

https://gerrit.wikimedia.org/r/1249415

Ok we have two options now:

1. disable: mailman: disable web posting
2. reroute in exim to add spam checking: mailman: send web posting through Spamassassin

Collab services, any thoughts?

Thanks for the 2 patches!

In T386559#11690217, @jhathaway wrote:

Ok we have two options now:

1. disable: mailman: disable web posting

Is web posting widely used?

1. reroute in exim to add spam checking: mailman: send web posting through Spamassassin

imho that could be the best solution, we would avoid removing an existing feature while fixing the problem.
Arbitration between the suggestions, I guess, depends on mailman's web posting feature being used or not.

I would also vote for the rerouting option. Thank you, Jesse.

Okay, the posting flow is a bit different than I understood it at first glance. A user can only use the web form if they are logged in with an account. Otherwise the buttons appear, but they are mailto links which open the users email client.

So the spam we received thus far was created by existing accounts, e.g:

wikimedia-l@lists.wikimedia.org selfish.macaw.fzac@hidingmail.com MemberRole.member None
wikitech-l@lists.wikimedia.org selfish.macaw.fzac@hidingmail.com MemberRole.member None
wikimedia-l@lists.wikimedia.org 95nicholle@dollicons.com MemberRole.member None
wikitech-l@lists.wikimedia.org 95nicholle@dollicons.com MemberRole.member Action.hold

I’m not confident that routing web form posts through Spamassassin is wise, for two reasons.

1. We lose most of the modern email spam signals, DKIM, SPF, and DMARC. As well as IP block lists. We would only be adding the content heuristics.
2. The email settings appear global to the django app, so we would be routing user confirmation emails through Spamassasin as well, which could result in automated messages being dropped.

So I think the options are:

1. Allow web posting, but block accounts which emit spam
2. Disable web posting

I think option (2) is more sustainable at present, given the current design of the feature.

I use web posting from time to time (when I'm subscribed with one email address but not the other and for various reasons I need to write a response or an email with that one) but I can live without it. No objections on my side to disabling web posting (at least for now)

We had another one right now: https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/SDQP4WYVSFFOI4DDD4QFSUD2LYRKZUY7/ I'd say let's just disable it :D

In T386559#11764020, @Ladsgroup wrote:

I'd say let's just disable it :D

+1 from me on turning off the web submission as a first step. We can announce that action and wait to hear from folks who have been disrupted somehow to decide if we need to spend more time on complicated remedies. I'm sure there will be a few folks who have been using the web UI to do things other than send spam, but hopefully we will see that they can find other frontends for posting to the lists.

Change #1248895 merged by Arnaudb:

[operations/puppet@production] mailman: disable web posting

https://gerrit.wikimedia.org/r/1248895

I asked Arnaud to merge the patch. If anyone has any objections, talk to me :)