This task tracks:
Rspam training in the same way as is currently done with Spamassassin
Mentionned in T378028: Replace Exim on VRTS servers with Postfix
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | jhathaway | T232343 Consider Postfix as MTA for our MXes (and OTRS/Mailman/Phab) | |||
| Open | None | T325394 Replace Exim with Postfix on mail servers | |||
| Open | jhathaway | T378028 Replace Exim on VRTS servers with Postfix | |||
| Resolved | ABran-WMF | T402260 Replace Spamassassin with Rspam for VRTS on Postfix |
Our current setup trains spamassassin by exporting to mails to the mbox format. This format is not supported by rspam. I'm considering how to address this issue.
I'll probably use formail with a wrapper integrated in the existing logic and avoid unnecessary changes.
Change #1251331 had a related patch set uploaded (by Arnaudb; author: Arnaudb):
[operations/puppet@production] vrts: add dual SpamAssassin and Rspamd training
I've been able to test my change on Pontoon:
root@arndb-tst-vrts-01:/home/arnaudb# train_mail_filters.py --train-spamassassin --train-rspamd --rspamd-controller localhost:11334 --rspamd-password-file /etc/vrts/rspamd-controller.password --debug 2026-03-26 15:55:02 INFO: starting training run (spool_dir=/var/spool/spam, lock_file=/run/lock/train_mail_filters.lock, train_spamassassin=True, train_rspamd=True, log_file=/var/log/train_mail_filters.log, retention_days=7, debug=True) 2026-03-26 15:55:02 DEBUG: acquired lock /run/lock/train_mail_filters.lock 2026-03-26 15:55:02 INFO: moved /var/spool/spam/spam to /var/spool/spam/spam.processing for processing 2026-03-26 15:55:02 INFO: starting spamassassin training for spam from /var/spool/spam/spam.processing 2026-03-26 15:55:02 DEBUG: running command: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/sa-learn --spam --mbox /var/spool/spam/spam.processing 2026-03-26 15:55:04 INFO: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/sa-learn --spam --mbox /var/spool/spam/spam.processing: Learned tokens from 3 message(s) (3 message(s) examined) 2026-03-26 15:55:04 DEBUG: running command: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/spamassassin --add-to-blacklist --mbox 2026-03-26 15:55:05 INFO: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/spamassassin --add-to-blacklist --mbox: 3 message(s) examined. 2026-03-26 15:55:05 ERROR: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/spamassassin --add-to-blacklist --mbox: Mar 26 15:55:04.314 [157434] warn: config: created user preferences file: /var/lib/spamassassin/.spamassassin/user_prefs 2026-03-26 15:55:05 INFO: processed /var/spool/spam/spam.processing as spam for spamassassin 2026-03-26 15:55:05 INFO: starting rspamd training for spam from /var/spool/spam/spam.processing via localhost:11334 2026-03-26 15:55:05 DEBUG: running command: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: Results for file: stdin (0.192 seconds) 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: success = true; 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: filename = "stdin"; 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: scan_time = 0.192001; 2026-03-26 15:55:05 DEBUG: running command: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: Results for file: stdin (0.016 seconds) 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: success = true; 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: filename = "stdin"; 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: scan_time = 0.016000; 2026-03-26 15:55:05 DEBUG: running command: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: Results for file: stdin (0.02 seconds) 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: success = true; 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: filename = "stdin"; 2026-03-26 15:55:05 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_spam: scan_time = 0.020000; 2026-03-26 15:55:05 INFO: processed 3 messages from /var/spool/spam/spam.processing as spam for rspamd 2026-03-26 15:55:05 INFO: finished processing spam mbox /var/spool/spam/spam.processing 2026-03-26 15:55:05 INFO: moved /var/spool/spam/ham to /var/spool/spam/ham.processing for processing 2026-03-26 15:55:05 INFO: starting spamassassin training for ham from /var/spool/spam/ham.processing 2026-03-26 15:55:05 DEBUG: running command: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/sa-learn --ham --mbox /var/spool/spam/ham.processing 2026-03-26 15:55:06 INFO: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/sa-learn --ham --mbox /var/spool/spam/ham.processing: Learned tokens from 3 message(s) (3 message(s) examined) 2026-03-26 15:55:06 DEBUG: running command: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/spamassassin --add-to-whitelist --mbox 2026-03-26 15:55:07 INFO: /bin/su - debian-spamd -s /bin/sh -c /usr/bin/spamassassin --add-to-whitelist --mbox: 3 message(s) examined. 2026-03-26 15:55:07 INFO: processed /var/spool/spam/ham.processing as ham for spamassassin 2026-03-26 15:55:07 INFO: starting rspamd training for ham from /var/spool/spam/ham.processing via localhost:11334 2026-03-26 15:55:07 DEBUG: running command: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham 2026-03-26 15:55:07 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: Results for file: stdin (0.04 seconds) 2026-03-26 15:55:07 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: success = true; 2026-03-26 15:55:07 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: filename = "stdin"; 2026-03-26 15:55:07 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: scan_time = 0.040000; 2026-03-26 15:55:07 DEBUG: running command: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: Results for file: stdin (0.028 seconds) 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: success = true; 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: filename = "stdin"; 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: scan_time = 0.028000; 2026-03-26 15:55:08 DEBUG: running command: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: Results for file: stdin (0.024 seconds) 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: success = true; 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: filename = "stdin"; 2026-03-26 15:55:08 INFO: /usr/bin/rspamc -h localhost:11334 -P <redacted> learn_ham: scan_time = 0.024000; 2026-03-26 15:55:08 INFO: processed 3 messages from /var/spool/spam/ham.processing as ham for rspamd 2026-03-26 15:55:08 INFO: finished processing ham mbox /var/spool/spam/ham.processing 2026-03-26 15:55:08 INFO: done root@arndb-tst-vrts-01:/home/arnaudb# train_mail_filters.py --train-spamassassin --train-rspamd --rspamd-controller localhost:11334 --rspamd-password-file /etc/vrts/rspamd-controller.password --debug 2026-03-26 15:57:37 INFO: starting training run (spool_dir=/var/spool/spam, lock_file=/run/lock/train_mail_filters.lock, train_spamassassin=True, train_rspamd=True, log_file=/var/log/train_mail_filters.log, retention_days=7, debug=True) 2026-03-26 15:57:37 DEBUG: acquired lock /run/lock/train_mail_filters.lock 2026-03-26 15:57:37 DEBUG: no spam source mbox found in /var/spool/spam 2026-03-26 15:57:37 DEBUG: no ham source mbox found in /var/spool/spam 2026-03-26 15:57:37 INFO: done
The new training flow keeps the existing VRTS export unchanged: vrts.TicketExport2Mbox.pl still produces mbox input in /var/spool/spam/{spam,ham}, and SpamAssassin is still trained with sa-learn --mbox and spamassassin --add-to-blacklist/--add-to-whitelist --mbox.
For Rspamd, instead of introducing an extra shell pipeline with formail, train_mail_filters.py reads the same mbox input directly, splits it message-by-message with mailbox.mbox, and feeds each message to rspamc learn_spam/learn_ham via stdin. The new script handles .processing files and per-backend completion markers (.sa_done, .rspamd_done) to prevent race conditions and support retries after partial failures.
So the new logic is on par with the legacy SpamAssassin training process, while adding Rspamd training, active-only execution, and passive-node Rspamd synchronization.
Small update after another round of Pontoon testing. A few things changed while testing the patch:
Change #1251331 merged by Arnaudb:
[operations/puppet@production] vrts: add dual SpamAssassin and Rspamd training
Following up on yesterday's merge, I created a dashboard, also containing the pre-existing VRTS dashboard's metrics. Everything seems to be running properly.
Change #1293667 had a related patch set uploaded (by Arnaudb; author: Arnaudb):
[operations/alerts@master] vrts: alerts for the new antispam pipeline
Change #1293667 merged by jenkins-bot:
[operations/alerts@master] vrts: alerts for the new antispam pipeline
Change #1294240 had a related patch set uploaded (by Arnaudb; author: Arnaudb):
[operations/alerts@master] vrts: skip pint validation on active/passive alerts
Change #1294240 merged by jenkins-bot:
[operations/alerts@master] vrts: skip pint validation on active/passive alerts