From 54e4c1c2acd96abcabfbfed6db9b75a6908a3741 Mon Sep 17 00:00:00 2001 From: Christoph Haas Date: Sat, 18 Oct 2025 22:59:18 +0200 Subject: [PATCH] finish the spam page --- public/imapsieve/imapsieve-learn-spam.svg | 3 + .../docs/ispmail-trixie/300-catching-spam.mdx | 252 ++++++++---------- 2 files changed, 119 insertions(+), 136 deletions(-) create mode 100644 public/imapsieve/imapsieve-learn-spam.svg diff --git a/public/imapsieve/imapsieve-learn-spam.svg b/public/imapsieve/imapsieve-learn-spam.svg new file mode 100644 index 0000000..e5b14a6 --- /dev/null +++ b/public/imapsieve/imapsieve-learn-spam.svg @@ -0,0 +1,3 @@ + + +
mailbox Junk {
sieve_script spam {
type = before
cause = copy
path = /etc/dovecot/sieve/learn-spam.sieve
}
}
require ["vnd.dovecot.pipe", "copy", "imapsieve"];
pipe :copy "rspamd-learn-spam.sh";
#!/bin/sh
exec /usr/bin/rspamc learn_spam
IMAP sieve
Sieve
Shell
Page 1
https://webmail.example.org
Inbox
Junk
Trash
The blue pill for your hardest night. Buy now.
MOVE
From: wienermeds@gmail.com
Watch this folder
\ No newline at end of file diff --git a/src/content/docs/ispmail-trixie/300-catching-spam.mdx b/src/content/docs/ispmail-trixie/300-catching-spam.mdx index 6a7f8c4..efce033 100644 --- a/src/content/docs/ispmail-trixie/300-catching-spam.mdx +++ b/src/content/docs/ispmail-trixie/300-catching-spam.mdx @@ -68,7 +68,7 @@ to complain about it to the alleged sender. Trust me: the sender you see on a sp it. You would complain to the wrong person which is called [backscatter](). -## Score metrics +## Spam scores rspamd will however not reject all spam email. It computes a score that tells how likely a certain email is spam. You can tell it which scores you would accept, flag as spam or make the incoming email get rejected. Rspamd checks incoming @@ -189,9 +189,6 @@ sieve_script spam-to-junk-folder { path = /etc/dovecot/sieve/spam-to-junk-folder.sieve } -# Uncomment this line to get more verbose logs on sieve handling -# log_debug=category=sieve - # Enable the execution of Sieve rules when Postfix sends an email to Dovecot over LMTP protocol lmtp { mail_plugins { @@ -214,7 +211,6 @@ systemctl reload dovecot Now we need to create that Sieve script (`/etc/dovecot/sieve/spam-to-junk-folder.sieve`) that is run on each delivery. Its job is to check if the `X-Spam: yes` header is present. If it is, the email is filed into the user's `Junk` folder. -Dovecot can only understand _compiled_ Sieve files so we call `sievec` to make it machine-readable. ```sh # Create the directory for Sieve files @@ -229,9 +225,6 @@ if header :contains "X-Spam" "Yes" { stop; } EOF - -# Compile the .sieve file into a .svbin file -sievec /etc/dovecot/sieve/spam-to-junk-folder.sieve ``` Let's give it a test using Swaks. This time we impersonate Postfix and inject an email with an `X-Spam: yes` header @@ -272,7 +265,7 @@ journalctl -eu dovecot The alleged spam email has been moved to the _Junk_ folder just like we wanted. -## Enable auto-learning +## Automatic spam training One of rspamd’s features is analyzing word patterns using probability theory. That functionality is contained in its [statistical module](https://rspamd.com/doc/configuration/statistic.html). (Yes, the name is misleading.) Essentially @@ -325,7 +318,7 @@ The defaults for auto-learning are: See [rspamd's documentation](https://docs.rspamd.com/configuration/statistic/#statistics-configuration) if you want to fine-tune that. -``` +```sh rspamc stat ``` @@ -369,17 +362,32 @@ users_enabled = true; ## Learning from user actions -Now we are getting to something really cool. Imagine that you receive a spam email into your inbox that rspamd did not -detect properly. Sure, you can move it to your _Junk_ folder. But that will not improve the detection rate for that kind -of spam. But we can fix that. +Now we are getting to something really cool. Imagine you receive a spam email in your inbox that Rspamd didn’t catch. +You could just move it to your Junk folder — but that wouldn’t help Rspamd learn to recognize that kind of spam in the +future. Luckily, we can change that. -Let’s tell Dovecot that moving emails into the Junk folder teaches rspamd instantly that the email is **spam**. And if -the email is moved out of the _Junk_ folder, then learn it as **ham**. That can be done using another _sieve_ script. -Sieve script not only apply when an email is delivered. Thanks to Dovecot's +Let’s tell Dovecot that moving emails **into** the Junk folder teaches rspamd instantly that the email is **spam**. And +if the email is moved **out of** the Junk folder, then learn it as **ham**. That can be done using another _sieve_ +script. Sieve scripts not only apply when an email is delivered. Thanks to Dovecot's [IMAPSieve](https://doc.dovecot.org/2.4.1/core/plugins/imap_sieve.html#imapsieve-plugin-imap-sieve/) plugin, such scripts can also be triggered if a user moves a mail between folders. -Add another new configuration file to enable and configure that plugin: +The setup requires three parts: + +1. Tap into the IMAP connection to know when an email is moved out of or into the Junk folder. +2. Create Sieve scripts that call Shell scripts. +3. Create Shell scripts that actually call `rspamd learn_ham` or `rspamd learn_spam`. + +Or as a diagram: + +Diagram showing how to learn ham/spam when moving mails + +Create a new Dovecot configuration file (`99-ispmail-imapsieve.conf`) to enable the required functionality and add +triggers on the Junk folder: ```sh cat > /etc/dovecot/conf.d/99-ispmail-imapsieve.conf << 'EOF' @@ -400,6 +408,9 @@ sieve_global_extensions { vnd.dovecot.pipe = yes } +# Where to look for Sieve scripts that use the Pipe functionality +sieve_pipe_bin_dir = /etc/dovecot/sieve + # Moved into Junk? -> Learn as spam. mailbox Junk { sieve_script spam { @@ -428,7 +439,7 @@ The first rule tells Dovecot to run a Sieve script at `/etc/dovecot/sieve/learn The second rule sets the other way. Whenever an email is moved **out** of the "Junk" folder to any other folder, then the `/etc/dovecot/sieve/learn-ham.sieve` Sieve script is called. -Let's create both scripts: +Let's create both Sieve scripts: ```sh # Create spam learning script @@ -443,7 +454,7 @@ require ["vnd.dovecot.pipe", "copy", "imapsieve", "variables"]; pipe :copy "rspamd-learn-ham.sh"; EOF -# Compile both script into machine-readable format +# Compile the Sieve scripts sievec /etc/dovecot/sieve/learn-spam.sieve sievec /etc/dovecot/sieve/learn-ham.sieve @@ -452,97 +463,67 @@ chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin} chown vmail:vmail /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin} ``` -And the last step is to create the simple shell scripts that do the actual spam/ham training. The first file is -`/etc/dovecot/sieve/rspamd-learn-spam.sh` which contains: +By the way: the `sievec` comment compiles the `*.sieve` file into `*.svbin` files that Dovecot can work with. If you are +not doing this step, Dovecot will do that during runtime. However then you need to make sure that Dovecot has proper +write permissions to the target directory. The `sievec` command is also useful because it scans a `*.sieve` file for +errors. So if your changes do not seem to work it might be worth compiling them and looking for errors. -``` +And the last step is to create the simple shell scripts that do the actual spam/ham training: + +```sh +# Create the shell script for learning spam +cat > /etc/dovecot/sieve/rspamd-learn-spam.sh << 'EOF' #!/bin/sh +# Receives an email from Dovecot's Sieve script and pipe it into rspamc exec /usr/bin/rspamc learn_spam -``` +EOF -That looks simple, doesn’t it? Nothing more is actually needed. The spam email is piped to this script and rspamd learns -it as a spam email and adjusts its spam detection database accordingly. - -The second script teaches ham and is called `/etc/dovecot/sieve/rspamd-learn-ham.sh`. Make it contain: - -``` +# Create the shell script for learning ham +cat > /etc/dovecot/sieve/rspamd-learn-ham.sh << 'EOF' #!/bin/sh +# Receives an email from Dovecot's Sieve script and pipe it into rspamc exec /usr/bin/rspamc learn_ham -``` +EOF -These two shell scripts must be made executable: - -``` +# Fix permissions of the shell scripts chmod u=rwx,go= /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh chown vmail:vmail /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh ``` -I hope you haven’t lost your mind yet. It’s really just a chain of things to happen. Let’s reiterate how this process -works: +I am sure you are eager to try it out. Of course we don't blindly believe that it works. We want to see it do its job. +Usually you will not see much unless you increase the log level to _debug_. Let's do that: -1. a user moves a spam email into their "Junk" folder -2. Dovecot realizes that this triggers the Sieve rule "imapsieve_mailbox1" so it calls the Sieve - script /etc/dovecot/sieve/learn-spam.sieve (in fact the \*.svbin version of the script) -3. Sieve will take the email and send ("pipe") it to the executable rspamd-learn-spam.sh shell script -4. the script in turn runs the email through the "/usr/bin/rspamc learn_spam" command +```sh +cat > /etc/dovecot/conf.d/99-ispmail-sieve-debug.conf << 'EOF' +# Enable detailed logging of Sieve scripts +log_debug=category=sieve +EOF -This works equally for the other way or learning ham emails of course. +# Restart Dovecot +systemctl reload dovecot -I am sure you are eager to try it out. However to see that it actually works I suggest you edit -the /etc/dovecot/conf.d/10-logging.conf file and set "mail_debug=yes". That will add a lot more detail to the -/var/log/mail.log file but on a busy server may also lead to headaches. 🙂 - -Restart Dovecot… - -``` -systemctl restart dovecot +# Follow Dovecot's logs +journalctl -fu dovecot ``` -…and watch the /var/log/mail.log file… +Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to John's +Junk folder. The mail log will show a lot of things that are going on. You will see a whole screen full of things that +are going on. Some relevant lines: ``` -tail -f /var/log/mail.log +imap(john@example.org)<1737478>: Debug: Mailbox Junk: imapsieve: storage spam: file: Using Sieve script path: /etc/dovecot/sieve/learn-spam.sieve +imap(john@example.org)<1737478>: Debug: Mailbox Junk: imapsieve: storage spam: file: script 'learn-spam': Opened from 'spam' +imap(john@example.org)<1737478>: Debug: sieve: Opening script 1 of 1 from 'spam/learn-spam' +imap(john@example.org)<1737478>: Debug: sieve: Executing script from '/etc/dovecot/sieve/learn-spam.svbin' +imap(john@example.org)<1737478>: Debug: sieve: action pipe: running program: rspamd-learn-spam.sh +imap(john@example.org)<1737478>: Debug: sieve: uid=19: Finalize pipe action (status=ok, action_status=ok, commit_status=ok, pre-commit=yes) +imap(john@example.org)<1737478>: Debug: sieve: execute exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (1737484): Connected to program +imap(john@example.org)<1737478>: Debug: sieve: uid=19: pipe action: piped message to program 'rspamd-learn-spam.sh' ``` -Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to your -Junk folder. The mail log will show a lot of things that are going on. I tried to compact the output so that you better -understand that it worked: +Something similar should happen when you drag the email back from the Junk folder to the Inbox. -``` -imapsieve: Static mailbox rule [1]: mailbox=`Junk' from=`*' causes=(COPY) => before=`file:/etc/dovecot/sieve/learn-spam.sieve' after=(none) -imapsieve: Static mailbox rule [2]: mailbox=`*' from=`Junk' - causes=(COPY) => before=`file:/etc/dovecot/sieve/learn-ham.sieve' - after=(none) - -imapsieve: Matched static mailbox rule [1] - -sieve: file storage: script: Opened script `learn-spam' - from `/etc/dovecot/sieve/learn-spam.sieve' - -sieve: action pipe: running program: rspamd-learn-spam.sh - -program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: - Pass environment: USER=john@example.org - -program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: - Pass environment: HOME=/var/vmail/example.org/john - -program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh: - Pass environment: HOST=narnia - -Mailbox Junk: UID 1: Opened mail because: mail stream - -sieve: uid=1: Execute storing into mailbox 'Junk' -``` - -Dovecot finds two Sieve rules: \[1\] and \[2\]. And it finds that rule \[1\] matches your current action and runs it. -That in turn calls the "rspamd-learn-spam.sh" script to train that email as spam. And at the very end the email is -actually moved to the "Junk" folder. - -And if you pull an email out of the "Junk" folder you should see mailbox rule \[2\] be called and the email being -learned as ham. - -Don’t forget to switch off "mail_debug" again or your users’ actions will quickly fill your log file. +If you are happy and see no errors (they are highlighted in red), then switch off the debugging again. ## Logging @@ -559,43 +540,42 @@ found in the rspamd.log, too: Andi Olsen pointed out that Dovecot has introduced a [feature](https://wiki.dovecot.org/MailboxSettings) to automatically delete emails in a folder that reach a certain age. This is especially useful for the "Trash" and "Junk" -folders. To enable this feature just edit the `/etc/dovecot/conf.d/15-mailboxes.conf` file and add the *autoexpunge* -parameter where desired. Example: +folders. To enable this feature, create yet another configuration file: -``` +```sh +cat > /etc/dovecot/conf.d/99-ispmail-autoexpunge.conf << 'EOF' +# Remove mails from the Junk and Trash folders after 30 days mailbox Junk { special_use = \Junk auto = subscribe - autoexpunge = 30d + mailbox_autoexpunge = 30d } mailbox Trash { special_use = \Trash auto = subscribe - autoexpunge = 30d + mailbox_autoexpunge = 30d } + +# Make expunging more efficient +mailbox_list_index = yes +mail_always_cache_fields = date.save +EOF + +# Restart Dovecot +systemctl reload dovecot ``` -The "auto = subscribe" makes sure that the "Junk" and "Trash" folders are automatically created for every user. -Otherwise spam emails cannot be moved to the "Junk" folder later. - -## The web interface +## rspamd's web interface rspamd comes with a neat bonus feature: a web interface. It allows you to check emails for spam, get statistics and -fine-tune scores. It is already installed and enabled by default and expects HTTP requests on port 11334 on the -localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail +fine-tune scores. It is already installed and enabled by default and expects HTTP (not HTTPS!) requests on port 11334 on +the localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail configuration to get access. ![rspamd dashboard](images/catching-spam-rspamd-dashboard.png) -First you need to enable Apache’s modules for HTTP proxying and rewriting: - -``` -a2enmod proxy_http -a2enmod rewrite -``` - You can either create a new virtual host configuration or just edit the -/etc/apache2/sites-available/webmail.**example.org**-https.conf file. Anywhere within the *VirtualHost* tags add: +/etc/apache2/sites-enabled/000-default-le-ssl.conf file. Anywhere within the *VirtualHost* tags add: ``` @@ -603,43 +583,43 @@ You can either create a new virtual host configuration or just edit the RewriteEngine On -RewriteRule ^/rspamd$ /rspamd/ \[R,L\] -RewriteRule ^/rspamd/(.\*) http://localhost:11334/$1 \[P,L\] +RewriteRule ^/rspamd$ /rspamd/ [R,L] +RewriteRule ^/rspamd/(.*) http://localhost:11334/$1 [P,L] +``` + +You also need to enable Apache’s modules for HTTP proxying and rewriting: + +```sh +# Enable the required Apache modules +a2enmod proxy_http +a2enmod rewrite + +# Restart Apache +systemctl restart apache2 ``` This piece of configuration will forward any requests to `https://webmail.example.org/rspamd` to localhost:11334 and thus give you access to the rspamd web interface. -The interface is password protected. Let’s generate a new access password: +The web interface is password protected. Let’s generate a new access password, hash it and add it to rspamd's +configuration: -``` -pwgen 15 1 -``` +```sh +# Generate a random password that is 15 characters long +PW=$(pwgen 15 1) -This gives you a password like "eiLi1lueTh9mia4". You could put that password in an rspamd configuration file. But -cleartext passwords in configuration files are not quite elegant. Let’s create a hash of the password: +# Create a hash of that password +HASH=$(rspamadm pw -p $PW) -``` -rspamadm pw -Enter passphrase: … -$2$icoahes75e7g9wxapnrbmqnpuzjoq7z… -``` +# Add the hashed password to rspamd's configuration +echo "password = \"$HASH\"" > /etc/rspamd/local.d/worker-controller.inc -Feed it the password that pwgen created for you and you will get a long hashed password. This procedure by the way is -also documented on the [rspamd pages](https://rspamd.com/doc/tutorials/quickstart.html#setting-the-controller-password). - -Create a new configuration file `/etc/rspamd/local.d/worker-controller.inc` and put your hashed password in there: - -``` -password = "$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…" -``` - -That’s it for the configuration. Finally restart both rspamd and Apache to load your changed configuration: - -``` +# Restart rspamd systemctl restart rspamd -systemctl restart apache2 + +# Show the password +echo "Please note the password for the web interface: $PW" ``` -If everything went as expected you should now be able to access the rspamd web interface at -`https://webmail.example.org/rspamd` +This gives you a password like "sae8thaoTaengeo". If everything went as expected you should now be able to access the +rspamd web interface at `https://mail.example.org/rspamd` (of course you will use your own server name here).