finish the spam page

This commit is contained in:
Christoph Haas 2025-10-18 22:59:18 +02:00
parent 71b9de9c01
commit 54e4c1c2ac
2 changed files with 119 additions and 136 deletions

File diff suppressed because one or more lines are too long

After

Width:  |  Height:  |  Size: 284 KiB

View file

@ -68,7 +68,7 @@ to complain about it to the alleged sender. Trust me: the sender you see on a sp
it. You would complain to the wrong person which is called
[backscatter](<https://en.wikipedia.org/wiki/Backscatter_(email)>).
## Score metrics
## Spam scores
rspamd will however not reject all spam email. It computes a score that tells how likely a certain email is spam. You
can tell it which scores you would accept, flag as spam or make the incoming email get rejected. Rspamd checks incoming
@ -189,9 +189,6 @@ sieve_script spam-to-junk-folder {
path = /etc/dovecot/sieve/spam-to-junk-folder.sieve
}
# Uncomment this line to get more verbose logs on sieve handling
# log_debug=category=sieve
# Enable the execution of Sieve rules when Postfix sends an email to Dovecot over LMTP
protocol lmtp {
mail_plugins {
@ -214,7 +211,6 @@ systemctl reload dovecot
Now we need to create that Sieve script (`/etc/dovecot/sieve/spam-to-junk-folder.sieve`) that is run on each delivery.
Its job is to check if the `X-Spam: yes` header is present. If it is, the email is filed into the user's `Junk` folder.
Dovecot can only understand _compiled_ Sieve files so we call `sievec` to make it machine-readable.
```sh
# Create the directory for Sieve files
@ -229,9 +225,6 @@ if header :contains "X-Spam" "Yes" {
stop;
}
EOF
# Compile the .sieve file into a .svbin file
sievec /etc/dovecot/sieve/spam-to-junk-folder.sieve
```
Let's give it a test using Swaks. This time we impersonate Postfix and inject an email with an `X-Spam: yes` header
@ -272,7 +265,7 @@ journalctl -eu dovecot
The alleged spam email has been moved to the _Junk_ folder just like we wanted.
## Enable auto-learning
## Automatic spam training
One of rspamds features is analyzing word patterns using probability theory. That functionality is contained in its
[statistical module](https://rspamd.com/doc/configuration/statistic.html). (Yes, the name is misleading.) Essentially
@ -325,7 +318,7 @@ The defaults for auto-learning are:
See [rspamd's documentation](https://docs.rspamd.com/configuration/statistic/#statistics-configuration) if you want to
fine-tune that.
```
```sh
rspamc stat
```
@ -369,17 +362,32 @@ users_enabled = true;
## Learning from user actions
Now we are getting to something really cool. Imagine that you receive a spam email into your inbox that rspamd did not
detect properly. Sure, you can move it to your _Junk_ folder. But that will not improve the detection rate for that kind
of spam. But we can fix that.
Now we are getting to something really cool. Imagine you receive a spam email in your inbox that Rspamd didnt catch.
You could just move it to your Junk folder — but that wouldnt help Rspamd learn to recognize that kind of spam in the
future. Luckily, we can change that.
Lets tell Dovecot that moving emails into the Junk folder teaches rspamd instantly that the email is **spam**. And if
the email is moved out of the _Junk_ folder, then learn it as **ham**. That can be done using another _sieve_ script.
Sieve script not only apply when an email is delivered. Thanks to Dovecot's
Lets tell Dovecot that moving emails **into** the Junk folder teaches rspamd instantly that the email is **spam**. And
if the email is moved **out of** the Junk folder, then learn it as **ham**. That can be done using another _sieve_
script. Sieve scripts not only apply when an email is delivered. Thanks to Dovecot's
[IMAPSieve](https://doc.dovecot.org/2.4.1/core/plugins/imap_sieve.html#imapsieve-plugin-imap-sieve/) plugin, such
scripts can also be triggered if a user moves a mail between folders.
Add another new configuration file to enable and configure that plugin:
The setup requires three parts:
1. Tap into the IMAP connection to know when an email is moved out of or into the Junk folder.
2. Create Sieve scripts that call Shell scripts.
3. Create Shell scripts that actually call `rspamd learn_ham` or `rspamd learn_spam`.
Or as a diagram:
<img
alt="Diagram showing how to learn ham/spam when moving mails"
src="/imapsieve/imapsieve-learn-spam.svg"
width="100%"
/>
Create a new Dovecot configuration file (`99-ispmail-imapsieve.conf`) to enable the required functionality and add
triggers on the Junk folder:
```sh
cat > /etc/dovecot/conf.d/99-ispmail-imapsieve.conf << 'EOF'
@ -400,6 +408,9 @@ sieve_global_extensions {
vnd.dovecot.pipe = yes
}
# Where to look for Sieve scripts that use the Pipe functionality
sieve_pipe_bin_dir = /etc/dovecot/sieve
# Moved into Junk? -> Learn as spam.
mailbox Junk {
sieve_script spam {
@ -428,7 +439,7 @@ The first rule tells Dovecot to run a Sieve script at `/etc/dovecot/sieve/learn
The second rule sets the other way. Whenever an email is moved **out** of the "Junk" folder to any other folder, then
the `/etc/dovecot/sieve/learn-ham.sieve` Sieve script is called.
Let's create both scripts:
Let's create both Sieve scripts:
```sh
# Create spam learning script
@ -443,7 +454,7 @@ require ["vnd.dovecot.pipe", "copy", "imapsieve", "variables"];
pipe :copy "rspamd-learn-ham.sh";
EOF
# Compile both script into machine-readable format
# Compile the Sieve scripts
sievec /etc/dovecot/sieve/learn-spam.sieve
sievec /etc/dovecot/sieve/learn-ham.sieve
@ -452,97 +463,67 @@ chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
chown vmail:vmail /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
```
And the last step is to create the simple shell scripts that do the actual spam/ham training. The first file is
`/etc/dovecot/sieve/rspamd-learn-spam.sh` which contains:
By the way: the `sievec` comment compiles the `*.sieve` file into `*.svbin` files that Dovecot can work with. If you are
not doing this step, Dovecot will do that during runtime. However then you need to make sure that Dovecot has proper
write permissions to the target directory. The `sievec` command is also useful because it scans a `*.sieve` file for
errors. So if your changes do not seem to work it might be worth compiling them and looking for errors.
```
And the last step is to create the simple shell scripts that do the actual spam/ham training:
```sh
# Create the shell script for learning spam
cat > /etc/dovecot/sieve/rspamd-learn-spam.sh << 'EOF'
#!/bin/sh
# Receives an email from Dovecot's Sieve script and pipe it into rspamc
exec /usr/bin/rspamc learn_spam
```
EOF
That looks simple, doesnt it? Nothing more is actually needed. The spam email is piped to this script and rspamd learns
it as a spam email and adjusts its spam detection database accordingly.
The second script teaches ham and is called `/etc/dovecot/sieve/rspamd-learn-ham.sh`. Make it contain:
```
# Create the shell script for learning ham
cat > /etc/dovecot/sieve/rspamd-learn-ham.sh << 'EOF'
#!/bin/sh
# Receives an email from Dovecot's Sieve script and pipe it into rspamc
exec /usr/bin/rspamc learn_ham
```
EOF
These two shell scripts must be made executable:
```
# Fix permissions of the shell scripts
chmod u=rwx,go= /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
chown vmail:vmail /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
```
I hope you havent lost your mind yet. Its really just a chain of things to happen. Lets reiterate how this process
works:
I am sure you are eager to try it out. Of course we don't blindly believe that it works. We want to see it do its job.
Usually you will not see much unless you increase the log level to _debug_. Let's do that:
1. a user moves a spam email into their "Junk" folder
2. Dovecot realizes that this triggers the Sieve rule "imapsieve_mailbox1" so it calls the Sieve
script /etc/dovecot/sieve/learn-spam.sieve (in fact the \*.svbin version of the script)
3. Sieve will take the email and send ("pipe") it to the executable rspamd-learn-spam.sh shell script
4. the script in turn runs the email through the "/usr/bin/rspamc learn_spam" command
```sh
cat > /etc/dovecot/conf.d/99-ispmail-sieve-debug.conf << 'EOF'
# Enable detailed logging of Sieve scripts
log_debug=category=sieve
EOF
This works equally for the other way or learning ham emails of course.
# Restart Dovecot
systemctl reload dovecot
I am sure you are eager to try it out. However to see that it actually works I suggest you edit
the /etc/dovecot/conf.d/10-logging.conf file and set "mail_debug=yes". That will add a lot more detail to the
/var/log/mail.log file but on a busy server may also lead to headaches. 🙂
Restart Dovecot…
```
systemctl restart dovecot
# Follow Dovecot's logs
journalctl -fu dovecot
```
…and watch the /var/log/mail.log file…
Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to John's
Junk folder. The mail log will show a lot of things that are going on. You will see a whole screen full of things that
are going on. Some relevant lines:
```
tail -f /var/log/mail.log
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: Mailbox Junk: imapsieve: storage spam: file: Using Sieve script path: /etc/dovecot/sieve/learn-spam.sieve
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: Mailbox Junk: imapsieve: storage spam: file: script 'learn-spam': Opened from 'spam'
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: sieve: Opening script 1 of 1 from 'spam/learn-spam'
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: sieve: Executing script from '/etc/dovecot/sieve/learn-spam.svbin'
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: sieve: action pipe: running program: rspamd-learn-spam.sh
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: sieve: uid=19: Finalize pipe action (status=ok, action_status=ok, commit_status=ok, pre-commit=yes)
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: sieve: execute exec:/etc/dovecot/sieve/rspamd-learn-spam.sh (1737484): Connected to program
imap(john@example.org)<1737478><FnGIR2BBA/5bar5M>: Debug: sieve: uid=19: pipe action: piped message to program 'rspamd-learn-spam.sh'
```
Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to your
Junk folder. The mail log will show a lot of things that are going on. I tried to compact the output so that you better
understand that it worked:
Something similar should happen when you drag the email back from the Junk folder to the Inbox.
```
imapsieve: Static mailbox rule [1]: mailbox=`Junk' from=`*' causes=(COPY) => before=`file:/etc/dovecot/sieve/learn-spam.sieve' after=(none)
imapsieve: Static mailbox rule [2]: mailbox=`*' from=`Junk'
causes=(COPY) => before=`file:/etc/dovecot/sieve/learn-ham.sieve'
after=(none)
imapsieve: Matched static mailbox rule [1]
sieve: file storage: script: Opened script `learn-spam'
from `/etc/dovecot/sieve/learn-spam.sieve'
sieve: action pipe: running program: rspamd-learn-spam.sh
program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh:
Pass environment: USER=john@example.org
program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh:
Pass environment: HOME=/var/vmail/example.org/john
program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh:
Pass environment: HOST=narnia
Mailbox Junk: UID 1: Opened mail because: mail stream
sieve: uid=1: Execute storing into mailbox 'Junk'
```
Dovecot finds two Sieve rules: \[1\] and \[2\]. And it finds that rule \[1\] matches your current action and runs it.
That in turn calls the "rspamd-learn-spam.sh" script to train that email as spam. And at the very end the email is
actually moved to the "Junk" folder.
And if you pull an email out of the "Junk" folder you should see mailbox rule \[2\] be called and the email being
learned as ham.
Dont forget to switch off "mail_debug" again or your users actions will quickly fill your log file.
If you are happy and see no errors (they are highlighted in red), then switch off the debugging again.
## Logging
@ -559,43 +540,42 @@ found in the rspamd.log, too:
Andi Olsen pointed out that Dovecot has introduced a [feature](https://wiki.dovecot.org/MailboxSettings) to
automatically delete emails in a folder that reach a certain age. This is especially useful for the "Trash" and "Junk"
folders. To enable this feature just edit the `/etc/dovecot/conf.d/15-mailboxes.conf` file and add the *autoexpunge*
parameter where desired. Example:
folders. To enable this feature, create yet another configuration file:
```
```sh
cat > /etc/dovecot/conf.d/99-ispmail-autoexpunge.conf << 'EOF'
# Remove mails from the Junk and Trash folders after 30 days
mailbox Junk {
special_use = \Junk
auto = subscribe
autoexpunge = 30d
mailbox_autoexpunge = 30d
}
mailbox Trash {
special_use = \Trash
auto = subscribe
autoexpunge = 30d
mailbox_autoexpunge = 30d
}
# Make expunging more efficient
mailbox_list_index = yes
mail_always_cache_fields = date.save
EOF
# Restart Dovecot
systemctl reload dovecot
```
The "auto = subscribe" makes sure that the "Junk" and "Trash" folders are automatically created for every user.
Otherwise spam emails cannot be moved to the "Junk" folder later.
## The web interface
## rspamd's web interface
rspamd comes with a neat bonus feature: a web interface. It allows you to check emails for spam, get statistics and
fine-tune scores. It is already installed and enabled by default and expects HTTP requests on port 11334 on the
localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail
fine-tune scores. It is already installed and enabled by default and expects HTTP (not HTTPS!) requests on port 11334 on
the localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail
configuration to get access.
![rspamd dashboard](images/catching-spam-rspamd-dashboard.png)
First you need to enable Apaches modules for HTTP proxying and rewriting:
```
a2enmod proxy_http
a2enmod rewrite
```
You can either create a new virtual host configuration or just edit the
/etc/apache2/sites-available/webmail.**example.org**-https.conf file. Anywhere within the *VirtualHost* tags add:
/etc/apache2/sites-enabled/000-default-le-ssl.conf file. Anywhere within the *VirtualHost* tags add:
```
<Location /rspamd>
@ -603,43 +583,43 @@ You can either create a new virtual host configuration or just edit the
</Location>
RewriteEngine On
RewriteRule ^/rspamd$ /rspamd/ \[R,L\]
RewriteRule ^/rspamd/(.\*) http://localhost:11334/$1 \[P,L\]
RewriteRule ^/rspamd$ /rspamd/ [R,L]
RewriteRule ^/rspamd/(.*) http://localhost:11334/$1 [P,L]
```
You also need to enable Apaches modules for HTTP proxying and rewriting:
```sh
# Enable the required Apache modules
a2enmod proxy_http
a2enmod rewrite
# Restart Apache
systemctl restart apache2
```
This piece of configuration will forward any requests to `https://webmail.example.org/rspamd` to localhost:11334 and
thus give you access to the rspamd web interface.
The interface is password protected. Lets generate a new access password:
The web interface is password protected. Lets generate a new access password, hash it and add it to rspamd's
configuration:
```
pwgen 15 1
```
```sh
# Generate a random password that is 15 characters long
PW=$(pwgen 15 1)
This gives you a password like "eiLi1lueTh9mia4". You could put that password in an rspamd configuration file. But
cleartext passwords in configuration files are not quite elegant. Lets create a hash of the password:
# Create a hash of that password
HASH=$(rspamadm pw -p $PW)
```
rspamadm pw
Enter passphrase: …
$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…
```
# Add the hashed password to rspamd's configuration
echo "password = \"$HASH\"" > /etc/rspamd/local.d/worker-controller.inc
Feed it the password that pwgen created for you and you will get a long hashed password. This procedure by the way is
also documented on the [rspamd pages](https://rspamd.com/doc/tutorials/quickstart.html#setting-the-controller-password).
Create a new configuration file `/etc/rspamd/local.d/worker-controller.inc` and put your hashed password in there:
```
password = "$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…"
```
Thats it for the configuration. Finally restart both rspamd and Apache to load your changed configuration:
```
# Restart rspamd
systemctl restart rspamd
systemctl restart apache2
# Show the password
echo "Please note the password for the web interface: $PW"
```
If everything went as expected you should now be able to access the rspamd web interface at
`https://webmail.example.org/rspamd`
This gives you a password like "sae8thaoTaengeo". If everything went as expected you should now be able to access the
rspamd web interface at `https://mail.example.org/rspamd` (of course you will use your own server name here).