add redis. rewrote autolearn part

This commit is contained in:
Christoph Haas 2025-10-11 20:25:40 +02:00
parent fea866111f
commit 818adae366
2 changed files with 56 additions and 84 deletions

View file

@ -19,7 +19,7 @@ DEBIAN_FRONTEND=noninteractive \
php-intl php-mbstring php-xml unzip certbot \ php-intl php-mbstring php-xml unzip certbot \
roundcube-mysql roundcube roundcube-plugins swaks libnet-ssleay-perl \ roundcube-mysql roundcube roundcube-plugins swaks libnet-ssleay-perl \
ufw mutt unattended-upgrades mariadb-server \ ufw mutt unattended-upgrades mariadb-server \
rspamd opendkim-tools bind9-dnsutils rspamd redis-server opendkim-tools bind9-dnsutils
``` ```
While the server is downloading and installing the packages, let me give you a quick explanation of each package: While the server is downloading and installing the packages, let me give you a quick explanation of each package:
@ -60,6 +60,9 @@ While the server is downloading and installing the packages, let me give you a q
- **rspamd** \ - **rspamd** \
It reliably detects and blocks spam. Also handles adding DKIM signature to outgoing email to prevent spoofing your It reliably detects and blocks spam. Also handles adding DKIM signature to outgoing email to prevent spoofing your
domains. domains.
- **redis-server** \
Rspamd requires a [Redis](https://redis.io/docs/latest/operate/oss_and_stack/) server as its storage of learned spam
emails.
- **opendkim-tools** \ - **opendkim-tools** \
Not strictly necessary. But a nice tool to verify DKIM signatures. You know, for science. Not strictly necessary. But a nice tool to verify DKIM signatures. You know, for science.
- **bind9-dnsutils** \ - **bind9-dnsutils** \

View file

@ -14,7 +14,7 @@ import { Aside } from "@astrojs/starlight/components";
You have come a long way in this guide and your mail server is already fully functional. Now its time to deal with the You have come a long way in this guide and your mail server is already fully functional. Now its time to deal with the
dark side: spam. And there will be lots of it. So we need to detect spam emails and filter them out. I found that dark side: spam. And there will be lots of it. So we need to detect spam emails and filter them out. I found that
[rspamd](https://rspamd.com/) is a well-performing choice for that purpose both in speed and detection. rspamd keeps a [rspamd](https://rspamd.com/) is well-performing choice for that purpose both in speed and detection. rspamd keeps a
permanent process running on your mail server that listens to connections from Postfix using permanent process running on your mail server that listens to connections from Postfix using
the [milter](http://www.postfix.org/MILTER_README.html) (=**m**ail f**ilter**) protocol. Every time an email enters your the [milter](http://www.postfix.org/MILTER_README.html) (=**m**ail f**ilter**) protocol. Every time an email enters your
system, Postfix will send it to rspamd to have its content checked. rspamd runs a lot of checks on the email and system, Postfix will send it to rspamd to have its content checked. rspamd runs a lot of checks on the email and
@ -286,17 +286,39 @@ swaks --to john@example.org --header-X-Spam "yes" --socket /var/spool/postfix/pr
Take a look at Dovecot's log: Take a look at Dovecot's log:
```sh ```sh
journalctl -fu dovecot journalctl -eu dovecot
``` ```
It should read: It should read:
``` ```
mailserver dovecot[1434406]: lmtp(1436598): Connect from local mailserver dovecot[1434406]: lmtp(1436598): Connect from local
mailserver dovecot[1434406]: lmtp(john@example.org)<1436598><kh4lDNl26Wi26xUA5ANL0g>: sieve: msgid=<20251010211257.1436597@auenland>: fileinto action: stored mail into mailbox 'Junk' mailserver dovecot[1434406]: lmtp(john@example.org)<1436598><kh4lDNl26Wi26xUA5ANL0g>: sieve: msgid=<20251010211257.1436597@mailserver>: fileinto action: stored mail into mailbox 'Junk'
mailserver dovecot[1434406]: lmtp(1436598): Disconnect from local: Logged out (state=READY) mailserver dovecot[1434406]: lmtp(1436598): Disconnect from local: Logged out (state=READY)
``` ```
<details class="collapsible">
<summary>Click here to get more detailed logs…</summary>
If you find that the delivery to the _Junk_ folder did not work, you may want to increase the log level. Uncomment the
line
```
log_debug=category=sieve
```
in the `99-ispmail-sieve.conf` file.
That will give you a much deeper insight of what Dovecot has been doing. Restart Dovecot, send another email with
_swaks_ and check the logs again:
```sh
systemctl reload dovecot
journalctl -eu dovecot
```
</details>
The alleged spam email has been moved to the _Junk_ folder. Just like we wanted. The alleged spam email has been moved to the _Junk_ folder. Just like we wanted.
## About Redis ## About Redis
@ -309,99 +331,46 @@ and values. There arent several fields/columns like in SQL. But it is lightni
server it handles around 50,000 requests per second. It gets it speed from its simplicity and from keeping the data in server it handles around 50,000 requests per second. It gets it speed from its simplicity and from keeping the data in
RAM. So it doesnt access the disk to fetch information. (But it copies its data to disk frequently to prevent data RAM. So it doesnt access the disk to fetch information. (But it copies its data to disk frequently to prevent data
loss.) People use Redis as a cache or for very fast lookups of simple data structures. And so loss.) People use Redis as a cache or for very fast lookups of simple data structures. And so
[does rspamd](https://rspamd.com/doc/configuration/redis.html). [does rspamd](https://docs.rspamd.com/configuration/redis/).
You installed the "redis-server" package earlier. And thats all you needed to do. It started automatically and listens ## Enable auto-learning
ton incoming connections on TCP port 6379 on localhost. In Rspamd the Redis backend is enabled by default. You just have
to tell it the IP address of your Redis server. Add a file `/etc/rspamd/override.d/redis.conf` and insert:
```
servers = "127.0.0.1";
```
Restart rspamd and you are done.
```
systemctl restart rspamd
```
Feel free to use Redis for [other lookups](https://rspamd.com/doc/configuration/redis.html), too.
## Training the spam detection
One of rspamds features is analyzing word patterns using probability theory. That functionality is contained in its One of rspamds features is analyzing word patterns using probability theory. That functionality is contained in its
"[statistical module](https://rspamd.com/doc/configuration/statistic.html)". (Yes, the name is pretty misleading.) "[statistical module](https://rspamd.com/doc/configuration/statistic.html)". (Yes, the name is misleading.) Essentially
Essentially you show rspamd lots of ham (good) and spam (bad) emails and its detection gets better over time. you show rspamd lots of ham (good) and spam (bad) emails and its detection gets better over time.
<Aside type="tip" title="Redis replaces SQLite"> You have installed the "redis-server" package earlier. It has started automatically and listens to incoming connections
Rspamd recommends (and defaults to) using Redis to store spam training data. Thats what we will use now. on TCP port 6379 on localhost. You just need to tell rspamd to use it:
</Aside>
### (a) Auto-learning ```sh
# Create a config file to enable automatic spam training
cat > /etc/rspamd/local.d/classifier-bayes.conf << 'EOF'
# Store training data in the Redis database
servers = "127.0.0.1:6379";
backend = "redis";
You can start with an empty training database. This is not as bad as it sounds. rspamd has way more functionality to # Enable automatic training
determine if an email is ham or spam. Autolearning takes email that are likely ham or spam and uses them to train the autolearn = true; # if rspamd is sure that an email is spam, it will be learned
spam filter. The [rspamd documentation](https://rspamd.com/doc/configuration/statistic.html) has further examples how to min_learns = 200; # do not trust the data before at least 200 mails have been learned
fine-tune auto learning. After a few hundred emails the training will contribute towards a better detection rate. EOF
If you want to use _autolearning_ just create a new file `/etc/rspamd/override.d/classifier-bayes.conf` and make it # Restart rspamd
contain:
```
autolearn = [-5, 10];
```
That will train emails with a spam score of less than -5 as ham (good). And emails with a spam score of more than 10 as
spam (unwanted). Feel free to chose other values.
### (b) Migrating training data from previous mail server
Have you used the old SQLite-based training file on the old server? Look for files like /var/lib/rspamd/\*.sqlite on the
old server. In that case please follow these
[simple instructions from the rspamd documentation](https://rspamd.com/doc/faq.html#which-backend-should-i-use-for-statistics)
to convert them to data in Redis.
If instead you have used Redis already then you just need to copy over the Redis database from the old server. Stop
Redis on the new server. Copy over the /var/lib/redis/dump.rdb from the old server to the new server. Start Redis again.
And restart rspamd. So on the new server run:
```
systemctl stop redis
scp root@old-server:/var/lib/redis/dump.rdb /var/lib/redis
systemctl start redis
systemctl restart rspamd systemctl restart rspamd
``` ```
To check if that worked you can ask Rspamd using "rspamc stat" and look for… You can start with an empty training database. This is not as bad as it sounds. rspamd checks many properties of an
email to determine if an email is ham or spam. Autolearning takes email that are likely ham or spam and uses them to
train the spam filter. The [rspamd documentation](https://rspamd.com/doc/configuration/statistic.html) has further
examples how to fine-tune auto learning. After a few hundred emails the training will contribute towards a better
detection rate.
``` The defaults for auto-learning are:
Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0;
total blocks: 0; free: 0.00%; learned: 21164; users: 214; languages: 0
Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0;
total blocks: 0; free: 0.00%; learned: 1411; users: 62; languages: 0
```
### (c) Training from your existing ham and spam emails - score &lt; -0.5 ➞ learn as ham/good
- score &gt; 4.0 ➞ learn as spam/bad
Have you been running a mail server with mailboxes in a _Malidir_ structure before but without rspamd? Then you probably See [rspamd's documentation](https://docs.rspamd.com/configuration/statistic/#statistics-configuration) if you want to
have a good amount of ham and spam emails. Lets use those to train rspamd. It is important to train both ham and spam fine-tune that.
emails. The *rspamc* command will allow you to feed entire directories/folders of emails to the learning process. An
example to train spam:
```
rspamc learn\_spam /var/vmail/example.org/john/Maildir/.Junk/cur
```
And this would be an example to train ham from Johns inbox:
```
rspamc learn\_ham /var/vmail/example.org/john/Maildir/cur
```
Of course the quality of spam detection will depend on how good the source data is. If users put emails in their Junk
folder which are not typical spam they will soil the detection.
Check the number of emails you learned by running…
``` ```
rspamc stat rspamc stat