re-add bookworm and add deprecation notice on every page
|
|
@ -29,6 +29,7 @@ export default defineConfig({
|
|||
// https://expressive-code.com/key-features/word-wrap/#configuration
|
||||
components: {
|
||||
Footer: "./src/components/Footer.astro",
|
||||
Banner: "./src/components/Banner.astro",
|
||||
},
|
||||
customCss: ["./src/styles/custom.css"],
|
||||
sidebar: [
|
||||
|
|
@ -37,6 +38,11 @@ export default defineConfig({
|
|||
// slug: "ispmail-trixie",
|
||||
autogenerate: { directory: "ispmail-trixie" },
|
||||
},
|
||||
{
|
||||
label: "ISPmail for Debian 12",
|
||||
// slug: "ispmail-bookworm",
|
||||
autogenerate: { directory: "ispmail-bookworm" },
|
||||
},
|
||||
{
|
||||
label: "Misc articles",
|
||||
autogenerate: { directory: "articles" },
|
||||
|
|
|
|||
32
src/components/Banner.astro
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
---
|
||||
const { banner } = Astro.locals.starlightRoute.entry.data;
|
||||
const isBookworm = Astro.url.pathname.startsWith("/ispmail-bookworm/");
|
||||
---
|
||||
|
||||
{banner && <div class="sl-banner" data-pagefind-ignore set:html={banner.content} />}
|
||||
|
||||
<style>
|
||||
@layer starlight.core {
|
||||
.sl-banner {
|
||||
--__sl-banner-text: var(--sl-color-banner-text, var(--sl-color-text-invert));
|
||||
padding: var(--sl-nav-pad-y) var(--sl-nav-pad-x);
|
||||
background-color: var(--sl-color-banner-bg, var(--sl-color-bg-accent));
|
||||
color: var(--__sl-banner-text);
|
||||
line-height: var(--sl-line-height-headings);
|
||||
text-align: center;
|
||||
text-wrap: balance;
|
||||
box-shadow: var(--sl-shadow-sm);
|
||||
}
|
||||
.sl-banner :global(a) {
|
||||
color: var(--__sl-banner-text);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
|
||||
{
|
||||
isBookworm && (
|
||||
<section style="background-color: #ffefef; padding: 1rem; border: 1px solid #f00; text-align: center;">
|
||||
⚠️ This section is outdated. Please use the guide for <a href="/ispmail-trixie/">Debian Trixie</a>.
|
||||
</section>
|
||||
)
|
||||
}
|
||||
47
src/content/docs/ispmail-bookworm/10-start.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
---
|
||||
title: Start here
|
||||
lastUpdated: 2023-09-24
|
||||
slug: ispmail-bookworm
|
||||
sidebar:
|
||||
order: 10
|
||||
---
|
||||
|
||||
Once upon a time there were many mail servers on the internet. If your organisation wanted to receive and send emails then you would have your system administrator set up a mail server. He would add a DNS record and create a cryptic Sendmail configuration file.
|
||||
|
||||
Fast forward a few decades and suddenly we have arrived in a monopolistic dystopy. The world of internet services is dominated by a few companies. Having worked in IT for 25 years I feel frustrated about the ignorance and laziness of decision makers. Cloud providers rip obscene profits and lock clueless customers in, while their IT staff is evolving into dumb victims instead of doing their actual job.
|
||||
|
||||
If you are like me then you want to stay as independent as possible. And that includes being in charge of your own email service. Maybe not for your employer but at least for yourself, your friends and family. Become your own internet service provider (ISP) for email. Hence the name: ISPmail.
|
||||
|
||||
This is a complete and free guide that teaches you how to set up and run your own mail server. At the end of the guide you will have your own fully featured mail server based on open-source software using a cheap virtual server for a few bucks per month. And you will have learned all about the various components, protocols and technologies. This guide has been battle-tested by thousands of other sysadmins and constantly evolved over the last 20 years. I am publishing this guide without any commercial motives, so all you would have to invest is your time.
|
||||
|
||||
## What your mail server can do
|
||||
|
||||
- **Receive** emails on your domains.
|
||||
- **Filter** out spam. (We will not deal with detection of Wind*ws malware though.)
|
||||
- **Send** emails out to any other servers/domains on the internet. Connections will be encrypted when possible.
|
||||
- Add automatic cryptographic signatures (**[DKIM / Domain Keys](https://en.wikipedia.org/wiki/DomainKeys)**) to outgoing emails to prove that you are the owner of your domain.
|
||||
- **Store** as many emails for as many email addresses as you have disk space. Set limits (“quotas”) per user. The only limit is the size of your disk.
|
||||
- Let your users fetch email using **IMAP** or **POP3** and send email through your servers using **SMTP**.
|
||||
- Allow users to manage server-based **filter rules**. Distribute incoming emails to different folders. Forward copies. Or send out-of-office notifications.
|
||||
- Provide a **webmail** interface so users can access their emails securely from any location using a web browser.
|
||||
- Mitigate **brute force** attacks.
|
||||
|
||||
## What you will need
|
||||
|
||||
- **Linux experience**. Preferably a Debian-derivative. No godlike skills required. But know your basics: navigating through the file system, editing files, watching log files. Have some basic understanding of DNS. Bonus points if you have played with an SQL database and know about SELECT, INSERT, rows, columns.
|
||||
- **Time**. 2 hours to 2 days.
|
||||
- A **server** that runs Debian Bookworm. 1 GB of RAM and a 20 GB disk/SSD is fine for your friends and family. Rent a cheap virtual server. Or use a decommissioned laptop. Other Linux distributions likely come with other versions of the software and have different configurations and paths on disk. It will work but you will need to deviate from this guide.
|
||||
- The provider providing you with the server needs to allow **SMTP send and receive**. These providers are known to be troublesome:
|
||||
- Hetzner (DE, FI, US): Will allow SMTP after a month of paying for their service and per a request issued at https://console.hetzner.cloud/limits
|
||||
- DigitalOcean (US): Blocks SMTP and tells you that running your own mail server is a bad idea. Avoid.
|
||||
- Your server needs to have a public **IP address** that does not belong to a range of typical ISP customers. You usually can’t operate the mail server from a dialup IP address at home because those IP ranges are blacklisted by most other mail servers. Make sure that your IP address is not [blacklisted](https://multirbl.valli.org/) before you start. If you rent a virtual server from your favorite hosting company you probably won’t have any problems.
|
||||
- An **internet domain** (or several) to receive emails for. You need to be able to set A, MX and TXT records for that domain. You should also be able to set PTR records for your IP address because some mail servers on the internet require you to have matching forward and reverse DNS records.
|
||||
- **Patience**. We will proceed slowly and after every step ensure that are still on track. Don’t hurry and skip parts even if they appear confusing at first. If you get lost just submit your question at the bottom of any page throughout this guide and help is on the way. Or join the [chat channel](https://riot.im/app/#/room/#ispmail:matrix.org).
|
||||
|
||||
## What this is not about
|
||||
|
||||
If you just want to have a working mail server and do not care how it works then this guide is not for you. Check out ready solutions like [mailinabox](https://mailinabox.email/) or [iRedMail](http://www.iredmail.org/). Running a mail server requires technical understanding. And that’s what the ISPmail guide is for. Experience from giving support to other sysadmins shows that most problems appear because some detail in a complex setup goes wrong and they have no idea how to fix it. Email has evolved a lot over the past 40 years. Go with ready solutions if you like. But I have a feeling that we meet again. And you will probably not save time either taking the supposedly easy route.
|
||||
|
||||
## Ready?
|
||||
|
||||
The entire tutorial is split into several pages. You can find the different chapters on the left. The navigation within a chapter can be found on the right. Let’s go.
|
||||
102
src/content/docs/ispmail-bookworm/100-postfix-access-mariadb.mdx
Normal file
|
|
@ -0,0 +1,102 @@
|
|||
---
|
||||
title: Let Postfix access MariaDB
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/making-postfix-get-its-information-from-the-mariadb-database
|
||||
sidebar:
|
||||
order: 100
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
In the previous chapter you have created the SQL database schema and inserted some data to play with. Let’s start with the entry point for all email on your system: Postfix. So we need to tell Postfix how to get the information from the database. First let’s tell it how to find out if a certain domain is a valid email domain.
|
||||
|
||||
## virtual\_mailbox\_domains
|
||||
|
||||
As described earlier a mapping in Postfix is just a table that contains a left-hand side (LHS) and a right-hand side (RHS). To make Postfix get information about virtual domains from the database we need to create a ‘cf’ file (_configuration file_). Start by creating a file called `/etc/postfix/mysql-virtual-mailbox-domains.cf` for the virtual\_mailbox\_domains mapping. Make it contain:
|
||||
|
||||
```
|
||||
user = mailserver
|
||||
password = x893dNj4stkHy1MKQq0USWBaX4ZZdq
|
||||
hosts = 127.0.0.1
|
||||
dbname = mailserver
|
||||
query = SELECT 1 FROM virtual_domains WHERE name='%s'
|
||||
```
|
||||
|
||||
Please enter your own password for the _mailserver_ database user here. It is the first one you created before.
|
||||
|
||||
Imagine that Postfix receives an email for `somebody@example.org` and wants to find out if example.org is a virtual mailbox domain. It will run the above SQL query and replace ‘%s’ by ‘example.org’. If it finds such a row in the virtual\_domains table it will return a ‘1’. Actually it does not matter what exactly is returns as long as there is a result. Remember the puppies and kittens?
|
||||
|
||||
Now you need to make Postfix use this database mapping:
|
||||
|
||||
```sh
|
||||
postconf virtual_mailbox_domains=mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
|
||||
```sh
|
||||
|
||||
The “postconf” command conveniently adds configuration lines to your `/etc/postfix/main.cf` file. It also activates the new setting instantly so you do not have to reload the Postfix process.
|
||||
|
||||
The test data you created earlier added the domain “example.org” as one of your mailbox domains. Let’s ask Postfix if it recognizes that domain:
|
||||
|
||||
```sh
|
||||
postmap -q example.org mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
|
||||
```sh
|
||||
|
||||
<Aside type="tip" title="Access denied?">
|
||||
If you get an error like “postmap: warning: connect to mysql server 127.0.0.1: Access denied for user ‘mailserver’@’localhost'” then please double check the password of the ‘mailserver’ database user. It is either wrong in the CF file or you did not create the database user correctly using the GRANT query on the previous page.
|
||||
</Aside>
|
||||
|
||||
You should get ‘1’ as a result. That means your first mapping is working. Feel free to try that with other domains after the `-q` in that line. You should not get a response.
|
||||
|
||||
## virtual\_mailbox\_maps
|
||||
|
||||
You will now define the _virtual\_mailbox\_maps_. It will map a recipient’s email address (left-hand side) to the location of the user’s mailbox on your hard disk (right-hand side). Postfix has a built-in transport service called “virtual” that can receive the email and store it into the recipient’s email directory. That service is pretty limited, so we will delegate that to Dovecot as it allows us better control.
|
||||
|
||||
Postfix will forward all emails to Dovecot for further delivery. But we need to make sure that the recipient actually exists before we do that. So Postfix needs to check whether an email address belongs to a valid mailbox. That simplifies things a bit because we just need the left-hand side of the mapping.
|
||||
|
||||
Similar to the above virtual\_domains mapping you need an SQL query that searches for an email address and returns “1” if it is found.
|
||||
|
||||
To accomplish that please create another configuration file at `/etc/postfix/mysql-virtual-mailbox-maps.cf`:
|
||||
|
||||
```
|
||||
user = mailserver
|
||||
password = **x893dNj4stkHy1MKQq0USWBaX4ZZdq**
|
||||
hosts = 127.0.0.1
|
||||
dbname = mailserver
|
||||
query = SELECT 1 FROM virtual_users WHERE email='%s'
|
||||
```
|
||||
|
||||
Again please use your actual password for the ‘mailserver’ database user.
|
||||
|
||||
Tell Postfix that this mapping file is supposed to be used for the virtual\_mailbox\_maps mapping:
|
||||
|
||||
```sh
|
||||
postconf virtual_mailbox_maps=mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
|
||||
```
|
||||
|
||||
Test if Postfix is happy with this mapping by asking it where the mailbox directory of our `john@example.org` user would be:
|
||||
|
||||
## virtual\_alias\_maps
|
||||
|
||||
The virtual\_alias\_maps mapping is used for forwarding emails from one email address to one or more others. In the database multiple targets are achieved by using multiple rows.
|
||||
|
||||
Create another “.cf” file at `/etc/postfix/mysql-virtual-alias-maps.cf`:
|
||||
|
||||
```
|
||||
user = mailserver
|
||||
password = **x893dNj4stkHy1MKQq0USWBaX4ZZdq**
|
||||
hosts = 127.0.0.1
|
||||
dbname = mailserver
|
||||
query = SELECT destination FROM virtual\_aliases WHERE source='%s'
|
||||
```
|
||||
|
||||
Make Postfix use this database mapping:
|
||||
```
|
||||
postconf virtual_alias_maps=mysql:/etc/postfix/mysql-virtual-alias-maps.cf
|
||||
```
|
||||
Test if the mapping file works as expected:
|
||||
```
|
||||
postmap -q jack@example.org mysql:/etc/postfix/mysql-virtual-alias-maps.cf
|
||||
```
|
||||
You should see the expected destination:
|
||||
|
||||
`john@example.org`
|
||||
|
||||
80
src/content/docs/ispmail-bookworm/110-catchall-aliases.mdx
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
---
|
||||
title: Catch-all aliases
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/catchall-aliases
|
||||
sidebar:
|
||||
order: 110
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
<Aside type="tip" title="Optional feature">
|
||||
This page is entirely optional and will not affect your mail server if you skip it.
|
||||
</Aside>
|
||||
|
||||
As explained earlier in the tutorial there is way to forward all email addresses in a domain to a certain destination email address. This is called a _catch-all_ alias. Those aliases catch all emails for a domain if there is no specific virtual user for that email address. Catchalls are considered a bad idea. It is tempting to generally forward all email addresses to one person if e.g. your marketing department requests a new email alias every week. But the drawback is that you will get more spam because spammers will send their stuff to any address of your domain. Or perhaps a sender mixed up the proper spelling of a recipient but the mail server will forward the email instead of rejecting it for a good reason. So think twice before using catchalls.
|
||||
|
||||
You still want to use catch-all addresses? Well, okay. Let’s do it then. A catchall alias looks like “@example.org” and forwards email for the whole domain to other addresses. We have created the `john@example.org` user and would like to forward all other email on the domain to `kerstin@example.com`. So we would add a catchall alias like:
|
||||
|
||||
| source | destination |
|
||||
| -------------- | --------------------- |
|
||||
| `@example.org` | `kerstin@example.com` |
|
||||
|
||||
But there is a small catch. Postfix always checks the _virtual_alias_maps_ mapping before looking up a user in the _virtual_mailbox_maps_. Imagine what happens when Postfix receives an email for ‘john@example.org’. Postfix checks the aliases in the _virtual_alias_maps_ table. It finds the catchall entry as above and since there is no more specific alias the catchall account matches and the email is redirected to ‘kerstin@example.com’.
|
||||
|
||||
In other words: the aliases are always processed first. So a catch-all alias will steal the email. John will never get any email. This is not what you want.
|
||||
|
||||
But imagine that the aliases would contain a second entry like this:
|
||||
|
||||
| email | destination |
|
||||
| ---------------------- | ---------------------- |
|
||||
| `@example.org` | `kerstin@example.com` |
|
||||
| **`john@example.org`** | **`john@example.org`** |
|
||||
|
||||
So any email address on the example.org domain will be forwarded to kerstin's address. But what is that second line? Why should we forward john's emails to himself? That doesn't make any sense.
|
||||
|
||||
Actually it does. Postfix will consider more specific aliases first. And `john@example.org` is more specific than `@example.org`.
|
||||
Consider that someone is trying to reach `john@example.org`'s mailbox.
|
||||
If Postfix read this table just from top to bottom, then it would see `@example.org` first, which would be a match. It would then redirect that email to kerstin. John would never again get an email.
|
||||
|
||||
So to make a mixture of catch-all addresses and specific addresses work, we need this little trickery.
|
||||
|
||||
Postfix will lookup all these mappings for each of:
|
||||
|
||||
- john@example.org (most specific)
|
||||
- @example.org (catchall – least specific)
|
||||
|
||||
This is outlined in the [virtual(5) man page](http://www.postfix.org/virtual.5.html) in the _TABLE SEARCH ORDER_ section.
|
||||
|
||||
We do not want to add that “more specific” entry for each email address manually. Fortunately we can easily automate that. For that “john-to-himself” mapping you need to create another “.cf” file `/etc/postfix/mysql-email2email.cf` for the latter mapping:
|
||||
|
||||
```
|
||||
user = mailserver
|
||||
password = x893dNj4stkHy1MKQq0USWBaX4ZZdq
|
||||
hosts = 127.0.0.1
|
||||
dbname = mailserver
|
||||
query = SELECT email FROM virtual_users WHERE email='%s'
|
||||
```
|
||||
|
||||
Check that you get John’s email address back when you ask Postfix if there are any aliases for him:
|
||||
```
|
||||
postmap -q john@example.org mysql:/etc/postfix/mysql-email2email.cf
|
||||
```
|
||||
The result should be the same address:
|
||||
```
|
||||
john@example.org
|
||||
```
|
||||
|
||||
Now you need to tell Postfix that it should check both the aliases and the “john-to-himself”:
|
||||
|
||||
<pre class="wrap">
|
||||
postconf virtual_alias_maps=mysql:/etc/postfix/mysql-virtual-alias-maps.cf,mysql:/etc/postfix/mysql-email2email.cf
|
||||
</pre>
|
||||
|
||||
The order of the two mappings is not important here. Postfix will check all ‘cf’ files anyway and merges what it finds.
|
||||
|
||||
You did it! All mappings are set up and the database is generally ready to be filled with domains and users. Make sure that only ‘root’ and the ‘postfix’ user can read the “.cf” files – after all your database password is stored there:
|
||||
```
|
||||
chgrp postfix /etc/postfix/mysql-*.cf
|
||||
chmod u=rw,g=r,o= /etc/postfix/mysql-*.cf
|
||||
```
|
||||
261
src/content/docs/ispmail-bookworm/120-setting-up-dovecot.mdx
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
---
|
||||
title: Setting up Dovecot
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/setting-up-dovecot
|
||||
sidebar:
|
||||
order: 120
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
This chapter of our journey leads us to Dovecot – the software that…
|
||||
|
||||
- gets emails destined to your users from Postfix and saves them to disk
|
||||
- executes user-based _sieve_ filter rules (can be used to e.g. move emails to different folders based on certain
|
||||
criteria or to send automated vacation responses)
|
||||
- allows the user to fetch emails using POP3 or IMAP
|
||||
|
||||
Before we get to the actual configuration for security reasons I recommend that you create a new system user that will
|
||||
own all virtual mailboxes. The following shell commands will create a system group “vmail” with GID (group ID) 5000 and
|
||||
a system user “vmail” with UID (user ID) 5000. (Make sure that UID and GID are not yet used or choose another – the
|
||||
number can be anything between 1000 and 65000 that is not yet used):
|
||||
|
||||
```
|
||||
groupadd -g 5000 vmail
|
||||
useradd -g vmail -u 5000 vmail -d /var/vmail -m
|
||||
```
|
||||
|
||||
If the /var/vmail directory was already there because you assigned it a dedicated mount point then you should make sure
|
||||
that the permissions are set correctly:
|
||||
|
||||
```
|
||||
chown -R vmail:vmail /var/vmail
|
||||
```
|
||||
|
||||
The configuration files for Dovecot are found in `/etc/dovecot/conf.d`. All these files are loaded by Dovecot. This is
|
||||
done by this magical line at the end of the `/etc/dovecot/dovecot.conf` file:
|
||||
|
||||
```
|
||||
!include conf.d/*.conf
|
||||
```
|
||||
|
||||
It loads all files in `/etc/dovecot/conf.d/` that end in “.conf” in alphanumerical order. So “10-auth.conf” is loaded
|
||||
first and “90-sieve-extprograms.conf” is loaded last. The big advantage is that you can edit or replace parts of the
|
||||
configuration without having to overwrite the entire configuration. The main `/etc/dovecot/dovecot.conf` file does not
|
||||
require any changes. Those other files in conf.d/ however do.
|
||||
|
||||
## conf.d/
|
||||
|
||||
### 10-auth.conf
|
||||
|
||||
The most common
|
||||
[authentication mechanism](https://doc.dovecot.org/configuration_manual/authentication/authentication_mechanisms/#authentication-authentication-mechanisms)
|
||||
is called _PLAIN_. However if you have Outl\*\*k users then you may need to add the _LOGIN_ mechanism, too.:
|
||||
|
||||
```
|
||||
auth_mechanisms = plain login
|
||||
```
|
||||
|
||||
These two mechanisms would ask for a password without enforcing encryption to secure the password. But don’t worry. By
|
||||
default Dovecot sets `disable_plaintext_auth = yes` which ensures that authentication is only accepted over
|
||||
TLS-encrypted connections.
|
||||
|
||||
At the end of this file you will find various authentication backends that Dovecot ships with. By default it will use
|
||||
system users (those from the /etc/passwd). But we want to use the MariaDB database backend so go ahead and change this
|
||||
block to:
|
||||
|
||||
```
|
||||
#!include auth-system.conf.ext
|
||||
!include auth-sql.conf.ext
|
||||
#!include auth-ldap.conf.ext
|
||||
#!include auth-passwdfile.conf.ext
|
||||
#!include auth-checkpassword.conf.ext
|
||||
#!include auth-static.conf.ext
|
||||
```
|
||||
|
||||
### 10-mail.conf
|
||||
|
||||
Change the mail_location setting to:
|
||||
|
||||
```
|
||||
mail_location = maildir:~/Maildir
|
||||
```
|
||||
|
||||
This is the directory where Dovecot will look for the emails of a specific user. The tilde character (~) means the
|
||||
user’s _home directory_. That does not make sense yet. But further down on this page we will tell Dovecot what the _home
|
||||
directory_ is supposed to mean. For example `john@example.org` will have his home directory in
|
||||
/var/vmail/example.org/john.
|
||||
|
||||
Further down in the 10-mail.conf file you will find sections defining the
|
||||
[namespaces](https://doc.dovecot.org/configuration_manual/namespace/). Those are folder structures that your email
|
||||
program sees when connecting to the mail server. If you use POP3 you can only access the “inbox” – which is where all
|
||||
incoming email is stored. Using the IMAP protocol you get access to a hierarchy of folders and subfolders. And you can
|
||||
even share folders between users. Or use a public folder that can be accessed by anyone – even anonymously. So IMAP is
|
||||
generally to be preferred.
|
||||
|
||||
Also edit the “mail*plugins” line to enable the \_quota* plugin we will configure later and turn it into:
|
||||
|
||||
```
|
||||
mail_plugins = quota
|
||||
```
|
||||
|
||||
<Aside type="danger" title="Check your separator setting!">
|
||||
Migrating from a previous server? Previous versions of this guide told you to set the “separator” to either “.” or “/”. The default leads to a folder structure like:
|
||||
"/var/vmail/example.org/john/Maildir/.INBOX.staff.marketing.simon".
|
||||
|
||||
If you see folders like this… "/var/vmail/example.org/john/Maildir/INBOX/staff/marketing/simon" …then please read
|
||||
Dovecot’s notes on the directory structure and the hierarchy separator. Hint: LAYOUT=fs.
|
||||
|
||||
</Aside>
|
||||
|
||||
### 10-master.conf
|
||||
|
||||
This configuration file deals with typical service ports like IMAP or POP3.
|
||||
|
||||
<Aside type="tip" title="Plaintext services? Really?">
|
||||
Don’t worry about the standard unencrypted TCP ports 110 (for POP3) and 143 (for IMAP). They can be kept accessible.
|
||||
If a user connects to these ports they will have to issue a _STARTTLS_ command to switch into encrypted mode before
|
||||
they are allowed to send their password. There is basically no difference between using an plaintext port like 110 for
|
||||
POP3 and then using _STARTTLS_ – or connecting to the encrypted 995 port for POP3S (=secure). See the [Dovecot
|
||||
documentation](https://doc.dovecot.org/admin_manual/ssl/) for another explanation.
|
||||
</Aside>
|
||||
|
||||
So most settings are sane here and do not have to be changed. However one change is required in the “service auth”
|
||||
section because we want Postfix to allow Dovecot as an authentication service. Make it look like this:
|
||||
|
||||
```
|
||||
# Postfix smtp-auth
|
||||
unix_listener /var/spool/postfix/private/auth {
|
||||
mode = 0660
|
||||
user = postfix
|
||||
group = postfix
|
||||
}
|
||||
```
|
||||
|
||||
Well, Postfix runs in a chroot environment located at /var/spool/postfix. It can't access anything outside of that
|
||||
directory. So to allow communication with Postfix we tell Dovecot to place a communication socket into that chroot.
|
||||
|
||||
### 10-ssl.conf
|
||||
|
||||
Earlier in this guide you created both a key and a certificate file to encrypt the communication with POP3, IMAPs and
|
||||
HTTPS between the users and your mail server. You need to tell Dovecot where to find these files:
|
||||
|
||||
```
|
||||
ssl\_cert = \</etc/letsencrypt/live/webmail.example.org/fullchain.pem
|
||||
ssl\_key = \</etc/letsencrypt/live/webmail.example.org/privkey.pem
|
||||
```
|
||||
|
||||
And enforce TLS encryption by setting:
|
||||
|
||||
```
|
||||
ssl = required
|
||||
```
|
||||
|
||||
See the [Dovecot documentation on SSL encryption](https://doc.dovecot.org/admin_manual/ssl/) for more information.
|
||||
|
||||
Next let’s take a look at how Dovecot knows about users and their passwords:
|
||||
|
||||
### auth-sql.conf.ext
|
||||
|
||||
Dovecot reads the `auth-sql.conf.ext` which defines how to find user information in your database. Open the file. There
|
||||
are two sections:
|
||||
|
||||
- userdb: where to find a user’s mailbox in the file system
|
||||
- passdb: where to find the user’s hashed password
|
||||
|
||||
By default Dovecot will run two queries at your database. One for the _userdb_ that gets information like the user ID,
|
||||
group ID, home directory and quota. And another for the _passdb_ that gets the hashed password.
|
||||
|
||||
The “userdb” section already reads:
|
||||
|
||||
```
|
||||
userdb {
|
||||
driver = sql
|
||||
args = /etc/dovecot/dovecot-sql.conf.ext
|
||||
}
|
||||
```
|
||||
|
||||
As you can see Dovecot uses an SQL database lookup to get that information. And it refers to the dovecot-sql.conf.ext
|
||||
file for more information. Let’s see…
|
||||
|
||||
## /etc/dovecot/dovecot-sql.conf.ext
|
||||
|
||||
(This configuration file is one level up and not in “conf.d”.)
|
||||
|
||||
You will find this file well documented although all configuration directives are commented out. Add these lines at the
|
||||
bottom of the file:
|
||||
|
||||
```
|
||||
driver = mysql
|
||||
|
||||
connect = \
|
||||
host=127.0.0.1 \
|
||||
dbname=mailserver \
|
||||
user=mailserver \
|
||||
password=x893dNj4stkHy1MKQq0USWBaX4ZZdq
|
||||
|
||||
user_query = SELECT email as user, \
|
||||
concat('*:bytes=', quota) AS quota_rule, \
|
||||
'/var/vmail/%d/%n' AS home, \
|
||||
5000 AS uid, 5000 AS gid \
|
||||
FROM virtual_users WHERE email='%u'
|
||||
|
||||
password_query = SELECT password FROM virtual_users WHERE email='%u'
|
||||
|
||||
iterate_query = SELECT email AS user FROM virtual_users
|
||||
```
|
||||
|
||||
<Aside type="tip" title="Backslashes">
|
||||
Ending a line with a backslash (\) means that it is continued on the next line. It keeps the configuration more
|
||||
readable when it is split over multiple lines.
|
||||
</Aside>
|
||||
|
||||
What these lines mean:
|
||||
|
||||
- driver: the kind of database. MariaDB is the same kind as MySQL.
|
||||
- connect: where to find the MySQL database and how to access it (username, password)
|
||||
- user_query: an SQL query that returns the user name (=the email address), the quota, the home directory, user ID and
|
||||
group ID.
|
||||
- password_query: this SQL query just gets the password hash from the database
|
||||
- iterate_query: ‘doveadm’ uses this query to get a list of all users. That allows you to use the “doveadm user ‘\*'”
|
||||
command later.
|
||||
|
||||
The _user_query_ gets several pieces of information from the database. Let’s look at it one by one:
|
||||
|
||||
- email AS user
|
||||
It gets the the _email_ field from the database which corresponds to the user name. Dovecot expects it in the _user_
|
||||
field so we set an alias to _“user”._
|
||||
- userdb_quota_rule
|
||||
This is the user’s quota in bytes. Think of it as the maximum possible space on disk that the user can occupy. As
|
||||
[documented](https://doc.dovecot.org/configuration_manual/quota/#per-user-quota) Dovecot expects the quota in a
|
||||
special format like “\*:bytes=10000” if the user should not be able to store more than 10,000 bytes. That’s why we
|
||||
begin the string with ‘\*:bytes=’.
|
||||
- userdb_home
|
||||
This leads to the directory where all emails and various control files for this user are located. The placeholder
|
||||
‘%d’ replaces the domain and ‘%n’ the user part. So for John that makes it “/var/vmail/example.org/john”.
|
||||
- userdb*uid and userdb_gid
|
||||
Those are the user ID and group ID of \_vmail* user – 5000 for both. Dovecot uses it to set the permissions of files
|
||||
it creates. As all users share the same system user “vmail” this is just a static number.s
|
||||
|
||||
## Fix permissions
|
||||
|
||||
Make sure that only root can access the SQL configuration file so nobody else is reading your database access passwords:
|
||||
|
||||
```
|
||||
chown root:root /etc/dovecot/dovecot-sql.conf.ext
|
||||
chmod go= /etc/dovecot/dovecot-sql.conf.ext
|
||||
```
|
||||
|
||||
Restart Dovecot from the shell:
|
||||
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
|
||||
Look at your /var/log/mail.log logfile. You should see:
|
||||
|
||||
```
|
||||
... Dovecot v2.3.13 (f79e8e7e4) starting up for imap, lmtp, sieve, pop3 (core dumps disabled)
|
||||
```
|
||||
|
||||
If you get any error messages please double-check your configuration files.
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
---
|
||||
title: Let Postfix send emails to Dovecot
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/let-postfix-send-emails-to-dovecot
|
||||
sidebar:
|
||||
order: 130
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
I hope you haven’t lost your mind yet. If you are unsure how Postfix and Dovecot work together take a moment and go back to the _big picture_ page.
|
||||
|
||||
In a previous chapter we made sure that Postfix knows which emails it is allowed to receive. Now what to do with the email? It has to be saved to disk into the mailbox of the mail user who is eagerly waiting for it. You could let Postfix handle that using its built-in mail delivery agent (MDA) called “_virtual_“. However compared to the capabilities that Dovecot provides like server-based sieve rules or quotas the Postfix delivery agent is pretty basic. We are using Dovecot anyway to provide the IMAP (and optionally POP3) service. So let’s use its _delivery agent_.
|
||||
|
||||
How can we make Postfix hand over the email to Dovecot? There are generally two ways to establish that link.
|
||||
|
||||
1. Using the _dovecot-lda_ (local delivery agent) process. It can process one email at a time. And it starts up a new process for every email. This was for long the default way. But as you can imagine that it does not scale well.
|
||||
2. The better option is to use [LMTP (local mail transport protocol)](https://en.wikipedia.org/wiki/Local_Mail_Transfer_Protocol) that was conceived for this purpose. It can handle multiple recipients at the same time and has a permanently running process which provides a better performance than using the LDA. In short, LMTP is a variant of SMTP with fewer features. It is meant for email communication between internal services that trust each other.
|
||||
|
||||
You guessed it already – we will go for the second option. You installed the _dovecot-lmtpd_ package earlier. So let’s configure it.
|
||||
|
||||
## Tell Dovecot where to listen for LMTP connections from Postfix
|
||||
|
||||
Edit Dovecot’s configuration file that deals with the LMTP daemon – you can find it at `/etc/dovecot/conf.d/10-master.conf`. Look for the “service lmtp” section and edit it so that it looks like:
|
||||
|
||||
```
|
||||
service lmtp {
|
||||
unix_listener /var/spool/postfix/private/dovecot-lmtp {
|
||||
group = postfix
|
||||
mode = 0600
|
||||
user = postfix
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This makes Dovecot’s _lmtp daemon_ create a UNIX socket at /var/spool/postfix/private/dovecot-lmtp. Just like in the section dealing with setting up Dovecot we make it put a socket into the /var/spool/postfix chroot directory because Postfix is restricted to that directory and cannot access anything outside of it. So from Postfix’s point of view the socket is located at “/private/dovecot-lmtp”.
|
||||
|
||||
Restart Dovecot…
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
Check if dovecot accepted that change:
|
||||
```
|
||||
systemctl status dovecot
|
||||
```
|
||||
The output should contain “Active: active (running)”.
|
||||
|
||||
|
||||
## Tell Postfix to deliver emails to Dovecot using LMTP
|
||||
|
||||
This is even easier. The “_virtual\_transport_” in Postfix defines the service to use for delivering emails to the local system. Dovecot has created a socket file and is ready to listen to incoming LMTP connections. We just need to tell Postfix to send emails there:
|
||||
|
||||
```
|
||||
postconf virtual_transport=lmtp:unix:private/dovecot-lmtp
|
||||
```
|
||||
The syntax looks crazy, but it’s actually simple. You just told Postfix to use the LMTP protocol. And that we want to use a UNIX socket on the same system (instead of a TCP connection). And the socket file is located at `/var/spool/postfix/private/dovecot-lmtp`.
|
||||
|
||||
(You will find further information on these steps in the [Dovecot configuration on Postfix integration](https://doc.dovecot.org/configuration_manual/howto/postfix_dovecot_lmtp/).)
|
||||
|
||||
## Enable server-side mail rules
|
||||
|
||||
One of my favorite features of Dovecot are automatic rules for incoming email that are processed on the server. You can sort away your mailing list emails into special folders. You can reject certain senders. Or you can set up vacation auto-responders. No need to have a mail client running – it all happens automatically on the server even when your mail users are not connected.
|
||||
|
||||
The open standard (RFC 5228) for such rules is called Sieve. Basically, Sieve is a way to manage server-side email rules. A rule consists of conditions and actions. For example if the sender address matches `steve@example.org` you could tell Dovecot to move such emails to your “steve” folder automatically. These rules are stored on the Dovecot server and executed automatically. Whether you connect from your smartphone your laptop or use the webmail access – the rules always work and require no configuration on the client side.
|
||||
|
||||
As we use LMTP that’s where we need to tell the lmtp service that we want to use Dovecot’s “sieve” plugin. Edit the file `/etc/dovecot/conf.d/20-lmtp.conf` and within the “protocol lmtp” section change the “mail\_plugins” line to:
|
||||
|
||||
```
|
||||
mail_plugins = $mail_plugins sieve
|
||||
```
|
||||
Restart Dovecot and you are done:
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
160
src/content/docs/ispmail-bookworm/140-quotas.mdx
Normal file
|
|
@ -0,0 +1,160 @@
|
|||
---
|
||||
title: Quotas
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/quotas
|
||||
sidebar:
|
||||
order: 140
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
<Aside type="tip" title="Optional feature">
|
||||
This feature is completely optional. If you are eager to get finished then skip this page and maybe come back later.
|
||||
</Aside>
|
||||
|
||||
Quotas are size limits for users. You can make sure that users do not waste arbitrary amounts of disk space but are forced to clean up old emails every now and then.
|
||||
|
||||
The magic happens in two places:
|
||||
|
||||
1. Postfix needs to reject new emails if the user’s mailbox is over quota.
|
||||
2. Dovecot needs to keep track of the quota and how much the user has already used up of it.
|
||||
|
||||
### Dovecot quota policy service
|
||||
|
||||
Let’s start with Dovecot. Find the file `/etc/dovecot/conf.d/90-quota.conf` and edit it. There are several `plugin {}` sections. Take one and make it look like:
|
||||
|
||||
```
|
||||
plugin {
|
||||
quota = count:User quota
|
||||
quota_vsizes = yes
|
||||
|
||||
quota_status_success = DUNNO
|
||||
quota_status_nouser = DUNNO
|
||||
quota_status_overquota = "452 4.2.2 Mailbox is full and cannot receive any more emails"
|
||||
}
|
||||
```
|
||||
|
||||
The first line defines that you want to calculate the used space in a user’s _maildir_. There are several [backends](https://doc.dovecot.org/configuration_manual/quota_plugin/) like that but the _[count](https://doc.dovecot.org/configuration_manual/quota/quota_count/#quota-backend-count)_ is the best choice in this context. (Previous guides used _maildir_ here.) The string “User quota” is just an arbitrary string that may be queried from a mail user agent.
|
||||
|
||||
The lines starting with “`quota_status_…`” set return values for the service that you will set up in a minute. It will tell Postfix that it will not interfere (_DUNNO_ – colloquial way to say “I don’t know”). And it will return a string with a return code 452 if the user is over quota. Codes starting with “4” mean temporary errors. It will tell the sending party that it is worth retrying at a later time. However if the user does not resolve the issue it will lead to a _bounce_ error email after three days.
|
||||
|
||||
In the same file (_90-quota.conf_) add another section:
|
||||
|
||||
```
|
||||
service quota-status {
|
||||
executable = /usr/lib/dovecot/quota-status -p postfix
|
||||
unix_listener /var/spool/postfix/private/quota-status {
|
||||
user = postfix
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This creates a new [Dovecot service](https://doc.dovecot.org/configuration_manual/service_configuration/) responding to requests from other processes. You surely recognize that we put it into the jail that Postfix runs in (_/var/spool/postfix_), so that Postfix can access it.
|
||||
|
||||
Time to restart Dovecot:
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
Take a look at the /var/spool/postfix/private directory. If all went as intended you will find a socket file called `quota-status` there. Otherwise please check the `/var/log/mail.log` file for errors.
|
||||
|
||||
### Postfix recipient restrictions
|
||||
|
||||
If we stopped here, then Dovecot would reject emails for users who have no space left. However Postfix would still happily receive new emails and attempt to forward them to Dovecot via LMTP. Dovecot however will deny that. It will then keep the email in its queue and retry for a while. In the end it will send a _bounce_ back to the sender telling them about the problem. So why is this bad?
|
||||
|
||||
1. The sender will assume that the email was delivered while it is stuck in the queue for up to three days.
|
||||
2. Spam emails use forged senders. So at the time that Postfix generates the _bounce email_ it will likely send it to an innocent person. This is called _backscatter_ and considered a mail server misconfiguration. Such a problem may get your mail server blacklisted. You don’t want that.
|
||||
|
||||
So the next logical step is to make Postfix check whether a mailbox is over quota whenever a new email arrives. Let’s hook up into the “RCPT TO” phase of the SMTP dialog when a new email comes in. Postfix checks its _smtpd\_recipient\_restrictions_ configuration at this stage. Run this command in the shell:
|
||||
|
||||
```
|
||||
postconf smtpd_recipient_restrictions=reject_unauth_destination, \
|
||||
"check_policy_service unix:private/quota-status"
|
||||
```
|
||||
|
||||
This adds two checks:
|
||||
|
||||
1. `reject_unauth_destination` checks whether the mail server is the final destination for the recipient’s email address. This is pretty much the default behavior if you do not define any restrictions.
|
||||
2. `check_policy_service` connects to the socket file at `/var/spool/postfix/private/quota-status` that was put there by Dovecot. It will use it to ask Dovecot whether the user is over quota in which case the email would get rejected.
|
||||
|
||||
### Test it
|
||||
|
||||
If you are curious to see this working, then set John’s mailbox quota to 5 KB:
|
||||
|
||||
```sql
|
||||
# mysql mailserver
|
||||
mysql> update virtual_users set quota=4000 where email='john@example.org';
|
||||
```
|
||||
|
||||
Send him a few emails using the ‘swaks’ tool:
|
||||
```
|
||||
swaks --server localhost --to john@example.org
|
||||
```
|
||||
After a few emails you will see the rejection message:
|
||||
|
||||
```
|
||||
-> RCPT TO:john@example.org
|
||||
<** 452 4.2.2 john@example.org: Recipient address rejected: Mailbox is full and cannot receive any more emails
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
|
||||
These are things you should consider if quotas do not seem to work properly:
|
||||
|
||||
- Check if you have enabled “quota” in the “mail\_plugins” in the 10-mail.conf file.
|
||||
- Your users may complain that they have deleted many emails but are still over quota. Let them check if they actually emptied the _Trash_ folder. Of course emails in that folder also contribute to the disk space usage. Once the Trash folder is expunged the problem should be gone. You may also allow your users more space in the Trash folder. That’s explained in the [Dovecot documentation](https://doc.dovecot.org/configuration_manual/quota/#quota-rules).
|
||||
- If you directly remove files from a user’s Maildir instead of properly accessing the mailbox using IMAP then you will screw up the quota calculation. In that case let Dovecot recalculate the quota:
|
||||
`doveadm quota recalc -u john@example.org`
|
||||
|
||||
### Automatic warning emails
|
||||
|
||||
The last step is to inform the poor users if they accidentally went over quota. After all they do not necessarily recognize that on their own. Let’s do that by sending them an email with a warning. Yes, we will make sure that the email gets through even if the quota is reached.
|
||||
|
||||
Edit the `90-quota.conf` file again. Add this section to the file (derived from the [Dovecot documentation](https://doc.dovecot.org/configuration_manual/quota/#quota-warning-scripts)):
|
||||
|
||||
```
|
||||
plugin {
|
||||
quota_warning = storage=95%% quota-warning 95 %u
|
||||
quota_warning2 = storage=80%% quota-warning 80 %u
|
||||
}
|
||||
service quota-warning {
|
||||
executable = script /usr/local/bin/quota-warning.sh
|
||||
unix_listener quota-warning {
|
||||
user = vmail
|
||||
group = vmail
|
||||
mode = 0660
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This section defines two automatic quota warnings. The first (quota\_warning) is triggered if the user reaches 95% of the quota. The second (quota\_warning2) at 80%. These lines follow this schema:
|
||||
|
||||
- **Trigger** (e.g. “storage=95%”). The “%” sign needs to be used twice if you want to emit a literal percent sign. So this is not a typo.
|
||||
- The **socket** you want to call in that case. Our socket is the “service quota-warning” that calls a shell script.
|
||||
- Additional **parameters** that are passed to the shell script in our case. They tell the script the percentage that has been reached (e.g. 95) and the address of the user who should get the warning.
|
||||
|
||||
Apparently we need the script to run. So please create a new file at `/usr/local/bin/quota-warning.sh` and put these lines into it:
|
||||
|
||||
```
|
||||
#!/bin/sh
|
||||
PERCENT=$1
|
||||
USER=$2
|
||||
cat << EOF | /usr/lib/dovecot/dovecot-lda -d $USER -o "plugin/quota=maildir:User quota:noenforcing"
|
||||
From: postmaster@webmail.example.org
|
||||
Subject: Quota warning - $PERCENT% reached
|
||||
|
||||
Your mailbox can only store a limited amount of emails.
|
||||
Currently it is $PERCENT% full. If you reach 100% then
|
||||
new emails cannot be stored. Thanks for your understanding.
|
||||
EOF
|
||||
```
|
||||
|
||||
Make this file executable:
|
||||
|
||||
```
|
||||
chmod +x /usr/local/bin/quota-warning.sh
|
||||
```
|
||||
Time to restart Dovecot again:
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
Dovecot’s quota limits can be configured in many ways. If you have special needs then give [their documentation](https://doc.dovecot.org/configuration_manual/quota/) a look.
|
||||
34
src/content/docs/ispmail-bookworm/150-testing-imap.mdx
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
---
|
||||
title: Testing IMAP
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/testing-imap
|
||||
sidebar:
|
||||
order: 150
|
||||
---
|
||||
|
||||
You have already completed the configuration of Dovecot. So fetching emails via IMAP should already work. Let’s give it
|
||||
a try using a simple-looking but powerful IMAP client: _mutt_.
|
||||
|
||||
```
|
||||
mutt -f imaps://john@example.org@webmail.example.org
|
||||
```
|
||||
|
||||
The connection URL may look a little confusing because of the two “@” characters. Usually _mutt_ expects the format
|
||||
`imaps://user@server`. And as we use the email address as the “user” part you get this look.
|
||||
|
||||
You should get prompted for the password which we set to “summersun”. If you get any certificate warnings then check if
|
||||
you used the correct server name to connect to and if you completed the certificate/LetsEncrypt part earlier in this
|
||||
guide.
|
||||
|
||||
After logging in you will see an empty inbox:
|
||||
|
||||

|
||||
|
||||
Or if you have played around with quotas in the previous section you will see a couple of emails plus the quota
|
||||
warnings:
|
||||
|
||||

|
||||
|
||||
Very good – IMAP connections and authentication works. That’s all we wanted to test. Exit _mutt_ by pressing “q”.
|
||||
|
||||
Congratulations. At this point your server can already receive emails.
|
||||
|
|
@ -0,0 +1,156 @@
|
|||
---
|
||||
title: Webmail using Roundcube
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/webmail-using-roundcube
|
||||
sidebar:
|
||||
order: 160
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
<Aside type="tip" title="Optional feature">
|
||||
This feature is completely optional. If you are eager to get finished then skip this page and maybe come back later. You can still access your mail server using a mail user agent like Thunderbird.
|
||||
</Aside>
|
||||
|
||||
Power users may still want to use a mail client like Thunderbird. But most users nowadays seem to prefer reading their emails in the web browser. Let us install a web application for that purpose: [Roundcube](https://roundcube.net/). Roundcube is the software that was also used in the previous versions of this guide. So if your users are used to it… just stay with it.
|
||||
|
||||
## Installation
|
||||
|
||||
Start by installing the software packages:
|
||||
|
||||
```
|
||||
apt install -y roundcube roundcube-plugins \
|
||||
roundcube-plugins-extra roundcube-mysql
|
||||
```
|
||||
|
||||
Roundcube stores user settings in the database. So you will get asked to set up database access:
|
||||
|
||||

|
||||
|
||||
Choose Yes.
|
||||
|
||||
When asked for a password just press _ENTER_.
|
||||
|
||||

|
||||
|
||||
## Configure Apache
|
||||
|
||||
Do you remember that earlier in this guide I asked you how want to name your mail server? Whether you want to use one common name like “webmail.example.org” for all your domains? Or if you prefer different host names for each domain like “webmail.domain1.com” and “webmail.domain2.com”? If you want to use just more then you will have to create one virtual host configuration per domain. The following instructions will just deal with one common host name.
|
||||
|
||||
To get Apache to serve the Roundcube application you need to edit the <tt>/etc/apache2/sites-available/**webmail.example.org**-https.conf</tt> file. I suggest you change the `DocumentRoot` line to:
|
||||
```
|
||||
DocumentRoot /var/lib/roundcube/public_html
|
||||
```
|
||||
All URLs are relative to that directory. So if you go to `https://webmail.example.com/` then files are looked up in that directory.
|
||||
|
||||
Also add this line within the same `VirtualHost` section to add a couple of prepared security settings:
|
||||
```
|
||||
Include /etc/roundcube/apache.conf
|
||||
```
|
||||
|
||||
And as usual Apache needs to be restarted after the configuration change:
|
||||
```
|
||||
systemctl restart apache2
|
||||
```
|
||||
Check that Apache is running properly:
|
||||
```
|
||||
systemctl status apache2
|
||||
```
|
||||
In case of a problem run “`apache2ctl configtest`” to find the cause.
|
||||
|
||||
## Limit access to localhost
|
||||
|
||||
The main configuration file of Roundcube is located at `/etc/roundcube/config.inc.php`. Feel free to customize the file. Fortunately nowadays the basic settings are already as we need them. However these two settings need to be changed by you:
|
||||
|
||||
```
|
||||
$config['imap_host'] = "tls://webmail.example.org:143";
|
||||
$config['smtp_host'] = 'tls://webmail.example.org:587';
|
||||
```
|
||||
So now when your users enter `https://webmail.example.org/` in their browser they should get the Roundcube login form:
|
||||
|
||||

|
||||
|
||||
Keep in mind that we are using the email address as the account name of the user. So when logging in please enter the email address as the user name. E.g. ‘john@example.org’ and password ‘summersun’.
|
||||
|
||||
<Aside type="tip" title="Login failed? Storage server can’t be reached?">
|
||||
In that case please double check your Dovecot 10-ssl.conf file if you set the path to your Let’s Encrypt certificate correctly. Also check the /var/lib/roundcube/logs/errors.log file for errors.
|
||||
</Aside>
|
||||
|
||||
|
||||
|
||||
## Plugins
|
||||
|
||||
Roundcube comes with various plugins that you can offer your users. I recommend at least these two:
|
||||
|
||||
- password: Let the user change their access password.
|
||||
- managesieve: Let the user manage rules that apply to incoming email. They can move mails to specific folders automatically for example.
|
||||
|
||||
Again edit the `/etc/roundcube/config.inc.php` file and look for the _plugins_ configuration. To enable the recommended plugins change it to:
|
||||
|
||||
```
|
||||
$config['plugins'] = array(
|
||||
'managesieve',
|
||||
'password',
|
||||
);
|
||||
```
|
||||
|
||||
### password plugin
|
||||
|
||||
Plugins are configured through files located in the `/etc/roundcube/plugins` directory. Let’s begin with the password plugin. Edit the `/etc/roundcube/plugins/password/config.inc.php` file.
|
||||
|
||||
Oops, that file looks pretty empty. But it refers us to an example file at `/usr/share/roundcube/plugins/password/config.inc.php.dist`. There are many different methods to let users change their passwords. As we store that information in the SQL database, that is the part we need to set up.
|
||||
|
||||
<Aside type="tip" title="No more doveadm">
|
||||
In previous versions of this guide I used the “doveadm pw” command to generate passwords. This is no longer needed. Roundcube can now generate the passwords in the right format to be understood by Dovecot.
|
||||
</Aside>
|
||||
|
||||
Remove the empty definition line of $config from your `config.inc.php` file. Let’s go through the required settings one by one:
|
||||
|
||||
- `$config['password_driver'] = 'sql';`\
|
||||
Simple. Use SQL as a backend.
|
||||
- `$config['password_minimum_length'] = 12;`\
|
||||
Allow no passwords shorter than 12 characters. I consider longer passwords more secure than short passwords with weird characters. You can even choose a larger minimum.
|
||||
- `$config['password_force_save'] = true;`\
|
||||
This will overwrite the password in the database even if it hasn’t changed. It helps us improve the strength of the password hash by re-encoding it with a better algorithm even if the user chooses to keep his old password.
|
||||
- `$config['password_algorithm'] = 'blowfish-crypt';`\
|
||||
The cryptographic algorithm to encode the password. This one is considered very secure and supported by Dovecot.
|
||||
- `$config['password_algorithm_prefix'] = '{CRYPT}';`\
|
||||
Prepend every password with this string so that Dovecot knows how we encrypted the password.
|
||||
- `$config['password_db_dsn'] = 'mysql://mailadmin:gefk6lA2brMOeb8eR5WYaMEdKDQfnF@localhost/mailserver';`\
|
||||
Connection information for the local database. Use your own password for the _mailadmin_ (!) database user here. We cannot use the restricted _mailserver_ user because we have to write to the database if the user changes his password.
|
||||
- `$config['password_query'] = "UPDATE virtual_users SET password=%P WHERE email=%u";`\
|
||||
The SQL query that is run to write the new password hash into the database. %P is a placeholder for the new password hash. And %u is the logged-in user and conveniently matches the email address.
|
||||
|
||||
Make sure that this config file is not world-readable:
|
||||
|
||||
```
|
||||
chown root:www-data /etc/roundcube/plugins/password/config.inc.php
|
||||
chmod u=rw,g=r,o= /etc/roundcube/plugins/password/config.inc.php
|
||||
```
|
||||
Try it. Log into Roundcube as `john@example.org` with password ‘summersun’. Go to the _Settings_. Choose _Password_. Enter a new password twice. You should get a success message at the bottom right. Now logout and login with the new password. Does it work? Great.
|
||||
|
||||
### sieve plugin
|
||||
|
||||
[Sieve](https://en.wikipedia.org/wiki/Sieve_\(mail_filtering_language\)) is a simple programming language to be used for server-side rules. Dovecot executes these rules every time a new email comes in. There are global rules that are executed for every email. And of course every user/mailbox can have its own rules. To manage sieve rules Dovecot offers the _managesieve_ interface that you enabled earlier. So we just need to tell Roundcube how to access it.
|
||||
|
||||
The configuration file for Roundcube’s _managesieve_ plugin is found at `/etc/roundcube/plugins/managesieve/config.inc.php`. Edit the file and again remove the empty or comment the `$config` line. You can again find all possible configuration options in the `/usr/share/roundcube/plugins/managesieve/config.inc.php.dist` file.
|
||||
|
||||
This time just one setting is required to tell Roundcube which server to talk to:
|
||||
```
|
||||
$config['managesieve_host'] = 'localhost';
|
||||
```
|
||||
Sieve rules are stored in a special syntax on the server. This is an example that moves all incoming emails to the _test_ folder that have “test” in the subject:
|
||||
```
|
||||
require ["fileinto"];
|
||||
if header :contains "subject" "test"
|
||||
{
|
||||
fileinto "INBOX/test";
|
||||
}
|
||||
```
|
||||
You do not need to learn this syntax though. Roundcube’s sieve rule editor is way more user-friendly.
|
||||
|
||||
Try adding a sieve rule for `john@example.org` in Roundcube. That feature is located in Settings/Filters. You will find the machine-readable sieve code at `/var/vmail/example.org/john/sieve/roundcube.sieve`.
|
||||
|
||||
The rule editor looks like this:
|
||||
|
||||

|
||||
168
src/content/docs/ispmail-bookworm/170-testing-email-delivery.mdx
Normal file
|
|
@ -0,0 +1,168 @@
|
|||
---
|
||||
title: Testing email delivery
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/testing-email-delivery
|
||||
sidebar:
|
||||
order: 170
|
||||
---
|
||||
|
||||
So far you have spent considerable time with theory and configuration. Are you worried whether all you did actually leads to a working mail server? Before we do the final steps let’s take a break and verify that everything you did so far works as expected.
|
||||
|
||||
At this point the /var/vmail directory should be empty or maybe contain an “example.org” directory if you played with the john@example.org account previously. You can get a list of all files and directories within by running:
|
||||
```
|
||||
find /var/vmail
|
||||
```
|
||||
Although there are not actually any emails on the server yet, you may still get something along the lines of:
|
||||
```
|
||||
/var/vmail
|
||||
/var/vmail/example.org
|
||||
/var/vmail/example.org/john
|
||||
/var/vmail/example.org/john/.dovecot.sieve
|
||||
/var/vmail/example.org/john/sieve
|
||||
/var/vmail/example.org/john/sieve/roundcube.sieve
|
||||
/var/vmail/example.org/john/sieve/tmp
|
||||
/var/vmail/example.org/john/Maildir
|
||||
/var/vmail/example.org/john/Maildir/subscriptions
|
||||
/var/vmail/example.org/john/Maildir/maildirfolder
|
||||
/var/vmail/example.org/john/Maildir/new
|
||||
/var/vmail/example.org/john/Maildir/cur
|
||||
/var/vmail/example.org/john/Maildir/.INBOX.test
|
||||
/var/vmail/example.org/john/Maildir/.INBOX.test/maildirfolder
|
||||
/var/vmail/example.org/john/Maildir/.INBOX.test/new
|
||||
/var/vmail/example.org/john/Maildir/.INBOX.test/dovecot.index.log
|
||||
/var/vmail/example.org/john/Maildir/.INBOX.test/cur
|
||||
/var/vmail/example.org/john/Maildir/.INBOX.test/dovecot-uidlist
|
||||
/var/vmail/example.org/john/Maildir/.INBOX.test/tmp
|
||||
/var/vmail/example.org/john/Maildir/dovecot-uidvalidity.5ddc842b
|
||||
/var/vmail/example.org/john/Maildir/tmp
|
||||
```
|
||||
Basically the schema you see here is /var/vmail/DOMAIN/USER/Maildir/…
|
||||
|
||||
Eeach IMAP mail folder has three subdirectories:
|
||||
|
||||
- `new` – every file here is an email that was stored in this mail folder but not yet read
|
||||
- `cur` – the same but for email that has been read already
|
||||
- `tmp` – for temporary files from the mail server
|
||||
|
||||
Nested folders (folders within folders) will be separated by a dot like this:
|
||||
|
||||
- …/Maildir/new/… – the main inbox
|
||||
- …/Maildir/.INBOX.reddit/new/… – the “reddit” mail folder below the inbox
|
||||
- …/Maildir/.INBOX.servers.inga/new/… – the “servers”/”inga” mail folder below the inbox
|
||||
|
||||
## Check Postfix
|
||||
|
||||
To check for obvious configuration error in Postfix please run:
|
||||
```
|
||||
postfix check
|
||||
```
|
||||
Did you get the error…
|
||||
“_error: open database /etc/aliases.db: No such file or directory_“? Don’t worry. Just run the “newaliases” command to create a new machine-readable file from what aliases were defined in the `/etc/aliases`.
|
||||
|
||||
Very likely you will get this error: “_postfix/postfix-script: warning: symlink leaves directory: /etc/postfix/./makedefs.out_“. That is a [harmless bug of the Debian package](https://bugs.debian.org/926331) that can safely be ignored.
|
||||
|
||||
## Send a test email
|
||||
|
||||
It is time to send a new email into the system. Open a new terminal window and run
|
||||
```
|
||||
tail -f /var/log/mail.log
|
||||
```
|
||||
to see what the mail server is doing. Now let’s send an email to John. My favorite tool for mail tests is _swaks_ that you installed earlier. In a second terminal run:
|
||||
```
|
||||
swaks --to john@example.org --server localhost
|
||||
```
|
||||
If all works as expected, your mail.log will show a lot of technical information about the email delivery. Let me explain what happens at each stage.
|
||||
|
||||
- `postfix/smtpd[29225]: connect from localhost.localdomain[127.0.0.1]`
|
||||
Postfix receives an incoming SMTP connection.
|
||||
- `postfix/smtpd[29225]: 8BA46A0A3A: client=localhost.localdomain[127.0.0.1]`
|
||||
Postfix assigns a unique identifier (8BA46A0A3A) to this connection so that you see which log lines belong together. This is especially important with busy mail servers where multiple mails are handled in parallel.
|
||||
- `postfix/cleanup[29233]: 8BA46A0A3A: *message-id=20191126153053.029243@webmail.example.org`
|
||||
_swaks_ created a unique [message id](https://en.wikipedia.org/wiki/Message-ID) to the email which helps you identify specific mails in the log file.
|
||||
- `postfix/qmgr[13667]: 8BA46A0A3A: from=root@webmail.example.org, size=485, nrcpt=1 (queue active)`
|
||||
The sender was root@webmail.example.org. This is logged after _swaks_ sent the “MAIL FROM” line during the SMTP dialog.
|
||||
- `postfix/smtpd[29225]: disconnect from localhost.localdomain[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5`
|
||||
The SMTP communication ends. Postfix has now received and queued the email.
|
||||
- `dovecot: lmtp(29237): Connect from local`
|
||||
Postfix connects to Dovecot to hand over the email via the LMTP interface.
|
||||
- `dovecot: lmtp(john@example.org)<29237><2PJTIh033V01cgAARGEcaw>: sieve: msgid=20191126153053.029243@webmail.example.org: stored mail into mailbox 'INBOX.test'`
|
||||
Dovecot received the email and even evaluated John’s sieve rule which made the email get stored to the ‘test’ folder of his mailbox.
|
||||
- `dovecot: lmtp(29237): Disconnect from local: Client has quit the connection (state=READY)`
|
||||
The LMTP connection between Postfix and Dovecot is closed.
|
||||
- `postfix/lmtp[29236]: 8BA46A0A3A: to=john@example.org, relay=webmail.example.org[private/dovecot-lmtp], delay=0.01, delays=0/0/0/0.01, dsn=2.0.0, status=sent (250 2.0.0 john@example.org 2PJTIh033V01cgAARGEcaw Saved)`
|
||||
This is the one of the most interesting lines in your mail log. It tells you what happened with a certain email. In this case it says that it was handed over to _dovecot-lmtp_ and that the delivery was successful (_status=sent_). The status codes like 2.0.0 are defined in [RFC 3463](https://tools.ietf.org/html/rfc3463) and work similar to status codes in HTTP. Codes beginning with ‘2’ are good. Those with ‘4’ are temporary errors. And ‘5’ stands for a permanent failure.
|
||||
|
||||
Your output may look slightly differently. Just focus on the parts that are printed in **bold letters**. If everything worked as expected Postfix has accepted the email and forwarded it to Dovecot which in turn wrote the email in John’s maildir. If you get any errors in the log file then try to understand the error message and find the cause of the problem before you proceed.
|
||||
|
||||
Look again:
|
||||
|
||||
find /var/vmail
|
||||
|
||||
Dovecot has now created a directory structure for John and created a new file:
|
||||
```
|
||||
/var/vmail/
|
||||
[…]
|
||||
/var/vmail/example.org/john/Maildir/new/1515485447.M404984P2636.mail,S=510,W=522
|
||||
[…]
|
||||
```
|
||||
The file will have a different name on your system – that’s okay. It is the only file in the “new” folder.
|
||||
|
||||
## Accessing the email as a file on disk
|
||||
|
||||
The file just contains the email:
|
||||
|
||||
```
|
||||
Return-Path: <root@mail.example.org>
|
||||
Delivered-To: john@example.org
|
||||
Received: from mail.example.org
|
||||
by mail.example.org (Dovecot) with LMTP id iHHHEwd5VFpMCgAA3BOsLQ
|
||||
for <john@example.org>; Tue, 09 Jan 2018 09:10:47 +0100
|
||||
Received: by mail.example.org (Postfix, from userid 0)
|
||||
id 30B4B41A98; Tue, 9 Jan 2018 09:10:47 +0100 (CET)
|
||||
Message-Id: <20180109081047.30B4B41A98@mail.example.org>
|
||||
Date: Tue, 9 Jan 2018 09:10:47 +0100 (CET)
|
||||
From: root@mail.example.org (root)
|
||||
|
||||
Di 9. Jan 09:10:47 CET 2018
|
||||
```
|
||||
The advantage of this is that backups are easy and you can even restore single files if a user has accidentally removed an email.
|
||||
|
||||
If anything went wrong then carefully check the last lines of your /var/log/mail.log. It will very likely point you to the problem. Or read the troubleshooting section. Or just add a comment to this page and ask other readers for help.
|
||||
|
||||
You can also use a slightly more comfortable tool to access Maildirs that will come handy for you as a mail server administrator: “mutt”.
|
||||
```
|
||||
mutt -f /var/vmail/example.org/john/Maildir
|
||||
```
|
||||
(You may get asked to create /root/Mail – this is standard procedure. Just press Enter.)
|
||||
|
||||
What you see now are the contents of John’s mailbox:
|
||||
|
||||

|
||||
|
||||
Using _mutt_ is a nice way to check mailboxes while you are logged in to the mail server.
|
||||
|
||||
To reiterate what happens when you receive an email:
|
||||
|
||||
1. Postfix receives the email (using the “swaks” command in this example – but usually through the network using the SMTP protocol from other servers)
|
||||
2. Postfix asks Dovecot if the user is over quota
|
||||
3. Postfix talks to Dovecot via LMTP and hands over the email
|
||||
4. Dovecot runs through the user’s Sieve rules
|
||||
5. Dovecot writes the email file to disk
|
||||
|
||||
|
||||
## Accessing the email via IMAP (Roundcube)
|
||||
|
||||
Now that the email has been delivered you can talk to Dovecot using the IMAP protocol to retrieve your email again. Are you still logged in via the Roundcube webmail interface? Then just reload and you will see the email:
|
||||
|
||||

|
||||
|
||||
## Access the email through IMAP (desktop client)
|
||||
|
||||
Your users are not likely to use “mutt” to read their email – unless they are immortal console nerds. So of course you can use a graphical mail client to access John’s emails. If you use Thunderbird, Evolution, Apple Mail or some other IMAP client then feel free to configure a new IMAP account and connect to the server. Remember that the user name and email address are both `john@example.org` and the password is “summersun”.
|
||||
|
||||
## POP3 versus IMAP
|
||||
|
||||
Aside from talking IMAP Dovecot also knows how to speak POP3. But let’s face it – POP3 is dead. I do not even offer it to my users any more. But as your users may ask let’s quickly review the differences of the two protocols:
|
||||
|
||||
- _POP3_ (Post Office Protocol) is a simple protocol that lets you fetch email from a single mailbox. You can’t create multiple folders on the server to organize your mails. It saves space on the mail server because the email usually gets moved to the user’s hard disk on their computer. This variant is pretty antiquated.
|
||||
- _IMAP_ (Internet Messaging Application Protocol) is predominantly focused upon leaving your mail on the server. The inbox is where your incoming emails are stored but users can also maintain folders and move emails to them. But users can move emails to different sub-folders. IMAP is useful when you want to access your email from different locations without losing mail because you fetched it from another location. The drawback is that lazy users leave their mail on the server thus filling up your server’s hard disk – for which quotas are a good solution.
|
||||
|
|
@ -0,0 +1,377 @@
|
|||
---
|
||||
title: Relaying outgoing emails through Postfix
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/relaying-outgoing-emails-through-postfix
|
||||
sidebar:
|
||||
order: 180
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
## Relaying
|
||||
|
||||
Your mail server is almost ready for use. But one puzzle piece is missing. Your users can receive emails but they cannot send them yet.
|
||||
|
||||
Please note that there is a difference on how **users** send emails versus how **servers** send emails. For comparison:
|
||||
|
||||
- A **mail server** fetches the _MX_ record for the domain name of the recipient’s email address. That tells it which mail server to talk to. Then it opens an SMTP connection (TCP port 25) and sends the email.
|
||||
- An **end user** with a mail client like Thunderbird, Evolution or Mutt cannot do it this way. The mail clients have no functionality built in for that _MX_ record fetching magic. And the user is most likely on a _dynamic IP address_ that other mail servers do not trust and reject. End users are meant to send emails to their provider’s (your!) mail server, send login information to authenticate themselves and then send the email. This is called _relaying_ because your mail server acts as a _relay_ between the user and other mail servers on the internet. For security reasons the user also has to authenticate to be allowed to send emails.
|
||||
|
||||
I created a couple of illustrations to explain it.
|
||||
|
||||
### Incoming email
|
||||
|
||||
When someone on the internet sends an email to `john@example.org`, some other mail server will deliver the email using SMTP to your mail server. Postfix will determine that it’s responsible for email addresses in the example.org domain and accept the email. John can then get the email from your server.
|
||||
|
||||

|
||||
|
||||
### Outgoing email (without authentication)
|
||||
|
||||
John is on the internet somewhere and wants to send an email to lisa@example.com. Your mail server is not responsible for the “example.com” domain so it receives John’s email and would have to forward (relay) it to the actual mail server that is responsible for …@example.com email addresses. This may seem like a harmless scenario but your mail server must deny that:
|
||||
|
||||

|
||||
|
||||
Why? Because anyone can claim to be John and make your mail server forward mail. If an attacker (like a spammer) would send millions of spam emails in John’s name through your server then other organisations will accuse _you_ as the operator of the mail server of spamming. Your mail server would be what people call an _[open relay](https://en.wikipedia.org/wiki/Open_mail_relay)_. That is not what you want because your mail server would get blacklisted and you will not be able to send out mail to most other servers. So without any proof that the sender is indeed John, your server must reject the email.
|
||||
|
||||
|
||||
### Outgoing email (with authentication)
|
||||
|
||||
So how does John prove his identity? He needs to use _authenticated_ SMTP. This is similar to the previous case but John’s email program will also send his username and password.
|
||||
|
||||

|
||||
|
||||
We are making sure that his authentication happens over an encrypted connection so John’s password is safe.
|
||||
|
||||
## Postfix setting “mynetworks”
|
||||
|
||||
In addition to using SMTP authentication you can tell Postfix to always allow relaying for certain IP addresses. The _mynetworks_ setting contains the list of IP networks or IP addresses that you trust. Usually you define your own local network here. The reason John had to authenticate in the above example is because his IP address is not part of _mynetworks_.
|
||||
|
||||
## Make Postfix use Dovecot for authentication
|
||||
|
||||
Enabling SMTP authentication in Postfix is surprisingly easy. You already configured Dovecot so it knows all about your users from the SQL database. So let’s just make Postfix use that by telling it to ask the Dovecot server to verify the username and password. Postfix just needs some extra configuration. Run these commands on the shell:
|
||||
```
|
||||
postconf smtpd_sasl_type=dovecot
|
||||
postconf smtpd_sasl_path=private/auth
|
||||
postconf smtpd_sasl_auth_enable=yes
|
||||
```
|
||||
This enables SMTP authentication and tells Postfix that it can talk to Dovecot through a socket file located at `/var/spool/postfix/private/auth`. Do you remember that Postfix runs in a sandboxed _chroot_ directory? That’s at /var/spool/postfix. It cannot access any files outside of that directory. But fortunately in a previous section you edited the /etc/dovecot/conf.d/10-master.conf file and made Dovecot place a socket file into `/var/spool/postfix/private/auth` to allow communication from Postfix.
|
||||
|
||||
## Enable encryption
|
||||
|
||||
The following settings enable encryption, set the key and certificate paths for Postfix. Just run these commands:
|
||||
```
|
||||
postconf smtpd_tls_security_level=may
|
||||
postconf smtpd_tls_auth_only=yes
|
||||
postconf smtpd_tls_cert_file=/etc/letsencrypt/live/webmail.example.org/fullchain.pem
|
||||
postconf smtpd_tls_key_file=/etc/letsencrypt/live/webmail.example.org/privkey.pem
|
||||
postconf smtp_tls_security_level=may
|
||||
```
|
||||
|
||||
|
||||
<Aside type="tip" title="smtp or smtpd?">
|
||||
Look closely. Some settings start with “smtp\_” and others with “smtpd\_”. That is not a typo. “smtp\_” refers to the SMTP _client_. That is the component that sends out emails **from** Postfix **to** other servers.
|
||||
|
||||
Whereas “smtpd\_” means the SMTP _server._ That in turn is the component that **receives** emails **from** other systems – either from a remote mail server or one of your users.
|
||||
</Aside>
|
||||
|
||||
The above settings allow encrypted incoming (smtpd\_) and outgoing (smtp\_) connections. But they do not enforce it. If a remote mail server does not have encryption enabled we will still accept their emails. You may be tempted to enforce encryption but that would reject email communication with servers who have been configured without encryption.
|
||||
|
||||
However the `smtpd_tls_auth_only=yes` setting makes sure that the user’s authentication information (email address and password) are always encrypted between the user and your mail server.
|
||||
|
||||
|
||||
<Aside type="tip" title="Restrictions">
|
||||
In the past you may have configured your _smtpd\_recipient\_restrictions_ to restrict relaying to authenticated users. Postfix nowadays has setting called “[smtpd\_relay\_restrictions](http://www.postfix.org/postconf.5.html#smtpd_relay_restrictions)” that deals with relaying requests in the “RCPT TO” phase of the SMTP dialog. So essentially it works like the good old “smtpd\_recipient\_restrictions” but is checked first. _smtpd\_relay\_restrictions_ has a reasonable default so authenticated relaying works automatically:
|
||||
|
||||
`smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination`
|
||||
</Aside>
|
||||
|
||||
### How SMTP authentication works
|
||||
|
||||
Are you curious how SMTP authentication looks on a low level? You probably are not. But let’s do it anyway. Just once, so that you get the idea.
|
||||
|
||||
In previous versions of this guide we used “telnet” to connect to TCP port 25 and speak SMTP. But now we enforce encryption and can’t do SMTP authentication unencrypted. Let’s try it anyway:
|
||||
|
||||
```
|
||||
telnet localhost smtp
|
||||
```
|
||||
|
||||
The server will let you in:
|
||||
|
||||
```
|
||||
Trying 127.0.0.1…
|
||||
Connected to localhost.
|
||||
Escape character is '^\]'.
|
||||
220 webmail ESMTP Postfix (Debian/GNU)
|
||||
```
|
||||
|
||||
Say hello:
|
||||
|
||||
```
|
||||
ehlo example.com
|
||||
```
|
||||
|
||||
Postfix will present a list of features that are available for you:
|
||||
|
||||
```
|
||||
250-mailtest
|
||||
250-PIPELINING
|
||||
250-SIZE 10240000
|
||||
250-VRFY
|
||||
250-ETRN
|
||||
250-STARTTLS
|
||||
250-ENHANCEDSTATUSCODES
|
||||
250-8BITMIME
|
||||
250-DSN
|
||||
250-SMTPUTF8
|
||||
250 CHUNKING
|
||||
```
|
||||
|
||||
Let me briefly explain what these lines mean:
|
||||
|
||||
- PIPELINING
|
||||
This is a feature to speed up SMTP communication. Usually the remote system has to wait for a response to every command it sends. Pipelining allows the remote server to send several commands in a batch without waiting for a response. Postfix will just store these commands and execute them one by one. If you told Postfix to forbid pipelining it would disconnect the remote server when it tries to send bulks of commands without waiting for the proper reply. It is mainly a feature against rogue senders.
|
||||
- SIZE 10240000
|
||||
The remote server is allowed to send emails up to 10 MB large. This has long been a common maximum size for emails. However nowadays 40 MB or even more are more common sizes because emails have grown larger. Some mail servers even allow 100 MB.
|
||||
- VRFY
|
||||
Allows remote servers to verify a given name or email address. For example the remote server could send “VRFY john” and your server might respond `250 John Doe <john@example.org>`. It can be used to verify that a certain recipient email address is deliverable
|
||||
- ETRN
|
||||
A command that a remote system can send to flush the Postfix queue of mails for a certain domain. It can be used if the remote system had technical problems and failed to receive email for a while. Then it could send an ETRN command to make your server start sending outstanding emails for that domain. It is rarely used.
|
||||
- STARTTLS
|
||||
This tells the remote system that it might start switching from this unencrypted to an encrypted connection by sending the “STARTTLS” command. It will then start negotiating a TLS-encrypted connection. You could compare it to an HTTP connection that suddenly switches over to an encrypted HTTPS connection. The advantage is that you can start talking SMTP on TCP port 25 and don’t have to open up a second TCP port like 465 which is the “SSMTP” (secure SMTP) port and only accepts encrypted connections.
|
||||
- ENHANCEDSTATUSCODES
|
||||
This enables more three-digit return codes for various conditions. See the [RFC2034](http://tools.ietf.org/html/rfc2034) if you are curious.
|
||||
- 8BITMIME
|
||||
In ancient times SMTP only processed 7-bit characters. You couldn’t transfer special characters like “Ä” or “ß” without special encoding. 8BITMIME allows a transmission of emails using 8-bit characters. Still many emails are specially encoded using ISO8859-1 or UTF-8.
|
||||
- DSN
|
||||
It enables DSNs (delivery status notofications) that allows the sender to control the messages that Postfix creates when an email could not be delivered as intended.
|
||||
- SMTPUTF8
|
||||
In addition to 8BITMIME you can use UTF8 encoded characters in header fields.
|
||||
- CHUNKING
|
||||
This feature (described in [RFC 3030](https://tools.ietf.org/html/rfc3030)) makes sending of large emails more efficient.
|
||||
|
||||
However one important line is missing here that would allow us to send our username and password:
|
||||
|
||||
```
|
||||
250-AUTH PLAIN LOGIN
|
||||
```
|
||||
|
||||
We told Postfix to only allow authentication when the connection is encrypted. So we are not offered authentication over this plain text connection. That’s what we want.
|
||||
|
||||
Are you still connected? Okay, good. So we need an encrypted connection using TLS. Using the STARTTLS feature we can switch over from unencrypted to encrypted without having to reconnect. Enter…
|
||||
|
||||
```
|
||||
STARTTLS
|
||||
```
|
||||
|
||||
And the server replies:
|
||||
|
||||
```
|
||||
220 2.0.0 Ready to start TLS
|
||||
```
|
||||
|
||||
However now it’s getting weird because you would have to speak TLS encryption which is not a language that humans speak. So let’s quit this using the “QUIT” command and do that differently:
|
||||
|
||||
```
|
||||
QUIT
|
||||
```
|
||||
|
||||
We can use OpenSSL to help us with the decryption. Run:
|
||||
|
||||
```
|
||||
openssl s_client -connect localhost:25 -starttls smtp
|
||||
```
|
||||
|
||||
You will see a lot of output. OpenSSL has connected to TCP port 25 and issued a STARTTLS command to switch to an encrypted connection. So whatever you type now will get encrypted. Enter:
|
||||
|
||||
```
|
||||
EHLO example.com
|
||||
```
|
||||
|
||||
And Postfix will send a list of capabilities that will look like this:
|
||||
|
||||
```
|
||||
250-PIPELINING
|
||||
250-SIZE 10240000
|
||||
250-VRFY
|
||||
250-ETRN
|
||||
250-AUTH PLAIN LOGIN
|
||||
250-ENHANCEDSTATUSCODES
|
||||
250-8BITMIME
|
||||
250 DSN
|
||||
```
|
||||
|
||||
And now that we are using an encrypted connection Postfix offers us to authenticate. So let us send the authentication string with a Base64-encoded password:
|
||||
|
||||
```
|
||||
AUTH PLAIN AGpvaG5AZXhhbXBsZS5vcmcAc3VtbWVyc3Vu
|
||||
```
|
||||
|
||||
Unless you have changed John’s password to something else than “summersun” the server should accept that authentication:
|
||||
|
||||
```
|
||||
235 2.7.0 Authentication successful
|
||||
```
|
||||
|
||||
Excellent. You are logged in through SMTP. You could now send an email to be forwarded to another mail server. I just wanted to show that authentication works if you use an encrypted connection.
|
||||
|
||||
Disconnect from Postfix:
|
||||
|
||||
```
|
||||
QUIT
|
||||
```
|
||||
|
||||
Authentication works. Well done.
|
||||
|
||||
## Base64-encoded passwords
|
||||
|
||||
Wait a minute. What was that crazy cryptic string? There was no username and password in it. Was it encrypted?
|
||||
|
||||
No, that was no encryption. It was merely an _encoded_ version of the username and password. Why that? Well usually in SMTP you can only transfer ASCII characters. But the password may contain special characters which are not covered by ASCII. So in the _PLAIN_ method that information is [Base64](https://en.wikipedia.org/wiki/Base64) encoded.
|
||||
|
||||
What is actually converted to Base64…
|
||||
|
||||
```
|
||||
NULL-BYTE + USERNAME + NULL-BYTE + PASSWORD
|
||||
```
|
||||
|
||||
So for John’s case you can easily create the Base64 string using:
|
||||
|
||||
```
|
||||
printf '\0john@example.org\0summersun' | base64
|
||||
```
|
||||
|
||||
As a result you will get the exact same string you used above with “AUTH PLAIN”.
|
||||
|
||||
## The submission port
|
||||
|
||||
Although I have been talking about SMTP on port 25 to relay mails, it is actually not the proper way. End users should not use port 25 but rather the _submission_ service on TCP port 587 (as described in [RFC 4409](https://tools.ietf.org/html/rfc4409)). The idea is to use port 25 for transporting email (_MTA = mail transport agent_) from server to server and port 587 for submitting (_MSA = mail submission agent_) email from a user to a mail server.
|
||||
|
||||
For comparison:
|
||||
|
||||
| TCP Port | 25 | 587 |
|
||||
| -------------- | ------------------- | ----------------------- |
|
||||
| Service name | SMTP | Submission |
|
||||
| Encryption | Optional | Mandatory |
|
||||
| Authentication | Optional | Mandatory |
|
||||
| Meant for | Server-to-Server | User-to-Server |
|
||||
| Your ISP | may block this port | should allows this port |
|
||||
|
||||
I hope that makes the distinction a bit clearer. So let’s enable the _submission_ service. All Postfix services are declared in the `/etc/postfix/master.cf` file. Please edit the file and look for the _submission_ section that is commented out by default. Turn that section into the following. Basically I removed the ‘#’ character on all lines of this section and removed the lines with the mua\_\* variables.
|
||||
|
||||
```
|
||||
submission inet n - y - - smtpd
|
||||
-o syslog_name=postfix/submission
|
||||
-o smtpd_tls_security_level=encrypt
|
||||
-o smtpd_sasl_auth_enable=yes
|
||||
-o smtpd_tls_auth_only=yes
|
||||
-o smtpd_reject_unlisted_recipient=no
|
||||
-o smtpd_client_restrictions=
|
||||
-o smtpd_helo_restrictions=
|
||||
-o smtpd_sender_restrictions=
|
||||
-o smtpd_relay_restrictions=
|
||||
-o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
|
||||
-o milter_macro_daemon_name=ORIGINATING
|
||||
```
|
||||
|
||||
Make sure to start the first line in the first column and indent the following lines.
|
||||
|
||||
This service uses the “smtpd” daemon (see the last word of the first line) which is the piece of software that responds if you open an SMTP connection on TCP port 25. But it gets a few extra options set…
|
||||
|
||||
- in the /var/log/mail.log log file you will see the connections to the submission port as “postfix/submission” (the _syslog\_name_ setting above)
|
||||
- enforce encryption on this port (_smtpd\_tls\_security\_level_)
|
||||
- enable authentication (_smtpd\_sasl\_auth\_enable_)
|
||||
- enforce encryption during authentication (_smtpd\_tls\_auth\_only_)
|
||||
- allow sending emails to recipients outside of this mail server (_smtpd\_reject\_unlisted\_recipient_)
|
||||
- remove special restrictions (_smtpd\_\*\_restrictions_)
|
||||
- allow relaying if the sender was authenticated (_smtpd\_recipient\_restrictions_)
|
||||
- send the string _ORIGINATING_ to milter services (_milter\_macro\_daemon\_name_) – you can just leave it like that
|
||||
|
||||
Restart the Postfix server:
|
||||
|
||||
```
|
||||
systemctl restart postfix
|
||||
```
|
||||
|
||||
Your users can now use the _submission_ port to send email. They just use the port 587 in their mail clients instead of port 25.
|
||||
|
||||
Feel free to send a test mail using the submission port and encryption. You will need to install the libnet-ssleay-perl package first, so that SWAKS can speak TLS:
|
||||
|
||||
```
|
||||
apt install -y libnet-ssleay-perl
|
||||
swaks --server localhost --to john@example.org \
|
||||
--port 587 -tls \
|
||||
--auth-user john@example.org \
|
||||
--auth-password summersun
|
||||
```
|
||||
|
||||
<Aside type="tip" title="What is Port 465?">
|
||||
This TCP port belongs to the “submission over TLS” service. It is used for the submission service but expects an encrypted connection from the first byte. This port is hardly ever used so you don’t have to care about it. The submission service you just configured is also encrypted but uses the STARTTLS mechanism to switch to a TLS connection after the welcome message.
|
||||
</Aside>
|
||||
|
||||
## Protecting against forged sender addresses
|
||||
|
||||
A “forged” sender address means that someone claims to be someone else. Let’s say that John has authenticated and the mail server trusts him. Nothing keeps John from impersonating someone else and sending email in his name? Most email service providers have restrictions that you can only send emails if the sender matches your actual email address. Let’s do that, too.
|
||||
|
||||
Postfix provides a setting called [smtpd\_sender\_login\_maps](http://www.postfix.org/postconf.5.html#smtpd_sender_login_maps) for that purpose. From the “maps” part you can deduce that it expects a _mapping_ again. This time the two columns mean…
|
||||
|
||||
- Left column: who you claim to be (email’s sender address)
|
||||
- Right column: who you logged in as (user name after authentiation)
|
||||
|
||||
As we use the email address also as a user name we simply need a mapping where the email address is listed on both sides. Fortunately we already have a mapping like that: our _email2email_ mapping that we used earlier as a workaround for catchall forwardings. Let’s reuse it. Please run…
|
||||
|
||||
```
|
||||
postconf smtpd_sender_login_maps=mysql:/etc/postfix/mysql-email2email.cf
|
||||
```
|
||||
|
||||
This sets the parameter both for the SMTP port (25) and the submission port (587). Defining these maps is not enough though. You also need to make Postfix act on this. Edit the /etc/postfix/master.cf again and in the _submission_ section add the following option. Make sure the line is indented like all other options:
|
||||
|
||||
```
|
||||
-o smtpd_sender_restrictions=reject_sender_login_mismatch,permit_sasl_authenticated,reject
|
||||
```
|
||||
|
||||
Restart Postfix after this change:
|
||||
|
||||
```
|
||||
systemctl restart postfix
|
||||
```
|
||||
|
||||
You can now try to send email as a different user than you are logged in. Let’s us swaks again to send a spoofed email:
|
||||
|
||||
```
|
||||
swaks --server localhost --from john@example.com \
|
||||
--to john@example.org --port 587 -tls \
|
||||
--auth-user john@example.org \
|
||||
--auth-password summersun
|
||||
```
|
||||
|
||||
Please note that John’s actual email address is **not** john@example.**com** but john@example.**org**. So it must be rejected. The server should tell you:
|
||||
|
||||
```
|
||||
~> MAIL FROM:john@example.com
|
||||
<~ 250 2.1.0 Ok
|
||||
~> RCPT TO:john@example.org
|
||||
<~ * 553 5.7.1 john@example.com: Sender address rejected: not owned by user john@example.org
|
||||
```
|
||||
|
||||
Sorry if the mixture of example.org and example.com in my examples makes this guide less readable. But I can’t use any real world domains here. Please just note that sometimes it’s “.com” and sometimes “.org”.
|
||||
|
||||
Of course you can test your new security feature in the Roundcube web mail interface as well. In Settings/Identities you can create another sender email address as in:
|
||||
|
||||

|
||||
|
||||
With this wrong sender address any email should get rejected. Your mail.log would read something like:
|
||||
|
||||
```
|
||||
NOQUEUE: reject: RCPT from foo.bar\[…\]:
|
||||
553 5.7.1 <someoneelse@example.com>:
|
||||
Sender address rejected:
|
||||
not owned by user john@example.org;
|
||||
from=<someoneelse@example.com> to=<…>
|
||||
```
|
||||
|
||||
|
||||
<Aside type="tip" title="Allow aliases?">
|
||||
If you want to allow users to send as one of their aliases you could create a new \*.cf file with a mapping query like this:
|
||||
|
||||
`SELECT email FROM virtual_users WHERE email='%s' UNION SELECT destination FROM virtual_aliases WHERE source='%s'`
|
||||
</Aside>
|
||||
590
src/content/docs/ispmail-bookworm/190-catching-spam.mdx
Normal file
|
|
@ -0,0 +1,590 @@
|
|||
---
|
||||
title: Catching spam with rspamd
|
||||
lastUpdated: 2024-12-21
|
||||
slug: ispmail-bookworm/catching-spam-with-rspamd
|
||||
sidebar:
|
||||
order: 190
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
<Aside type="tip" title="Optional feature">
|
||||
This feature is completely optional. If you are eager to get finished then skip this page and maybe come back later.
|
||||
</Aside>
|
||||
|
||||
You have come a long way in this guide and your mail server is already fully functional. Now it’s time to deal with the dark side: spam. And there will be lots of it. So we need to detect spam emails and filter them out. I found that [rspamd](https://rspamd.com/) is a well-performing choice for that purpose both in speed and detection. rspamd keeps a permanent process running on your mail server that listens to connections from Postfix using the [milter](http://www.postfix.org/MILTER_README.html) (=**m**ail f**ilter**) protocol. Every time an email enters your system, Postfix will send it to rspamd to have its content checked. rspamd runs a lot of checks on the email and computes a total score. The higher the score – the more likely the email should be considered unsolicited.
|
||||
|
||||
## How free is rspamd?
|
||||
|
||||
rspamd is quite effective against spam thanks to its many kinds of checks that incoming emails run through. Several of these checks rely on services provided by the rspamd project. There are web sites like email.rspamd.com, uribl.rspamd.com and maps.rspamd.com that provide information used by the default rspamd installation. That data is not freely available and there are no public mirrors.
|
||||
|
||||
You are only allowed to use their data under certain conditions described in rspamd’s [usage policy](https://rspamd.com/doc/usage_policy.html). Spoiler: heavy or commercial usage will get you blocked. So it may be wise to disable all features that include his email and URL blacklists and the “fuzzy” feature.
|
||||
|
||||
rspamd is being packaged and shipped by the Debian project. However the rspamd principal developer builds and endorses his own packages and has a history of getting angry and offensive against anyone using the packages shipped by Debian. I find that attitude a bit suspicious and rather stay with the Debian packages that have been built by a process that I trust.
|
||||
|
||||
## Make Postfix use rspamd
|
||||
|
||||
Let’s tell Postfix to send all incoming email through rspamd. Run these commands on the shell:
|
||||
|
||||
```
|
||||
postconf smtpd_milters=inet:127.0.0.1:11332
|
||||
postconf non_smtpd_milters=inet:127.0.0.1:11332
|
||||
postconf milter_mail_macros="i {mail_addr} {client_addr} {client_name} {auth_authen}"
|
||||
```
|
||||
|
||||
Postfix will connect to rspamd that is listening to TCP port 11332 on localhost (127.0.0.1) and send the email through that connection. The _smtpd\_milters_ setting defines that connection for emails that came into the system from the internet via the SMTP protocol. The _non\_smtpd\_milters_ setting is optional – it makes Postfix have all emails checked that originate from the system itself. Finally the _milter\_mail\_macros_ setting defines several variables that rspamd expects for better spam detection. rspamd then runs its checks and tells Postfix whether the email should pass or get rejected.
|
||||
|
||||
## Testing spam detection
|
||||
|
||||
For testing we can use a sample spam email that comes with SpamAssassin. It is called GTUBE (Generic Test for Unsolicited Bulk Email). It contains a certain artificial pattern that is recognized as spam by SpamAssassin. Do you know EICAR.COM to test virus scanners? This is the same thing for spam.
|
||||
|
||||
I suggest that you download the file on the server:
|
||||
|
||||
```
|
||||
wget http://spamassassin.apache.org/gtube/gtube.txt
|
||||
```
|
||||
|
||||
…and send that email to our test user John…
|
||||
|
||||
```
|
||||
sendmail john@example.org < gtube.txt
|
||||
```
|
||||
|
||||
Check your /var/log/mail.log. You will find something like this:
|
||||
|
||||
```
|
||||
Jan 13 09:21:01 mail postfix/cleanup[19945]: 95B7A42212: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 5.7.1 Gtube pattern; from=<root@mail.example.org> to=<john@example.org>
|
||||
|
||||
Jan 13 09:21:01 jen postfix/cleanup[19945]: 95B7A42212: to=<john@example.org>, relay=none, delay=0.18, delays=0.18/0/0/0, dsn=5.7.1, status=bounced (Gtube pattern)
|
||||
```
|
||||
|
||||
“milter-reject” tells that the milter (rspamd) recommended to Postfix to reject the email. It gave the reason “5.7.1 Gtube pattern”. In mail communication you often find these three digit codes. They are defined in [RFC 3463](https://tools.ietf.org/html/rfc3463). The first digit is most important:
|
||||
|
||||
- 2 = Success
|
||||
- 4 = Transient failure (temporary problem – come back later)
|
||||
- 5 = Permanent failure (do not try again in this way)
|
||||
|
||||
So 5.7.1 tells us that the result code is a permanent failure in delivery. If you looked up the RFC you would find that the 7 stands for an issue regarding the security policy. So it’s not a technical failure but instead a security-relevant component on the system has rejected the email. That’s what rspamd did. It even told us the reason: “Gtube pattern”. So you see that rspamd knows about the Gtube spam test pattern and reacts as expected.
|
||||
|
||||
Accordingly you won’t see this email in John’s inbox. This is a great advantage of using milters. Imagine Postfix receiving a spam email and confirm its reception. What should it do when it finds out that it’s unwanted email? According to the SMTP protocol it must not throw away any emails. Would you create a bounce message telling the sender that you did not accept the email? That would be a bad idea because the sender address in spam emails is very likely forged. You would send the bounce to an innocent person thus creating so called _backscatter_ and make it even worse. So the better approach is to check the email while the sending server is still connected to your Postfix. This allows Postfix to reject the email with a 5.x.x error code and let the other side figure out what to do.
|
||||
|
||||
## Score metrics
|
||||
|
||||
rspamd will however not reject all spam email. It computes a score that tells how likely a certain email is spam. You can tell it which scores you would accept, flag as spam or make the incoming email get rejected. Rspamd has a large ruleset that checks incoming emails in various ways and adds to the score. Take a look at the `/etc/rspamd/actions.conf` file:
|
||||
|
||||
```
|
||||
actions {
|
||||
reject = 15;
|
||||
add_header = 6;
|
||||
greylist = 4;
|
||||
}
|
||||
```
|
||||
|
||||
These are the default actions. If rspamd computes a score of at least 15 then the email will get rejected at the doorstep just like the _Gtube pattern_ in the previous test. Any other score above 6 will add a line “X-Spam: Yes” so that your mail software can detect them and maybe file the email to a different folder. And any other score above 4 will trigger _greylisting_ which is a mechanism that temporarily rejects the email with a 4.x.x code and waits if the sending server will try again. After a waiting time of a few minutes greylisting will accept the email. The idea is to reject email from systems that do not have a sending queue. Malware like on infected Wind\*ws computers used to try sending an email just once which triggered greylisting and successfully rejected the spammer. But even malware programmers have learned and may try again after a few minutes thus circumventing greylisting. Your mileage may vary. The problem with greylisting is that the recipient has to wait a couple of minutes for the email to be delivered which is often bothering the users.
|
||||
|
||||
If you like to change these defaults then create a new file in `/etc/rspamd/local.d/actions.conf` containing your desired limits:
|
||||
|
||||
```
|
||||
reject = 150;
|
||||
add_header = 6;
|
||||
greylist = 4;
|
||||
```
|
||||
|
||||
This would virtually never reject an email. The other two values are pretty sane defaults. I personally use this setting all the time so that users can find spam in their _Junk_ folder but don’t have to ask me if the mail server rejected it.
|
||||
|
||||
Please take a moment to understand how to change rspamd defaults. You can either create files in `/etc/rspamd/override.d/…` which will replace entire sections; or create files in `/etc/rspamd/local.d/…` which will change only parts of the configuration. There is a [helpful page in the rspamd documentation](https://rspamd.com/doc/developers/writing_rules.html) that contains examples. Whatever you do – never change the /etc/rspamd/\* files directly because a software update will try to replace them.
|
||||
|
||||
Restart rspamd after any configuration changes:
|
||||
|
||||
```
|
||||
systemctl restart rspamd
|
||||
```
|
||||
|
||||
To check if rspamd has picked up your configuration use this command to see the current configuration:
|
||||
|
||||
```
|
||||
rspamadm configdump
|
||||
```
|
||||
|
||||
You may test your configuration using
|
||||
|
||||
```
|
||||
rspamadm configtest
|
||||
```
|
||||
|
||||
Alternatively you may check if all required rspamd processes are running…
|
||||
|
||||
|
||||
```
|
||||
pgrep -a rspam
|
||||
|
||||
141693 rspamd: main process
|
||||
141694 rspamd: rspamd\_proxy process (localhost:11332)
|
||||
141695 rspamd: controller process (localhost:11334)
|
||||
104965 rspamd: normal process (localhost:11333)
|
||||
104966 rspamd: normal process (localhost:11333)
|
||||
```
|
||||
|
||||
## Adding headers
|
||||
|
||||
As you may know an email consists of the headers and the body. Your users will usually only see common header information like the _subject_, the _sender_, the _recipient_ and the _date and time_ the email was sent. But there is way more information like the route the email traveled or extended headers added by the various mail server on the way to the destination. Such extended headers begin with an “X-“. rspamd can add such headers to help you filter out spam. For that purpose create a new configuration override file at `/etc/rspamd/override.d/milter_headers.conf` with this content:
|
||||
|
||||
```
|
||||
extended_spam_headers = true;
|
||||
```
|
||||
|
||||
Again restart rspamd:
|
||||
|
||||
```
|
||||
systemctl restart rspamd
|
||||
```
|
||||
|
||||
As [documented](https://rspamd.com/doc/modules/milter_headers.html) it will add these headers:
|
||||
|
||||
```
|
||||
X-Rspamd-Server: mail
|
||||
Authentication-Results: dmarc=fail reason="No valid SPF, No valid DKIM" …
|
||||
X-Rspamd-Queue-Id: C22E55A005B3
|
||||
X-Spamd-Result: default: False [11.55 / 15.00]
|
||||
R_PARTS_DIFFER(0.27)[63.4%]
|
||||
FROM_NO_DN(0.00)[]
|
||||
RCVD_COUNT_ZERO(0.00)[0]
|
||||
R_DKIM_NA(0.00)[]
|
||||
FUZZY_DENIED(12.00)[1:19305c7fdd:1.00:bin,1:35699594fd:0.91:txt]
|
||||
RBL_SENDERSCORE(2.00)[55.181.23.94.bl.score.senderscore.com]
|
||||
ARC_NA(0.00)[]
|
||||
RCPT_COUNT_ONE(0.00)[1]
|
||||
RCVD_TLS_ALL(0.00)[]
|
||||
FROM_EQ_ENVFROM(0.00)[]
|
||||
R_SPF_SOFTFAIL(0.00)[~all]
|
||||
BAYES_HAM(-2.71)[98.75%]
|
||||
TO_MATCH_ENVRCPT_ALL(0.00)[]
|
||||
MIME_GOOD(-0.10)[multipart/related,multipart/alternative,text/plain]
|
||||
MID_RHS_MATCH_FROM(0.00)[]
|
||||
ASN(0.00)[asn:16276, ipnet:94.23.0.0/16, country:FR]
|
||||
TO_DN_NONE(0.00)[]
|
||||
DMARC_POLICY_SOFTFAIL(0.10)[Chronopost.fr : No valid SPF, No valid DKIM,none]
|
||||
SUBJECT_ENDS_EXCLAIM(0.00)[]
|
||||
X-Spam: Yes
|
||||
```
|
||||
|
||||
<Aside type="tip" title="No headers?">
|
||||
Please note that rspamd only adds headers if an email is received from outside your server. So if you use _swaks_ locally then you will not get anything.
|
||||
</Aside>
|
||||
|
||||
Each of the uppercase symbols like _FROM\_HAS\_DN_ means that a certain detection rule of rspamd was triggered. It does not necessarily mean something bad about the email. For example _R\_SPF\_ALLOW_ has a negative score that lowers the total score because it is something good about the email. There are a several symbols with a 0.00 score. These do not change the score but show you what rspamd has found. But if you consider certain criteria good or bad then you can define your own scores for them.
|
||||
|
||||
The last line here is especially interesting because next on our list is…
|
||||
|
||||
## Sending spam to the Junk folder
|
||||
|
||||
Your users will not realize that their spam emails have an added “X-Spam: Yes” header. It is not actively shown in their mail client. Nor does it move the email out of the inbox into their spam folder. Such emails just appear like normal in their inbox. So let’s aid them by moving spam to a separate _Junk_ folder beneath their inbox automatically. Dovecot has support for [Sieve](https://en.wikipedia.org/wiki/Sieve_\(mail_filtering_language\)) filters which are scripts that run automatically whenever an email comes in.
|
||||
|
||||
John could create a new Sieve filter (e.g. using the Roundcube webmail interface) for himself that would save any emails to his “Junk” folder if the header line “X-Spam: Yes” was found. This rule would be useful for all your users though so let’s find a more general solution.
|
||||
|
||||
Dovecot supports _global_ Sieve filters that apply to all users. Edit the file `/etc/dovecot/conf.d/90-sieve.conf`. Look for the “sieve\_after” lines. They are commented out. Add a new line there:
|
||||
|
||||
```
|
||||
sieve_after = /etc/dovecot/sieve-after
|
||||
```
|
||||
|
||||
And restart Dovecot:
|
||||
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
|
||||
The “_sieve after_” filters are executed after the users’ filters. John can define his own filter rules. And after that Dovecot will run any filter rules it finds in files in `/etc/dovecot/sieve-after`. That is just an arbitrary directory that you create:
|
||||
|
||||
```
|
||||
mkdir /etc/dovecot/sieve-after
|
||||
```
|
||||
|
||||
And add a new file `/etc/dovecot/sieve-after/spam-to-folder.sieve` reading:
|
||||
|
||||
```
|
||||
require ["fileinto"];
|
||||
|
||||
if header :contains "X-Spam" "Yes" {
|
||||
fileinto "Junk";
|
||||
stop;
|
||||
}
|
||||
```
|
||||
|
||||
The “require” lines include functionality to move emails into certain folders (fileinto) and to create folders if they don’t exist yet (mailbox). Then if rspamd marked an email as spam it gets moved into the INBOX.Junk folder which just appears as “Junk” to the user underneath their inbox.
|
||||
|
||||
Dovecot cannot deal with such human-readable files though. So we need to compile it:
|
||||
|
||||
```
|
||||
sievec /etc/dovecot/sieve-after/spam-to-folder.sieve
|
||||
```
|
||||
|
||||
That generated a machine-readable file /etc/dovecot/sieve-after/spam-to-folder.svbin.
|
||||
|
||||
Now all your users will automatically get spam emails moved to their Junk folder. Nice – isn’t it?
|
||||
|
||||
## About Redis
|
||||
|
||||
Many features in Rspamd use [Redis](https://redis.io/) to persist their data. Let me give you a quick explanation what Redis is.
|
||||
|
||||
Redis is a kind of database system. It is way more limited than a traditional SQL database because it just stores keys and values. There aren’t several fields/columns like in SQL. But it is lightning fast the way it works. On my aged server it handles around 50,000 requests per second. It gets it speed from its simplicity and from keeping the data in RAM. So it doesn’t access the disk to fetch information. (But it copies its data to disk frequently to prevent data loss.) People use Redis as a cache or for very fast lookups of simple data structures. And so [does rspamd](https://rspamd.com/doc/configuration/redis.html).
|
||||
|
||||
You installed the “redis-server” package earlier. And that’s all you needed to do. It started automatically and listens ton incoming connections on TCP port 6379 on localhost. In Rspamd the Redis backend is enabled by default. You just have to tell it the IP address of your Redis server. Add a file `/etc/rspamd/override.d/redis.conf` and insert:
|
||||
|
||||
```
|
||||
servers = "127.0.0.1";
|
||||
```
|
||||
|
||||
Restart rspamd and you are done.
|
||||
|
||||
```
|
||||
systemctl restart rspamd
|
||||
```
|
||||
|
||||
Feel free to use Redis for [other lookups](https://rspamd.com/doc/configuration/redis.html), too.
|
||||
|
||||
|
||||
## Training the spam detection
|
||||
|
||||
One of rspamd’s features is analyzing word patterns using probability theory. That functionality is contained in its “[statistical module](https://rspamd.com/doc/configuration/statistic.html)“. (Yes, the name is pretty misleading.) Essentially you show rspamd lots of ham (good) and spam (bad) emails and its detection gets better over time.
|
||||
|
||||
<Aside type="tip" title="Redis replaces SQLite">
|
||||
Rspamd recommends (and defaults to) using Redis to store spam training data. That’s what we will use now.
|
||||
</Aside>
|
||||
|
||||
### (a) Auto-learning
|
||||
|
||||
You can start with an empty training database. This is not as bad as it sounds. rspamd has way more functionality to determine if an email is ham or spam. Autolearning takes email that are likely ham or spam and uses them to train the spam filter. The [rspamd documentation](https://rspamd.com/doc/configuration/statistic.html) has further examples how to fine-tune auto learning. After a few hundred emails the training will contribute towards a better detection rate.
|
||||
|
||||
If you want to use _autolearning_ just create a new file `/etc/rspamd/override.d/classifier-bayes.conf` and make it contain:
|
||||
|
||||
```
|
||||
autolearn = [-5, 10];
|
||||
```
|
||||
|
||||
That will train emails with a spam score of less than -5 as ham (good). And emails with a spam score of more than 10 as spam (unwanted). Feel free to chose other values.
|
||||
|
||||
### (b) Migrating training data from previous mail server
|
||||
|
||||
Have you used the old SQLite-based training file on the old server? Look for files like /var/lib/rspamd/\*.sqlite on the old server. In that case please follow these [simple instructions from the rspamd documentation](https://rspamd.com/doc/faq.html#which-backend-should-i-use-for-statistics) to convert them to data in Redis.
|
||||
|
||||
If instead you have used Redis already then you just need to copy over the Redis database from the old server. Stop Redis on the new server. Copy over the /var/lib/redis/dump.rdb from the old server to the new server. Start Redis again. And restart rspamd. So on the new server run:
|
||||
|
||||
```
|
||||
systemctl stop redis
|
||||
scp root@old-server:/var/lib/redis/dump.rdb /var/lib/redis
|
||||
systemctl start redis
|
||||
systemctl restart rspamd
|
||||
```
|
||||
|
||||
To check if that worked you can ask Rspamd using “rspamc stat” and look for…
|
||||
|
||||
```
|
||||
Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0;
|
||||
total blocks: 0; free: 0.00%; learned: 21164; users: 214; languages: 0
|
||||
Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0;
|
||||
total blocks: 0; free: 0.00%; learned: 1411; users: 62; languages: 0
|
||||
```
|
||||
|
||||
### (c) Training from your existing ham and spam emails
|
||||
|
||||
Have you been running a mail server with mailboxes in a _Malidir_ structure before but without rspamd? Then you probably have a good amount of ham and spam emails. Let’s use those to train rspamd. It is important to train both ham and spam emails. The _rspamc_ command will allow you to feed entire directories/folders of emails to the learning process. An example to train spam:
|
||||
|
||||
```
|
||||
rspamc learn\_spam /var/vmail/example.org/john/Maildir/.Junk/cur
|
||||
```
|
||||
|
||||
And this would be an example to train ham from John’s inbox:
|
||||
|
||||
```
|
||||
rspamc learn\_ham /var/vmail/example.org/john/Maildir/cur
|
||||
```
|
||||
|
||||
Of course the quality of spam detection will depend on how good the source data is. If users put emails in their Junk folder which are not typical spam they will soil the detection.
|
||||
|
||||
Check the number of emails you learned by running…
|
||||
|
||||
```
|
||||
rspamc stat
|
||||
```
|
||||
|
||||
Bayes spam checking will not work before it learned at least 200 spam and ham emails. Teaching rspamd fewer emails or just spam emails will not work. This is defined by the _min\_learns_ variable defined in /etc/rspamd/statistic.conf.
|
||||
|
||||
In the output you will find lines beginning with “Statfile” like these…
|
||||
|
||||
```
|
||||
Statfile: BAYES_SPAM type: redis; length: 0; free blocks: 0;
|
||||
total blocks: 0; free: 0.00%; learned: 21164;
|
||||
users: 214; languages: 0
|
||||
|
||||
Statfile: BAYES_HAM type: redis; length: 0; free blocks: 0;
|
||||
total blocks: 0; free: 0.00%; learned: 1411;
|
||||
users: 62; languages: 0
|
||||
```
|
||||
|
||||
This is what you usually start with. The more emails you feed into the training process the better the detection rate will be. Some emails however may not be long enough or too similar to previously trained emails. So don’t worry if you are training 1000 emails but just get a count of 500 emails here.
|
||||
|
||||
## Per-user spam training
|
||||
|
||||
rspamd allows you to train the spam detection per user. It would not keep a global training database that applies to all users. Instead each user gets their own training.
|
||||
|
||||
Advantage: users work differently. Some have subscribed to a sales newsletter and now believe that marking it as spam gets them unsubscribed. Yes, that’s stupid but can thoroughly confuse the spam detection. Also you might be very interested in viagr\* product information while others do not.
|
||||
|
||||
Disadvantage: training still requires many ham and spam mails before it has any effect. So unless a user gets 200 samples of good and evil emails the spam detection cannot work. Many users will not get that many emails so due to the lack of spam training the detection will not be improved.
|
||||
|
||||
If you decide you want to use per-user spam training then add/edit the file `/etc/rspamd/local.d/classifier-bayes.conf` and insert:
|
||||
|
||||
```
|
||||
users_enabled = true;
|
||||
```
|
||||
|
||||
## Autoexpunge
|
||||
|
||||
Andi Olsen pointed out that Dovecot has introduced a [feature](https://wiki.dovecot.org/MailboxSettings) to automatically delete emails in a folder that reach a certain age. This is especially useful for the “Trash” and “Junk” folders. To enable this feature just edit the `/etc/dovecot/conf.d/15-mailboxes.conf` file and add the _autoexpunge_ parameter where desired. Example:
|
||||
|
||||
```
|
||||
mailbox Junk {
|
||||
special_use = \Junk
|
||||
auto = subscribe
|
||||
autoexpunge = 30d
|
||||
}
|
||||
mailbox Trash {
|
||||
special_use = \Trash
|
||||
auto = subscribe
|
||||
autoexpunge = 30d
|
||||
}
|
||||
```
|
||||
|
||||
The “auto = subscribe” makes sure that the “Junk” and “Trash” folders are automatically created for every user. Otherwise spam emails cannot be moved to the “Junk” folder later.
|
||||
|
||||
## Learning from user actions
|
||||
|
||||
Now we are getting to something really cool. Let’s tell Dovecot that moving emails into the Junk folder teaches rspamd instantly that the email is spam. And train an email as ham if it is moved out of the Junk folder. We will add triggers (actually “_sieve scripts_“) to the action of moving emails via IMAP.
|
||||
|
||||
The currently recommended way is to use the “[IMAPSieve](https://doc.dovecot.org/2.3/settings/pigeonhole-ext/imapsieve/)” plugin instead. There is nothing to install – it comes with the Dovecot packages. We just need to configure it.
|
||||
|
||||
First order of business is enabling the IMAPSieve plugin for the IMAP protocol/service in Dovecot. Edit the `/etc/dovecot/conf.d/20-imap.conf` file and look for the line reading “mail\_plugins”. Turn it into:
|
||||
|
||||
```
|
||||
mail_plugins = $mail_plugins quota imap_sieve
|
||||
```
|
||||
|
||||
We also need to edit Dovecot’s Sieve configuration to enable two plugins that are required for our task. Sieve is a scripting language that automates things in conjunction with emails and folders. Edit the file `/etc/dovecot/conf.d/90-sieve.conf` and put these lines into the `plugin {…}` section:
|
||||
|
||||
```
|
||||
# From elsewhere to Junk folder
|
||||
imapsieve_mailbox1_name = Junk
|
||||
imapsieve_mailbox1_causes = COPY
|
||||
imapsieve_mailbox1_before = file:/etc/dovecot/sieve/learn-spam.sieve
|
||||
|
||||
# From Junk folder to elsewhere
|
||||
imapsieve_mailbox2_name = *
|
||||
imapsieve_mailbox2_from = Junk
|
||||
imapsieve_mailbox2_causes = COPY
|
||||
imapsieve_mailbox2_before = file:/etc/dovecot/sieve/learn-ham.sieve
|
||||
|
||||
sieve_pipe_bin_dir = /etc/dovecot/sieve
|
||||
sieve_global_extensions = +vnd.dovecot.pipe
|
||||
sieve_plugins = sieve_imapsieve sieve_extprograms
|
||||
```
|
||||
|
||||
The first rule tells Dovecot to run the Sieve rules as defined in the `/etc/dovecot/sieve/learn-spam.sieve` file whenever an email is moved into a user’s “Junk” folder. We will create that Sieve script in a minute.
|
||||
|
||||
The second rule sets the other way. Whenever an email is moved from the “Junk” folder to any (\*) folder then the `/etc/dovecot/sieve/learn-ham.sieve` Sieve script is called.
|
||||
|
||||
The “sieve\_pipe\_bin\_dir” setting defines where executable scripts are allowed to reside. We will put our simple learning scripts there. And finally the “sieve\_global\_extensions” setting enables the pipe plugin that allows sending email to external commands.
|
||||
|
||||
Next up let’s create the Sieve scripts that we told Dovecot about. Create a new directory /etc/dovecot/sieve to put our new files in:
|
||||
|
||||
```
|
||||
mkdir /etc/dovecot/sieve
|
||||
```
|
||||
|
||||
Then create the file `/etc/dovecot/sieve/learn-spam.sieve` and let it contain:
|
||||
|
||||
```
|
||||
require ["vnd.dovecot.pipe", "copy", "imapsieve"];
|
||||
pipe :copy "rspamd-learn-spam.sh";
|
||||
```
|
||||
|
||||
Let’s do the same for `/etc/dovecot/sieve/learn-ham.sieve`
|
||||
|
||||
```
|
||||
require ["vnd.dovecot.pipe", "copy", "imapsieve", "variables"];
|
||||
if string "${mailbox}" "Trash" {
|
||||
stop;
|
||||
}
|
||||
pipe :copy "rspamd-learn-ham.sh";
|
||||
```
|
||||
|
||||
The above Sieve script avoids training an email as _ham_ if the user moves it to the _Trash_ folder. After all if you clear your _Junk_ folder you do not want to train your spam as regular emails.
|
||||
|
||||
Restart Dovecot:
|
||||
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
|
||||
These two scripts need to be compiled – that is turning them into machine-readable code:
|
||||
|
||||
```
|
||||
sievec /etc/dovecot/sieve/learn-spam.sieve
|
||||
sievec /etc/dovecot/sieve/learn-ham.sieve
|
||||
```
|
||||
|
||||
This creates two new files “learn-ham.svbin” and “learn-spam.svbin” that look like gibberish inside but are now in a format that Dovecot’s Sieve plugin can understand.
|
||||
|
||||
Let’s fix the permissions of these files, too, while we are at it:
|
||||
|
||||
```
|
||||
chmod u=rw,go= /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
|
||||
chown vmail:vmail /etc/dovecot/sieve/learn-{spam,ham}.{sieve,svbin}
|
||||
```
|
||||
|
||||
And the last step is to create the simple shell scripts that do the actual spam/ham training. The first file is `/etc/dovecot/sieve/rspamd-learn-spam.sh` which contains:
|
||||
|
||||
```
|
||||
#!/bin/sh
|
||||
exec /usr/bin/rspamc learn_spam
|
||||
```
|
||||
|
||||
That looks simple, doesn’t it? Nothing more is actually needed. The spam email is piped to this script and rspamd learns it as a spam email and adjusts its spam detection database accordingly.
|
||||
|
||||
The second script teaches ham and is called `/etc/dovecot/sieve/rspamd-learn-ham.sh`. Make it contain:
|
||||
|
||||
```
|
||||
#!/bin/sh
|
||||
exec /usr/bin/rspamc learn_ham
|
||||
```
|
||||
|
||||
These two shell scripts must be made executable:
|
||||
|
||||
```
|
||||
chmod u=rwx,go= /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
|
||||
chown vmail:vmail /etc/dovecot/sieve/rspamd-learn-{spam,ham}.sh
|
||||
```
|
||||
|
||||
I hope you haven’t lost your mind yet. It’s really just a chain of things to happen. Let’s reiterate how this process works:
|
||||
|
||||
1. a user moves a spam email into their “Junk” folder
|
||||
2. Dovecot realizes that this triggers the Sieve rule “imapsieve\_mailbox1” so it calls the Sieve script /etc/dovecot/sieve/learn-spam.sieve (in fact the \*.svbin version of the script)
|
||||
3. Sieve will take the email and send (“pipe”) it to the executable rspamd-learn-spam.sh shell script
|
||||
4. the script in turn runs the email through the “/usr/bin/rspamc learn\_spam” command
|
||||
|
||||
This works equally for the other way or learning ham emails of course.
|
||||
|
||||
I am sure you are eager to try it out. However to see that it actually works I suggest you edit the /etc/dovecot/conf.d/10-logging.conf file and set “mail\_debug=yes”. That will add a lot more detail to the /var/log/mail.log file but on a busy server may also lead to headaches. 🙂
|
||||
|
||||
Restart Dovecot…
|
||||
|
||||
```
|
||||
systemctl restart dovecot
|
||||
```
|
||||
|
||||
…and watch the /var/log/mail.log file…
|
||||
|
||||
```
|
||||
tail -f /var/log/mail.log
|
||||
```
|
||||
|
||||
Now open your IMAP client (Thunderbird, Evolution, Roundcube, mutt or whatever you prefer) and drag an email to your Junk folder. The mail log will show a lot of things that are going on. I tried to compact the output so that you better understand that it worked:
|
||||
|
||||
```
|
||||
imapsieve: Static mailbox rule [1]: mailbox=`Junk' from=`*' causes=(COPY) => before=`file:/etc/dovecot/sieve/learn-spam.sieve' after=(none)
|
||||
imapsieve: Static mailbox rule [2]: mailbox=`*' from=`Junk'
|
||||
causes=(COPY) => before=`file:/etc/dovecot/sieve/learn-ham.sieve'
|
||||
after=(none)
|
||||
|
||||
imapsieve: Matched static mailbox rule [1]
|
||||
|
||||
sieve: file storage: script: Opened script `learn-spam'
|
||||
from `/etc/dovecot/sieve/learn-spam.sieve'
|
||||
|
||||
sieve: action pipe: running program: rspamd-learn-spam.sh
|
||||
|
||||
program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh:
|
||||
Pass environment: USER=john@example.org
|
||||
|
||||
program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh:
|
||||
Pass environment: HOME=/var/vmail/example.org/john
|
||||
|
||||
program exec:/etc/dovecot/sieve/rspamd-learn-spam.sh:
|
||||
Pass environment: HOST=narnia
|
||||
|
||||
Mailbox Junk: UID 1: Opened mail because: mail stream
|
||||
|
||||
sieve: uid=1: Execute storing into mailbox 'Junk'
|
||||
```
|
||||
|
||||
Dovecot finds two Sieve rules: \[1\] and \[2\]. And it finds that rule \[1\] matches your current action and runs it. That in turn calls the “rspamd-learn-spam.sh” script to train that email as spam. And at the very end the email is actually moved to the “Junk” folder.
|
||||
|
||||
And if you pull an email out of the “Junk” folder you should see mailbox rule \[2\] be called and the email being learned as ham.
|
||||
|
||||
Don’t forget to switch off “mail\_debug” again or your users’ actions will quickly fill your log file.
|
||||
|
||||
## Logging
|
||||
|
||||
rspamd keeps a verbose log of its actions in /var/log/rspamd/rspamd.log. If a user complains that a certain email got blocked or at least flagged as spam then take a look at this log. You can match the /var/log/mail.log with it by comparing the Postfix queue ID. Those are the 12-digit hexadecimal number like “**95CE05A00547**“. Those IDs can be found in the rspamd.log, too:
|
||||
|
||||
```
|
||||
<40985d>; task; rspamd_task_write_log: id: <undef>, qid: <**95CE05A00547**>, ip: 12.13.51.194, from: <…>, (default: F (no action): [3.40/15.00] [MISSING_MID(2.50){},MISSING_DATE(1.00){},MIME_GOOD(-0.10){text/plain;},ARC_NA(0.00){},ASN(0.00){asn:8220, ipnet:212.123.192.0/18, country:GB;},FROM_EQ_ENVFROM(0.00){},FROM_NO_DN(0.00){},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},RCVD_TLS_ALL(0.00){},TO_DN_NONE(0.00){},TO_DOM_EQ_FROM_DOM(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 181, time: 16.000ms real, 6.385ms virtual, dns req: 0, digest: <69b289a82827c11f759837c033cd800a>, rcpts: <…>, mime_rcpt: <…>
|
||||
```
|
||||
|
||||
## The web interface
|
||||
|
||||
|
||||
rspamd comes with a neat bonus feature: a web interface. It allows you to check emails for spam, get statistics and fine-tune scores. It is already installed and enabled by default and expects HTTP requests on port 11334 on the localhost interface. I suggest you add a simple proxy configuration to your already working HTTPS-enabled web mail configuration to get access.
|
||||
|
||||

|
||||
|
||||
First you need to enable Apache’s modules for HTTP proxying and rewriting:
|
||||
|
||||
```
|
||||
a2enmod proxy_http
|
||||
a2enmod rewrite
|
||||
```
|
||||
|
||||
You can either create a new virtual host configuration or just edit the /etc/apache2/sites-available/webmail.**example.org**-https.conf file. Anywhere within the _VirtualHost_ tags add:
|
||||
|
||||
```
|
||||
<Location /rspamd>
|
||||
Require all granted
|
||||
</Location>
|
||||
|
||||
RewriteEngine On
|
||||
RewriteRule ^/rspamd$ /rspamd/ \[R,L\]
|
||||
RewriteRule ^/rspamd/(.\*) http://localhost:11334/$1 \[P,L\]
|
||||
```
|
||||
|
||||
This piece of configuration will forward any requests to `https://webmail.example.org/rspamd` to localhost:11334 and thus give you access to the rspamd web interface.
|
||||
|
||||
The interface is password protected. Let’s generate a new access password:
|
||||
|
||||
```
|
||||
pwgen 15 1
|
||||
```
|
||||
|
||||
This gives you a password like “eiLi1lueTh9mia4”. You could put that password in an rspamd configuration file. But cleartext passwords in configuration files are not quite elegant. Let’s create a hash of the password:
|
||||
|
||||
```
|
||||
rspamadm pw
|
||||
Enter passphrase: …
|
||||
$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…
|
||||
```
|
||||
|
||||
Feed it the password that pwgen created for you and you will get a long hashed password. This procedure by the way is also documented on the [rspamd pages](https://rspamd.com/doc/tutorials/quickstart.html#setting-the-controller-password).
|
||||
|
||||
Create a new configuration file `/etc/rspamd/local.d/worker-controller.inc` and put your hashed password in there:
|
||||
|
||||
```
|
||||
password = "$2$icoahes75e7g9wxapnrbmqnpuzjoq7z…"
|
||||
```
|
||||
|
||||
That’s it for the configuration. Finally restart both rspamd and Apache to load your changed configuration:
|
||||
|
||||
```
|
||||
systemctl restart rspamd
|
||||
systemctl restart apache2
|
||||
```
|
||||
|
||||
If everything went as expected you should now be able to access the rspamd web interface at `https://webmail.example.org/rspamd`
|
||||
27
src/content/docs/ispmail-bookworm/20-whats-new.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
title: What's new
|
||||
lastUpdated: 2023-09-24
|
||||
slug: ispmail-bookworm/whats-new
|
||||
sidebar:
|
||||
order: 20
|
||||
---
|
||||
|
||||
|
||||
I try to keep as many parts of the setup unchanged. Many of us are happy with their mail servers and want to change as little as possible. Email is a boring matter anyway and we do our jobs best if our users don’t even realize that we are there.
|
||||
|
||||
A few things though that are new in this version of the ISPmail guide:
|
||||
|
||||
- **Newer software versions** – it is a new Debian release after all. There are no breaking changes. Just business as usual. If you want to check the changelogs here are the version changes:
|
||||
- Postfix 3.5.18 -> 3.7.6
|
||||
- Dovecot 2.3.13 -> 2.3.19
|
||||
- PHP 7.4 -> 8.2
|
||||
- rspamd 2.7.1 -> 3.4.1
|
||||
- Apache 2.4.56 -> 2.4.57
|
||||
- Roundcube 1.4.13 -> 1.6.1
|
||||
- MariaDB 1.21 -> 1.22
|
||||
- Adminer 4.7.9 -> 4.8.1
|
||||
- An improved **Ansible** playbook to help you automate all the steps of this guide if you want to set up multiple servers without repeating all steps manually.
|
||||
- Clearer explanations. I have reworded complex parts of the configuration.
|
||||
- Fixed links. The Dovecot wiki has moved around quote a lot of pages and I didn’t notice earlier because it did not lead to a 404 or a redirect.
|
||||
- Dovecot now uses the “count” backend to compute a user’s quota. Previous guides still used “maildir”.
|
||||
- Two chapters are currently not published: the part about Thunderbird auto-configuration because there seem to be problems with the newest versions of Thunderbird. And the part about firewalling and fail2ban because there seems to be a bug with firewalld and fail2ban. These pages deal with optional features and are not a showstopper.
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
---
|
||||
title: Setting DNS records
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/setting-dns-records
|
||||
sidebar:
|
||||
order: 200
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
## MX entries
|
||||
|
||||
So now you have your working mail server. But how do other mail servers get to know you? The answer lies in the most important service on the internet: [DNS](http://en.wikipedia.org/wiki/Domain_Name_System) – the Domain Name System. Assume that a mail server somewhere on the other end of the internet wants to send an email to `john@example.org`. The server must find out the name (and IP address) of the mail server responsible for the example.org domain. This is how that works:
|
||||
|
||||

|
||||
|
||||
The remote server queries its DNS server for the MX (Mail eXchanger) record of the “example.org” domain. (If no MX record was found it tries again and asks for A (address) records. That’s a fallback solution.) Let’s run a query for a real domain to get an idea. Let’s ask which servers receive email for …@gmail.com.
|
||||
|
||||
```
|
||||
$> host -t MX gmail.com
|
||||
gmail.com mail is handled by 10 alt1.gmail-smtp-in.l.google.com.
|
||||
gmail.com mail is handled by 20 alt2.gmail-smtp-in.l.google.com.
|
||||
gmail.com mail is handled by 30 alt3.gmail-smtp-in.l.google.com.
|
||||
gmail.com mail is handled by 40 alt4.gmail-smtp-in.l.google.com.
|
||||
gmail.com mail is handled by 5 gmail-smtp-in.l.google.com.
|
||||
```
|
||||
|
||||
So as a result we get 5 different MX records. Each of them consists of a numeric priority and the host name of the mail server. A mail server would pick the entry with the highest priority (=the lowest number) and establish an SMTP connection to that host. In this example that would be the priority 5 server gmail-smtp-in.l.google.com. If that server could not be reached then the next best server with priority 10 would be used and so on. So all you have to do in your own DNS zone is add an MX entry pointing to your mail server. If you want to run a backup mail server (which is outside of the scope of this tutorial) then you can add a second entry with a an equal or lower priority.
|
||||
|
||||
A mistake some people make is using an IP address in MX records. That is not allowed. An MX record always points to a host name. You will have to add an A record for your MX record to point to the actual IP address of your mail server.
|
||||
|
||||
<Aside type="tip" title="For fun and science">
|
||||
In the above example it is very unlikely that a mail server will ever have to use the server with priority 40. Adventurous system administrators can add such a low-priority entry and see who connects to it. An interesting observation is that spammers often try these servers first – hoping that it is just for backup purposes and less restrictive or lazily configured than the main server. If you see someone connecting to the lowest-priority address first without having tried a higher-priority mail server then you can be pretty certain that it’s not a friend who’s knocking at your door.
|
||||
</Aside>
|
||||
|
||||
## Fallback to A entries
|
||||
|
||||
It is advised to explicitly name mail servers for your domain in the MX records. If you can’t do that for whatever reason then the remote mail server will just do an A record lookup for the IP address and then send email there. If you just run one server for both the web service and the email service then you can do that. But if the web server for your domain is located at another IP address than your mail server then this won’t work.
|
||||
|
||||
## Reverse DNS
|
||||
|
||||
Many mail servers check that your forward (name to IP) and reverse (IP to name) lead to the same results. This is how it should look:
|
||||
|
||||
example.org —(MX)–> mx.example.org
|
||||
mx.example.org —(A)–> 1.2.3.4
|
||||
1.2.3.4 —(PTR)–> mx.example.org
|
||||
|
||||
(1.2.3.4 in a PTR record is actually 4.3.2.1.in-addr.arpa.)
|
||||
|
||||
## IPv6
|
||||
|
||||
Fortunately many ISPs finally start to offer IPv6 in their networks. So if you rent a virtual server from a provider then check if you also got an IPv6 address assigned. In that case you also need to have proper DNS records for that as well. Like:
|
||||
|
||||
example.org —(MX)–> mx.example.org
|
||||
mx.example.org —(AAAA)–> fdeb:37d7:ef39:e063::1
|
||||
fdeb:37d7:ef39:e063 —(PTR)–> mx.example.org
|
||||
|
||||
Chances are that you haven’t dealt with IPv6 yet. Give it a try. IP addresses may look different at first but you will never again have to deal with odd subnet sizes or NAT. Thank me later. 🙂
|
||||
209
src/content/docs/ispmail-bookworm/210-prevent-spoofing-dkim.mdx
Normal file
|
|
@ -0,0 +1,209 @@
|
|||
---
|
||||
title: Prevent spoofing using DKIM
|
||||
lastUpdated: 2023-10-04
|
||||
slug: ispmail-bookworm/prevent-spoofing-using-dkim
|
||||
sidebar:
|
||||
order: 210
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
Email sender spoofing is the act of pretending to be in control of someone else’s email address. This is a common problem with [phishing](https://en.wikipedia.org/wiki/Phishing). Often scammers send emails with a sender address of something@paypal.com and hope that the recipient falls for it and trusts them. In fact SMTP does not care which sender address you send. Many mail service providers enforce that you send emails only using your own email address. But some do not. And spammer and scammers obviously could not care less.
|
||||
|
||||

|
||||
|
||||
So roughly ten years ago a new method was conceived that added a _cryptographic signature_ to the header of an email that the recipient could check to verify the authenticity of the sender and the integrity of the email. The signature is created using a private key that only the sending mail server has. It can then be verified by the recipient by downloading the corresponding public key from the DNS zone of the sending domain and running a signature check. This works very similar to PGP or S/MIME signing – just on a domain level. Your mail server can sign all outgoing emails automatically. The method used nowadays is called Domain Keys Identified Mail – or short: [DKIM](https://en.wikipedia.org/wiki/DomainKeys).
|
||||
|
||||
Let’s take an example. I have just sent an email from GMail to my personal email account on my own mail server. Google uses DKIM signing so the email got this additional header from Google’s mail servers:
|
||||
|
||||
```
|
||||
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
|
||||
d=gmail.com; s=20161025;
|
||||
h=mime-version:from:date:message-id:subject:to;
|
||||
bh=/FpkZfBuKR0WWcH2fFcr9M4qgX4Z4+/0dX4dpqycK28=;
|
||||
b=fZ4Efy1TuXAl1ho2twkEuNiVP8k5GRKqlol/f/dTawaxciAv1bwinkbu…
|
||||
```
|
||||
|
||||
I need Google’s DKIM public key to verify that signature. It is stored in their DNS zone as a TXT record of “**20161025**.\_domainkey.google.com”. The “**20161025**” is the key selector that is mentioned in the signature as “s=**20161025**“. You can use any number of keys as long as you create the signatures with the matching private key. The “\_domainkey” part is the standard subdomain for DKIM keys. So let’s get that TXT record:
|
||||
```
|
||||
dig +short 20161025._domainkey.google.com txt
|
||||
```
|
||||
|
||||
This returns…
|
||||
|
||||
```
|
||||
"k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwXNZF1j8sJPDleRjf9SPBNem0ik58kF1ilC1nUgKAttl9v7FX9hXJXPmLNhVtSKVZ8yruaeOZLeIxtgtk1s81zzIE5Mj0AiGn2wlFt4kYfqlDfYe95YLQHjynu4i7vj1Tj" "ksf62btcCbL+3XhbK+oD5PlqYhXHWuzoKoEp5L4lCihgkONvU/oy7NNeE6quqfF/y0YSLwF2WVA2Kd8L6R0Ar2dYT/3wZCFknI7xhvPqh9HNcIWBELGPwtXcsHbX1wvBlCgNQAUcdJrf2YWzAwqmZ564/1ipL1IMk1nafPJk75ktumVNz6ORuIn3jbZWp9rRpnaeI9cu/8KfSKH2EY9QIDAQAB"
|
||||
```
|
||||
|
||||
That is the public key that I can use to verify the signature. An automated check can be done using the “opendkim-testmsg” tool as described later. I can run it and paste the entire email including headers and body into it. If it doesn’t complain with an error message then the signature is correct.
|
||||
|
||||
Sounds good? Then let’s implement that for your email domain, too.
|
||||
|
||||
|
||||
## Creating a keypair
|
||||
|
||||
As explained above you need a private key that your mail server will use and a public key that gets added to your DNS zone. rspamd can create DKIM keys already. You may want to install “dig” though which allows to query DNS records. It works similar to “nslookup” but is more versatile.
|
||||
|
||||
```
|
||||
apt install dnsutils
|
||||
```
|
||||
|
||||
rspamd has its built-in DKIM signing module enabled by default.
|
||||
|
||||
If you put your key file into /var/lib/rspamd/dkim/ using a certain naming scheme it will pick it up automatically. Create that directory to store keys in:
|
||||
|
||||
```
|
||||
mkdir /var/lib/rspamd/dkim
|
||||
chown _rspamd:_rspamd /var/lib/rspamd/dkim
|
||||
```
|
||||
|
||||
Create your keypair:
|
||||
|
||||
```
|
||||
rspamadm dkim_keygen -d example.org -s 2019122401
|
||||
```
|
||||
|
||||
The selector (-s) I chose is **2019122401** because that’s the day I created it. The first (01) key on 2019-12-24. It doesn’t matter though – you can name it anything you want. If you are lazy and unconcerned you could even use “dkim” as the selector and later spare some work by not needing DKIM maps that define which key is supposed to be used for each domain. “dkim” is the default selector if you do not use maps. But you will probably some day want or need to replace the key so I recommend you rather use maps as explained further below. It gives you more flexibility and is pretty easy to do.
|
||||
|
||||
The output will look like this:
|
||||
|
||||
```
|
||||
-----BEGIN PRIVATE KEY-----
|
||||
MIICdwIBADANBgkqhkiG9w0BAQE…
|
||||
-----END PRIVATE KEY-----
|
||||
2019122401._domainkey IN TXT ( "v=DKIM1; k=rsa; " "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCZ8e4RtsvuvsgXyo3Jr63twdx3Zm2RtnYPf87wba5ZnFP0sQIagjJ0R3WL+djKhM5EteBeboTeaflLR+AUKWJdQnL5jzGil+yWThH5C+ixhkdFZPdabkxBji+3r6uRxJKBsN0gy2/+Gp4sGs7lmFgioki9OLgQnzzFh2nE/MSACwIDAQAB" ) ;
|
||||
```
|
||||
|
||||
The first part is the private key. And that includes the “…BEGIN…” and “…END…” lines. This key must be kept secret and will only be used by your mail server to sign outgoing emails.
|
||||
|
||||
The second part is the DNS record you need to add to your domain’s DNS zone. Let’s start with that.
|
||||
|
||||
|
||||
## Adding the DNS record
|
||||
|
||||
Before you start signing your emails you must make sure that the public key is properly present in your DNS zone for the domain you are sending emails from. Otherwise the recipient will be unable to verify the signature and may incorrectly assume that the email was spoofed.
|
||||
|
||||
Take a look at the TXT record. It will look something like this:
|
||||
|
||||
```
|
||||
2019122401._domainkey IN TXT ( "v=DKIM1; h=sha256; k=rsa; " "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCxenHupkYLPmFMbJjV9dQICKUl2xH/aexSRUwCuw7TJ9dkddqIN+6tyw4VKhnW8R0/UlbzlSFLmVgMU0uUkwTtVqyDHhtSU7LV/SkVYmUst4dTUF1r+8PvhAm7vobMYKdwvRsOq27ABtZc8P4oU2XXHqqa6LU8s4sNxs12hLW9swIDAQAB" ) ;
|
||||
```
|
||||
|
||||
If you are running your own DNS server you should be able to copy this entire file and put it into your DNS zone. However if your internet provider offers you just a web interface to manage your domains then create a new TXT record with a host name of “**2019122401**._domainkey” in your domain and put the string within the double-quotes into it as the value. In my example:
|
||||
|
||||
```
|
||||
2019122401._domainkey ➠p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCxenHupkYLPmFMbJjV9dQICKUl2xH/aexSRUwCuw7TJ9dkddqIN+6tyw4VKhnW8R0/UlbzlSFLmVgMU0uUkwTtVqyDHhtSU7LV/SkVYmUst4dTUF1r+8PvhAm7vobMYKdwvRsOq27ABtZc8P4oU2XXHqqa6LU8s4sNxs12hLW9swIDAQAB
|
||||
```
|
||||
|
||||
Be aware that the string you got contains two strings “…” + “…” that must be merged into one to work. (The syntax with quotes is meant for a DNS zone file if you run your own name server like _bind_.) There must usually not be any _quotes_ in the record data. Depending on your ISP it may take a while until the new record is visible on the internet. You can use _dig_ to verify that:
|
||||
|
||||
```
|
||||
dig +trace 2019122401._domainkey.example.org txt
|
||||
```
|
||||
|
||||
If you get the TXT entry like as follows then you are ready to enable DKIM signing in rspamd for that domain:
|
||||
|
||||
```
|
||||
TXT "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCxenHupkYLPmFMbJjV9dQICKUl2xH/aexSRUwCuw7TJ9dkddqIN+6tyw4VKhnW8R0/UlbzlSFLmVgMU0uUkwTtVqyDHhtSU7LV/SkVYmUst4dTUF1r+8PvhAm7vobMYKdwvRsOq27ABtZc8P4oU2XXHqqa6LU8s4sNxs12hLW9swIDAQAB" "" from server foo.bar in 24 ms.
|
||||
```
|
||||
|
||||
|
||||
## Enabling DKIM maps in rspamd
|
||||
|
||||
As explained above it is advised to use DKIM _maps_. It’s nothing fancy. Just a simple file defining which selector you want to use for a certain domain. rspamd will assume that your selector is always “dkim” unless specified otherwise in a map. If you used “dkim” then you may get into trouble when you later want to replace your key. DNS is a sluggish system and propagating a new DKIM public key may take a day. Emails signed with a newer key may get rejected while the DNS record is not yet known everywhere in the world.
|
||||
|
||||
Using maps is simple. First we need to change the _selector\_map_ setting of the dkim\_signing module. To do that create a new file in `/etc/rspamd/local.d/dkim_signing.conf` and make it contain just these two lines:
|
||||
|
||||
```
|
||||
path = "/var/lib/rspamd/dkim/$domain.$selector.key";
|
||||
selector_map = "/etc/rspamd/dkim_selectors.map";
|
||||
```
|
||||
|
||||
The configuration is pretty self-explaining. rspamd will look for the domain-to-key mapping in the _dkim\_selectors.map_ file. Create that file and make it contain this line:
|
||||
|
||||
```
|
||||
example.org 2019122401
|
||||
```
|
||||
|
||||
That’s all really. rspamd now knows that whenever it sees an outgoing email from anyone@example.org it will get the DKIM private key from /var/lib/rspamd/dkim/**example.org**.**2018022301**.key and use it to sign the email.
|
||||
|
||||
Reload the configuration:
|
||||
|
||||
```
|
||||
systemctl restart rspamd
|
||||
```
|
||||
|
||||
This method works well if you just have a few domains that virtually never change. If you are rather serving random customer domains you should consider putting the keys into a Redis database instead as [described in the documentation](https://rspamd.com/doc/modules/dkim_signing.html). There is not yet a way to manage DKIM keys in a database like MySQL.
|
||||
|
||||
<Aside type="tip" title="Selectors?">
|
||||
Selectors may seem unnecessary. But you will need them once you want to replace your current signing key later. In that case you need to generate a new key using a different selector/name. Now that both DNS records are online you can make your mail server use the new signing key. You should leave the old key in place for a while.
|
||||
</Aside>
|
||||
|
||||
|
||||
## Adding the domain key to rspamd
|
||||
|
||||
Take the private key that was created earlier (the multi-line string including “`…BEGIN PRIVATE KEY…`” and “`…END PRIVATE KEY…`“) and put it into a file at the location where rspamd will look for it:
|
||||
|
||||
```
|
||||
/var/lib/rspamd/dkim/example.org.2019122401.key
|
||||
```
|
||||
|
||||
The name of the file has to be DOMAIN + dot + SELECTOR + “.key” like above. If you name the file incorrectly you will get an error in your rspamd.log file like “_lua\_dkim\_sign\_handler: cannot load dkim key /var/lib/rspamd/dkim/example.org.dkim.key_“.
|
||||
|
||||
Make sure that only \_rspamd can read it:
|
||||
|
||||
```
|
||||
chown _rspamd /var/lib/rspamd/dkim/*
|
||||
chmod u=r,go= /var/lib/rspamd/dkim/*
|
||||
```
|
||||
|
||||
rspamd will automatically pick up the files and does not need to be restarted.
|
||||
|
||||
|
||||
## Send a test email
|
||||
|
||||
If you have another email account at another location then you could just send a test email there via your mail server. If you take a look at the received email it should have a DKIM header now like:
|
||||
|
||||
```
|
||||
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=example.org; s=2019122401; t=1519400838 h=from:subject:date:message-id:to:mime-version:content-type:content-transfer-encoding; bh=kpYel1IlDvqXEUc0SyIpXbMte3XpQOCXHl+zTyHQvGc=; b=NEUyWUoeKE…
|
||||
```
|
||||
|
||||

|
||||
|
||||
To verify the signature install the _opendkim-tools_ package, copy the entire test email (including headers and body), run _opendkim-testmsg_ in your shell and paste the email (finish with CTRL-D).
|
||||
|
||||
If you get no output then the signature verified correctly. But if you get something like “opendkim-testmsg: dkim\_eom(): Unable to verify” then double-check your DNS record.
|
||||
|
||||
You can also use websites like [dkimvalidator.com](http://dkimvalidator.com/), [isnotspam.com](http://isnotspam.com/) or [mail-tester.com](https://www.mail-tester.com/) service to verify that your signatures are working well.
|
||||
|
||||
|
||||
## SPF and DMARC
|
||||
|
||||
Adding DKIM signatures is a good first step. But you can take it further by telling receiving mail servers that they should not accept any email from your domain without a valid signature or from servers that you do no operate. There are two concepts that aim to help. The older [SPF](https://en.wikipedia.org/wiki/Sender_Policy_Framework) and the newer [DMARC](https://en.wikipedia.org/wiki/DMARC). Either of them means creating a machine-readable string in a predefined format and adding a TXT record to your DNS zone. Receiving mail servers can check those records and take your advice (as the domain owner) what to do if the criteria of the email are not met. It could accept the email anyway or flag it as spam or reject it altogether.
|
||||
|
||||
Let’s take a look at a typical SPF record:
|
||||
|
||||
```
|
||||
"v=spf1 ip4:157.97.194.11 mx ~all"
|
||||
```
|
||||
|
||||
What it means:
|
||||
|
||||
1. this is an SPF record of version 1 of the standard (there is currently no other version)
|
||||
2. please accept emails from the IP address 157.97.194.11
|
||||
3. alternatively accept emails from any server that is mentioned in our domain’s MX record (the server(s) that receive email for your domain)
|
||||
4. any other email should be considered suspicious – it might be spam or worse
|
||||
|
||||
There are [websites](https://easydmarc.com/tools/spf-record-generator) that help you create your SPF string to add to your DNS domain. Keep in mind though:
|
||||
|
||||
- You should know which mail servers send email from your domain. Do not forget to include mailing list or newsletter services that send in your name.
|
||||
- Start with “~all” to mark emails as spam that do not meet the criteria. If all goes well switch to “-all” after a few weeks if you like.
|
||||
- Note that forwarding emails from your domain may break SPF because suddenly the email appears to be coming from an IP address that is not authorized. This has been a common problem for mailing lists and is gradually being fixed by resending the email from the domain of the mailing list service.
|
||||
|
||||
I mentioned that DMARC is the newer standard. So why use SPF anyway? Because some email providers value your effort if you use SPF, too. Technically it’s sufficient to specify a DMARC entry. In my opinion restricting the IP addresses allowed to send is a little dangerous and a little inflexible. It is far more interesting to require that emails from your domain have a valid DKIM signature. Such a record may look like:
|
||||
|
||||
```
|
||||
"v=DMARC1; p=reject; adkim=s; ruf=postmaster@example.org"
|
||||
```
|
||||
|
||||
However to create a proper DMARC entry I suggest you use [one of](http://www.kitterman.com/dmarc/assistant.html) [the](https://dmarcian.com/dmarc-inspector/) [web](https://www.unlocktheinbox.com/dmarcwizard/) [sites](https://www.agari.com/resources/tools/dmarc) that aid you there and explain the restrictions and extra features.
|
||||
|
|
@ -0,0 +1,123 @@
|
|||
---
|
||||
title: Managing users, aliases and domains
|
||||
slug: ispmail-bookworm/managing-users-aliases-and-domains
|
||||
sidebar:
|
||||
order: 220
|
||||
---
|
||||
|
||||
import { Tabs, TabItem } from '@astrojs/starlight/components';
|
||||
|
||||
Maybe you already know what you have to do to create mail domains and mail users. After all I tried to explain the database schema in the section that dealt with preparing the database. But if that wasn’t clear enough let me explain what you need to do to manage your mail accounts.
|
||||
|
||||
## Using SQL queries
|
||||
|
||||
The following sections explain the changes and SQL queries you can use for common management tasks:
|
||||
|
||||
### Create a new mail domain
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Explanation">
|
||||
Insert a new row into the virtual_domains table and set the “name” to the name of the new domain. (Do not forget to set up SPF and DKIM.)
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
INSERT INTO virtual_domains (name) VALUES ("example.org");
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Delete a mail domain
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Explanation">
|
||||
Delete the row from the virtual_domains table that has the right “name”. All aliases and users will automatically be deleted, too. However the mailboxes will stay on disk at /var/vmail/… and you need to delete them manually.
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
DELETE FROM virtual_domains where name='example.org';
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Create a mail user
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Explanation">
|
||||
Find out the “id” of the right domain from the virtual_domains table. The insert a new row into the virtual_users table. Set the domain_id to the value you just looked up in the virtual_domains table. Set the “email” field to the complete email address of the new user. Create a new password in a shell using the “dovecot pw -s BLF-CRYPT” command and insert the result into the “password” field.
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
INSERT INTO virtual_users (domain_id, email, password) VALUES (SELECT id FROM virtual_domains WHERE name='example.org'), 'john@example.org','{BLF-CRYPT}$2y$05$.We…';
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Change the password of a user
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Explanation">
|
||||
Find the row in the virtual_users table by looking for the right “email” field. Create a new password in a shell using the “dovecot pw -s BLF-CRYPT” command and insert the result into the “password” field.
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
UPDATE virtual_users SET password='{BLF-CRYPT}$2y$05$.We…' WHERE email='email@address';
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Delete a mail user
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Explanation">
|
||||
Find the row in the virtual_users table by looking for the right “email” field and delete it. The mailbox will stay on disk at /var/vmail/… and you need to delete it manually
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
DELETE FROM virtual_users WHERE email='john@example.org';
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Create a mail forwarding
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Explanation">
|
||||
You can forward emails from one (source) email to other addresses (destinations) – even outside of your mail server. Find out the “id” of the right domain (the part after the “@” of the source email address) from the virtual\_domains table. Create a new row in the virtual\_aliases table for each destination (if you have multiple destination addresses). Set the “source” field to the complete source email address. And set the “destination” field to the respective complete destination email address.
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
INSERT INTO virtual_aliases (domain_id, source, destination) VALUES ( (SELECT id FROM virtual_domains WHERE name='example.org'), 'melissa@example.org', 'juila@example.net');
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### Delete a mail forwarding
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="Explanation">
|
||||
Find all rows in the virtual\_aliases table by looking for the right “source” email address. Remove all rows that you lead to “destination” addresses you don’t want to forward email to.
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
DELETE FROM virtual_aliases WHERE source='melissa@example.org';
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
## Web interfaces
|
||||
|
||||
If you don’t like using SQL queries to manage your mail server you may like to install a web-based management software. Several developers contributed web interfaces for earlier versions of this guide and they will probably still work because the database schema has not changed. Your experience with these projects, or links to further projects, is very welcome in the comments.
|
||||
|
||||
### ISPmail Admin
|
||||
|
||||
Homepage: [http://ima.jungclaussen.com/](http://ima.jungclaussen.com/)
|
||||
Demo: [http://ima.jungclaussen.com/demo/](http://ima.jungclaussen.com/demo/)
|
||||
|
||||

|
||||
|
||||
### ispmail-userctl
|
||||
|
||||
Christian G. has created a text-based program to help you manage your mail accounts. You may like it if you just want a little help adding accounts and setting passwords but not provide a full blown web interface.
|
||||
|
||||
You can find his [Python script at Github](https://github.com/cgzones/ispmail-userctl).
|
||||
|
|
@ -0,0 +1,60 @@
|
|||
---
|
||||
title: Monitoring and Backup
|
||||
slug: ispmail-bookworm/monitoring-and-backup
|
||||
sidebar:
|
||||
order: 230
|
||||
---
|
||||
|
||||
You have come a long way. Glad you made it. So many weird things complicated the journey that evolved from the very beginning of using email roughly 35 years ago. But before you take your much deserved break from these adventures let’s take care of some final tasks.
|
||||
|
||||
## Monitoring
|
||||
|
||||
For me there is nothing more awkward than having your users tell me that my systems are not working properly. As a good sysadmin I should notice outages in advance. The most frequent cause for mail server issues is a disk that is running full. Users are pretty bad with cleaning their mess up. For them the space of their mailbox is infinite. So one way is to use quotas where possible. But at least you should get notified of error conditions on your mail server.
|
||||
|
||||
I will not dive into monitoring deeply here. Setting up proper monitoring will easily fill a guide similar to the size of the one you are currently reading. But you should either have your own monitoring system set up or at least use an internet service to check if emails are getting through.
|
||||
|
||||
Things I suggest you check frequently:
|
||||
|
||||
- Is there enough space on disk? Especially /var/vmail?
|
||||
- Are the SMTP, Submission, POP and IMAP ports responding from other servers?
|
||||
- Is your mail server’s IP address blacklisted? There [are](http://rbl-check.org/) [several](https://www.debouncer.com/blacklistlookup) [services](https://mxtoolbox.com/blacklists.aspx) [you](https://www.dnsbl.info/) [can](https://dnschecker.org/ip-blacklist-checker.php) use.
|
||||
- Are any emails coming through? (I just do a simple count like “`grep -c relay= /var/log/mail.log`“)
|
||||
- Number of emails in the mail queue? (mailq)
|
||||
- Are your backups restoreable? A propos backups…
|
||||
|
||||
## Backup
|
||||
|
||||
No matter if you are running your family mail server or are in charge for an organisation – nobody likes to lose email. There are many different backup tools for different purposes. Just to name my open-source favorites that ship with Debian:
|
||||
|
||||
- [rsnapshot](https://rsnapshot.org/). A simple solution if you want to send backups to other servers using SSH/SCP. You can also keep backups for different points in time on your local server. But if your disk crashes then all would be lost. Also rsnapshot can become slow with many emails because every file is copied as a link. But you can access the files directly without any restore procedure. It’s basically a frontend for _rsync_ which is very reliable.
|
||||
- [borg](https://borgbackup.readthedocs.io/en/stable/). This tools needs to be installed both on the client and the server. Setting up an archive (where backups live) on the server feels a bit weird. But you are rewarded by compression, encryption and deduplication. Also data is stored in chunks which takes less I/O than rsnapshot. It is very efficient.
|
||||
- [restic](https://restic.net/). Similar to Borg in a way. It also offers deduplication, compression and encryption. It’s main advantage is to be able to store backups in S3 buckets. Consider cheap storage providers like [Wasabi](https://wasabi.com/cloud-storage-pricing/) or [Backblaze](https://www.backblaze.com/). However purging old data through S3 is a very resource-intensive job. And Restic generally eats a lot of RAM and disk cache.
|
||||
- [Bareos](https://www.bareos.org/). Formerly known as Bacula until its author started to have crazy ideas and these guys forked it. This is a full-fledged system to handle many servers in parallel and store things on different media – for example LTO magnetic tapes in a big tape library.
|
||||
- Snapshots. If your mail server is running on a virtual server then check if your ISP is offering snapshot backups. My favorite ISP charges 20% extra costs for the service but it is totally worth it. If you break your server for any reason you can just restore it to the old glory without fiddling with configuration files. It does not replace a file-based backup though. So if a user accidentally deletes an email and wants to get it back then this kind of backup will not be right.
|
||||
|
||||
If you cannot backup your entire disk then at least include these directories:
|
||||
|
||||
- /etc (configuration)
|
||||
- /var/vmail (the actual emails)
|
||||
- /var/log (in case the system breaks you may be able to figure out what happened last)
|
||||
- /var/backups (for general backups)
|
||||
- /var/lib/rspamd/dkim/ (private part of the DKIM keys)
|
||||
|
||||
Also make sure to exclude:
|
||||
|
||||
- /var/lib/mysql
|
||||
|
||||
Instead run…
|
||||
```
|
||||
mysqldump mailserver > /var/backups/mailserver.sql
|
||||
```
|
||||
…daily to get a proper database backup that can be restored later.
|
||||
|
||||
I also like to backup the output of these files because they help me figure out the specific layout of a server:
|
||||
```
|
||||
fdisk -l
|
||||
lshw
|
||||
lsb_release -a
|
||||
dpkg --get-selections
|
||||
```
|
||||
If one of your users just accidentally deleted an email and wants it back then simply get the files from /var/vmail/domain/user/Maildir/… from the backup and put them into the original location. Dovecot will handle everything else. You don’t need to restart anything.
|
||||
19
src/content/docs/ispmail-bookworm/240-ansible.mdx
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
---
|
||||
title: Automatic installation with Ansible
|
||||
slug: ispmail-bookworm/automatic-installation-with-ansible
|
||||
sidebar:
|
||||
order: 240
|
||||
---
|
||||
|
||||
If you want to set up multiple mail servers then automation is your friend. Nowadays [Ansible](https://docs.ansible.com/) is commonly used for that purpose. If you are dealing with system administration you should have such a tool in your toolbox.
|
||||
|
||||
I have created an [Ansible playbook on Github](https://github.com/Signum/ispmail-bookworm-ansible) that you can download and use. Using it is pretty simple:
|
||||
|
||||
```
|
||||
apt install ansible git
|
||||
git clone https://github.com/Signum/ispmail-bookworm-ansible.git
|
||||
cd ispmail-bookworm-ansible/ansible
|
||||
ansible-playbook ispmail.yml -c local
|
||||
```
|
||||
|
||||
Please follow the hints in the README.md within the repository.
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
title: Success stories
|
||||
slug: ispmail-bookworm/success-stories
|
||||
sidebar:
|
||||
order: 250
|
||||
---
|
||||
|
||||
I would appreciate if you leave a comment here telling others if your mail server is running. Perhaps even a word about the purpose of the mail server and the number of users you have. That allows others to see that their time is well-spent and indeed leads to a working mail server. Thanks for your feedback.
|
||||
53
src/content/docs/ispmail-bookworm/260-faq.mdx
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
---
|
||||
title: FAQ (frequently asked questions)
|
||||
slug: ispmail-bookworm/faq-frequently-asked-questions
|
||||
sidebar:
|
||||
order: 260
|
||||
---
|
||||
|
||||
## Can I use this guide for my own work?
|
||||
|
||||
Previously I claimed a copyright on the documentation. However let’s make things simpler. This Bullseye guide is using the [Creative Commons BY-NC-SA license](https://creativecommons.org/licenses/by-nc-sa/4.0/). Which means:
|
||||
|
||||
- BY = Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
|
||||
- NC = NonCommercial — You may not use the material for commercial purposes. Do what you want with your mail server. But let’s keep the knowledge free.
|
||||
- SA = ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
|
||||
|
||||
Besides that this guide has been a team effort. Many readers have contributed to it. I may spend the time to turn it into a guide but a lot of good ideas were not mine alone.
|
||||
|
||||
## Can I advertise in your guide?
|
||||
|
||||
No. There is no financial motivation. I just strive for freeing the world of email services once again.
|
||||
|
||||
## Why don’t you use Nginx/PostgreSQL?
|
||||
|
||||
I personally prefer that software, too. Just go for it. But for the sake of consistency let’s keep Apache and MySQL as default choices in this guide so that readers upgrading from previous versions don’t have to switch.
|
||||
|
||||
## Do you offer paid support?
|
||||
|
||||
Generally, yes. Aside from my day job, I am a sysadmin freelancer. If you are just stuck at some point you may want to ask in the [chat](https://riot.im/app/#/room/#ispmail:matrix.org) first though. You can find me there as well.
|
||||
|
||||
## How do I filter out malware?
|
||||
|
||||
I understand the need for it. If your users are using low-security operating systems (e.g. those that run EXE files) they are threatened by a lot of security problems. From the perspective of an email provider I would not want to take responsibility for filtering emails. That is only working on symptoms and not addressing the actual issues. Security is a fundamental concept and not something you slap on top of something inherently broken. I would not want to be blamed for a crypto trojan infection because the user relied on me saving them from bad things to happen. Client security in non-free systems is a topic on its own.
|
||||
|
||||
## Debian packages are too old and insecure
|
||||
|
||||
If you use a Debian stable release you have chosen to stay on a specific software version. “Stable” means “a version that will not change”. That’s the best choice for a server. Those software packages are by no means less secure though because they also get security updates. Debian also has up-to-date packages but you surely don’t want to use them for a mail server unless you feel adventurous or have an intense hatred for your users. For a detailed discussion see [this separate article](https://workaround.org/debian-packages-are-so-old/).
|
||||
|
||||
## Can I download the guide as one file?
|
||||
|
||||
Unfortunately there is no easy way to accomplish that. Especially with interactive components like tabs of quizzes. The reasons are technical – not malice.
|
||||
|
||||
## Your guide looks cool. How did you make it?
|
||||
|
||||
The site is generated using [Starlight](https://starlight.astro.build/). Discussions at the bottom of each page are powered by [Comentario](https://comentario.app/).
|
||||
Diagrams have been created using [Inkscape](https://inkscape.org/). User tracking is handled by [Matomo](https://matomo.org/).
|
||||
Everything here has been generated using open-source software and is not using any public cloud services.
|
||||
|
||||
## I can offer to translate your guide.
|
||||
|
||||
That is very kind to offer help. I understand the motivation to give back something. But trust me: you will lose interest quickly and regret your offer.
|
||||
The guide is huge. It may not appear like much but once you start typing you realize just how many months of work went into it.
|
||||
Even translating will take 1-2 weeks of your time. And after that someone would have to maintain both the original version and your translation.
|
||||
We tried that a couple of times and it did not turn out well.
|
||||
|
|
@ -0,0 +1,130 @@
|
|||
---
|
||||
title: Migrating from your old (Bullseye) server
|
||||
lastUpdated: 2023-09-24
|
||||
slug: ispmail-bookworm/migrating-from-a-bullseye-to-a-bookworm-server
|
||||
sidebar:
|
||||
order: 30
|
||||
---
|
||||
|
||||
|
||||
## Upgrade or fresh installation?
|
||||
|
||||
I recommend that you set up a new server and make sure that the new mail server is working correctly before you start migrating existing email users to it. You may argue that Debian can be upgraded easily using “apt-get dist-upgrade” but that is very dangerous on a live mail server. Automatic configuration changes may have evil side effects and you risk losing emails. At least you are causing a downtime for your users.
|
||||
|
||||
If you follow my advice then get a new server and install Debian Bookworm on it. Once the user mailboxes are migrated and all works well you can tear down the old server.
|
||||
|
||||
Once your new server is installed come back here.
|
||||
|
||||
## Read before you type
|
||||
|
||||
Follow this guide to the end. Only then start attempting a migration. Inform your users about the migration and set a time when you intend to move email accounts to the new server. The change will require a DNS change which takes a while to be visible worldwide so your users will have a period when incoming email is delayed. If you proceed careful though not a single email will be lost.
|
||||
|
||||
So you have your new server up and running and did everything to make it a working mail server? You really read all the pages in this guide and did what they told you? Okay, then let’s start.
|
||||
|
||||
## Turn down the TTL of your MX record
|
||||
|
||||
The DNS “MX” record for your domain contains the hostname of your mail server. When switching to the new server you need to change the MX record. Every DNS record has a TTL (time-to-live) that defines the period of time that a record will stay valid even after you change it. Usually that TTL is rather high like 86400 seconds (=1 day). This information is used by caching name servers that they can use the cached values for a day. Turn that TTL down temporarily to 60 seconds so that the rest of the internet will pick up your change quicker. However it will take a day until everyone else on the internet picks up your TTL change.
|
||||
|
||||
## Migrate the mailserver database
|
||||
|
||||
You need to copy the database that contains the control data about your email domains and accounts. Log into the old (Bullseye) server as root and back up the _mailserver_ database. That is as easy as running…
|
||||
|
||||
```
|
||||
mysqldump mailserver > mailserver.sql
|
||||
```
|
||||
Copy that file to the new server (using _scp_) and import it there:
|
||||
|
||||
```
|
||||
mysql mailserver < mailserver.sql
|
||||
```
|
||||
|
||||
Obviously any database changes on the old server from now on will have to be done on the new server as well until the migration is done.
|
||||
|
||||
## Roundcube contacts
|
||||
|
||||
If your users are using Roundcube as a webmail interface then you should migrate their data like their contact lists. Dump the SQL from the old server:
|
||||
|
||||
```
|
||||
mysqldump roundcube > roundcube.sql
|
||||
```
|
||||
|
||||
Copy that file to the new server and import it:
|
||||
|
||||
```
|
||||
mysql roundcube < roundcube.sql
|
||||
```
|
||||
|
||||
One caveat though. To distinguish multiple mail servers Roundcube stores the server’s name in the mail_host column of the users table. So as a last step change that column if your new mail server has a new FQDN/hostname by running this SQL query on the new roundcube database:
|
||||
|
||||
```sql
|
||||
UPDATE users SET mail_host='new.mail.server';
|
||||
```
|
||||
|
||||
## Migrate rspamd spam training data
|
||||
|
||||
If you have been using rspamd with the Redis backend then copy over the Redis database from your previous server. Details are found in the rspamd chapter.
|
||||
|
||||
## Migrate the Maildirs hot
|
||||
|
||||
Fortunately Dovecot uses the maildir format that stores emails as plain files on disk. Login to the new (Bookworm) server and use _rsync_ to copy over the mails from the old (Bullseye) mail server:
|
||||
|
||||
```
|
||||
rsync -va oldserver:/var/vmail/ /var/vmail/
|
||||
```
|
||||
|
||||
(Note the trailing slashes. Type them exactly as shown above or your files will end up in wrong places.)
|
||||
|
||||
There is no need to shut down Dovecot on your production Bullseye server. Copying the files while Dovecot is running will not break anything. This is called a “hot copy”. It may not be consistent but it will save time during the final synchronization.
|
||||
|
||||
## Copy certificates
|
||||
|
||||
Copy over everything in /etc/letsencrypt and /var/lib/rspamd/dkim from your old to the new server.
|
||||
|
||||
```
|
||||
rsync -va oldserver:/etc/letsencrypt/ /etc/letsencrypt/
|
||||
rsync -va oldserver:/var/lib/rspamd/dkim/ /var/lib/rspamd/dkim/
|
||||
```
|
||||
|
||||
## Downtime
|
||||
|
||||
You told your users about the downtime, right? The time has come? Okay. Shut down Dovecot on both servers.
|
||||
|
||||
## Migrate the emails cold
|
||||
|
||||
Let’s synchronize again. _rsync_ will only copy those files that have changed which makes it much faster than the first sync. On your new server run:
|
||||
|
||||
```
|
||||
rsync -va --delete oldserver:/var/vmail/ /var/vmail/
|
||||
```
|
||||
|
||||
(The “`--delete`” option makes sure that files that have been removed from the old server will also be deleted from the new server. So if a user has deleted an email it will be deleted on the new server as well.)
|
||||
|
||||
The new Dovecot version uses a slightly different indexing mechanism. So force rebuilding the users’ indices:
|
||||
|
||||
```
|
||||
doveadm force-resync -A '*'
|
||||
```
|
||||
|
||||
## Switch the DNS records
|
||||
|
||||
For all your domains you will have to change the DNS “MX” or “A” record to point to your new server.
|
||||
|
||||
## Enable soft_bounce
|
||||
|
||||
Accidents happen. And you don’t want to lose emails. So run this command to enable your safety net on the new server:
|
||||
|
||||
```
|
||||
postconf soft_bounce=yes
|
||||
```
|
||||
|
||||
This makes Postfix always keep emails in the queue that it would otherwise reject. So you can fix any errors and the queue will empty. Start Postfix and Dovecot on the new server. Watch your /var/log/mail.log and run “mailq” from time to time to see what emails get stuck in the queue. If you are certain that emails can be removed from the queue then use “postsuper -d QUEUE-ID” (as shown in the “mailq” output).
|
||||
|
||||
Once you are certain that emails are properly received and sent you can switch off the _soft_bounce_ mode again:
|
||||
|
||||
```
|
||||
postconf soft_bounce=no
|
||||
```
|
||||
|
||||
## Shut down the old server
|
||||
|
||||
If possible do a final backup of the old server. If users are not complaining then dismiss the old system after a week.
|
||||
98
src/content/docs/ispmail-bookworm/40-big-picture.md
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
---
|
||||
title: The big picture
|
||||
lastUpdated: 2023-09-24
|
||||
slug: ispmail-bookworm/big-picture
|
||||
sidebar:
|
||||
order: 40
|
||||
---
|
||||
|
||||
The mail server that you are about to set up uses several software components. Let me first explain briefly what the purpose of each software is:
|
||||
|
||||
- **[Debian](https://www.debian.org/)** “Bookworm” – the operating system
|
||||
- **[Postfix](http://www.postfix.org/)** receives incoming emails from the internet and sends out outgoing emails to other mail servers. It is the software that speaks SMTP.
|
||||
- **[rspamd](https://rspamd.com/)** runs sanity checks on an incoming email to determine whether it is spam.
|
||||
- **[Dovecot](https://www.dovecot.org/)** stores emails on your hard disk, applies filters and lets your users fetch their emails using the POP3 and IMAP protocols
|
||||
- **[Roundcube](https://roundcube.net/)** is a webmail interface so users can read their emails using a web browser, manage their email rules and change their password
|
||||
- **[MariaDB](https://mariadb.org/)** (formerly known as _MySQL_) is a database that stores information about your domains, email aliases and email accounts
|
||||
|
||||
## What happens when someone sends you an email?
|
||||
|
||||
Let us assume that you are responsible for the email domain _example.org_ and someone on the internet sends an email to `john@example.org`. This is what happens step by step:
|
||||
|
||||
1. **REMOTE:** Hey, DNS server. I have an email for someone in the _example.org_ domain. Can you tell me the name of the responsible mail server?
|
||||
**DNS:** According to the DNS zone of _example.org_ it is _mx.example.org_.
|
||||
2. **REMOTE**: Nice. Do you have an an IP address for _mx.example.org_?
|
||||
**DNS:** I have an IPv4 address here. It is _85.25.72.76_.
|
||||
3. _REMOTE connects to that IP address on TCP port 25 which is by definition used for SMTP – the [simple mail transport protocol](https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol)._
|
||||
**POSTFIX:** Welcome. I am Postfix. Who is there? (“220 mx.example.org ESMTP Postfix”)
|
||||
**REMOTE:** Hi, I am remote server. (“EHLO remoteserver”)
|
||||
**POSTFIX:** Nice to meet you. I can offer a few features like pipelining and encryption by the way. (“STARTTLS, PIPELINING, SIZE 4000000, …”)
|
||||
**REMOTE:** Let’s switch to an encrypted connection. (“STARTTLS”)
|
||||
_(The connection is now using TLS encryption.)_
|
||||
**REMOTE:** I have an email from `donald@duck.com` here. (“MAIL FROM:<`donald@duck.com`>”)
|
||||
**POSTFIX:** I see. (“Ok”)
|
||||
**REMOTE:** The email is meant for `john@example.org`. (“RCPT TO:<`john@example.org`>”)
|
||||
4. **POSTFIX:** Hey database. _(Connects to TCP port 3306 on the local host to talk to MariaDB.)_ Could you check if _example.org_ is one of our mail domains? (“SELECT … from virtual_domains …”)
|
||||
**MariaDB:** Yes, I have a domain like that.
|
||||
**POSTFIX:** Oh, good. And do you have a mailbox or forwarding for someone called `john@example.org`? (“SELECT … from virtual\_aliases/virtual\_users …”)
|
||||
**MariaDB:** Yes, there is a mailbox for that address.
|
||||
**POSTIFX:** Hey, remote server. The recipient looks good. (“Ok”)
|
||||
**REMOTE:** Roger. Then here’s the actual email. (“DATA”)
|
||||
_(The remote server sends the email header and body.)_
|
||||
5. _(POSTFIX connects to port 11332 on the local host to reach the rspamd.)_
|
||||
**POSTFIX**: Hey, rspamd. I have a new email here. Could you give it a look for, you know, spam and stuff?
|
||||
**RSPAMD:** Sure. Well, there are a few minor issues. But generally the mail looks good. I suggest you accept it.
|
||||
**POSTFIX:** Hey, remote server. Your email is fine.
|
||||
6. _(Postfix uses a socket file at /var/spool/postfix/private/dovecot-lmtp to talk to Dovecot.)_
|
||||
**POSTFIX:** Hey, Dovecot. Here is a new email for `john@example.org`.
|
||||
**DOVECOT:** Got it.
|
||||
7. _(Dovecot checks for additional Sieve rules and then stores the email on disk at /var/vmail/example.org/john/Maildir/INBOX)_
|
||||
|
||||
Phew, that was a lot. I hope it helps to understand though how the different components are involved until an email gets delivered.
|
||||
|
||||
The next situation I would like to explain is…
|
||||
|
||||
## What happens if a user fetches their email using IMAP/POP3?
|
||||
|
||||
This process is way simpler. It looks like this:
|
||||
|
||||

|
||||
|
||||
User connects through IMAP
|
||||
|
||||
1. The user usually has a mail client (also called a _[mail user agent](https://en.wikipedia.org/wiki/Email_client)_) that can use the POP3 or IMAP protocol to fetch emails from the mail server. I prefer Thunderbird for example. That mail client connects to the POP3 (TCP 110) or IMAP (TCP 143) port, sends the STARTTLS command that initiates an encrypted connection and sends the user’s username (which equals the email address in our case) and their password. The client may as well use the secure TLS-encrypted ports directly – 995 for POPs and 993 for IMAPs. If possible users should use IMAP. POP3 is pretty much deprecated nowadays and lacks support for multiple folders on the server.
|
||||
2. Dovecot sends a query to the MySQL database and verifies that the username and password belong to a known user. The password is not stored in plain text in the database. Instead it computes the password [hash](https://en.wikipedia.org/wiki/Hash_function) and compares it to the hash in the database. If the password is wrong then Dovecot will refuse the login.
|
||||
3. We have a fixed scheme how emails are stored on disk. So if the user is called `jane@example.org` then Dovecot will look for email files in _/var/vmail/example.org/jane/Maildir/…_ and send the user the requested emails.
|
||||
|
||||
Nowadays many users seem to prefer webmail to a mail client installed on their computers. As an email power user I honestly do not understand that trend. But who am I to judge. So let’s take a look how webmail works technically:
|
||||
|
||||
## What happens if a user reads their email using web mail?
|
||||
|
||||
The Roundcube software that provides the web mail interface is basically a PHP software that is a gateway between HTML pages and a built-in IMAP client. So when a user uses their browser to connect to the web mail interface…
|
||||
|
||||

|
||||
|
||||
User connects through webmail
|
||||
|
||||
1. The user points the browser to the HTTPS URL of the webmail interface. Apache web server receives the connection and runs the PHP scripts of Roundcube. Roundcube shows a login form for the username and password. The user enters the username (that equals the email address) and the password and submits the login form.
|
||||
2. Roundcube connects to Dovecot using IMAP and forwards the username and password to check if they are valid.
|
||||
3. Dovecot treats this connection similar to a mail client connection. It queries the MariaDB database to verify the username (=email address) and password.
|
||||
4. If the authentication was successful Dovecot fetches the mail files from disk and sends them through IMAP to Roundcube. Roundcube then renders the emails as HTML and the user can read it.
|
||||
|
||||
So you see that the web mail access also works through IMAP. The user does not realize that though.
|
||||
|
||||
Okay, the final scenario I would like to explain is…
|
||||
|
||||
## What happens if the user wants to send an email to the internet?
|
||||
|
||||
Of course your users also want to send emails to other internet users. But they cannot send the email directly to the destination mail server. First their mail client does not know which destination server is responsible for the recipient (hint: DNS) – that functionality just is not built in. And second the user is likely assigned a _dynamic IP address_ which is blocked by most mail servers because they tend to get abused by infected Windows PCs that send out spam. So the correct way to send an email to the internet is through your mail server. This is called _relaying_ because your mail server acts as a relay. In this example your user wants to send an email to `fred@example.net`.
|
||||
|
||||

|
||||
|
||||
Relaying/sending email via the mail server
|
||||
|
||||
1. The user writes the email in their mail client and clicks on “Send”. The mail client establishes an SMTP connection to Postfix. To make sure that the user is allowed to send email through your system it requires a username and password along with the email. This information is sent in an encrypted way.
|
||||
2. Postfix could now check the password in the database directly. But as Dovecot already knows how to handle authentication it is easier to ask Dovecot to verify the username and password. (SMTP authentication in Postfix is surprisingly ugly.)
|
||||
3. Dovecot now sends a query to the MariaDB database to check if the username and password (hash) are correct and tells Postfix the result.
|
||||
4. Postftix knows now that it is authorized to send the email on behalf of the user. It tells the user that it successfully accepted the email. The email is put into Postfix’s mail queue for further processing. Postfix will now query a DNS (name server) to determine the responsible destination mail server. As the recipient has an “…@example.net” address it checks the MX record of the “example.net” domain and then gets the respective IP address.
|
||||
5. Postfix now knows which mail server to send the email to. It opens an SMTP connection and delivers the email.
|
||||
249
src/content/docs/ispmail-bookworm/50-types-of-email-domains.mdx
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
---
|
||||
title: Types of email domains
|
||||
lastUpdated: 2023-09-25
|
||||
slug: ispmail-bookworm/types-of-email-domains
|
||||
sidebar:
|
||||
order: 50
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
This is the pretty boring but at the same time very important part of the tutorial. Do not skip it. Most problems that
|
||||
readers have with their mail servers are caused by a misunderstanding of the different types of email domains. There is
|
||||
nothing you need to do on your server right now. Just lean back and relax and make sure that you read and fully
|
||||
understand this page. It will spare you frustration.
|
||||
|
||||
As explained on the previous page Postfix is the software component that speaks SMTP. It receives emails from other
|
||||
server and sends email out. Of course you will only want to receive emails for your domains only.
|
||||
|
||||
Postfix will accept emails for any of these [classes of domains](http://www.postfix.org/ADDRESS_CLASS_README.html):
|
||||
|
||||
1. Canonical domains (also called _local domains_)
|
||||
2. Hosted domains
|
||||
- Virtual mailbox domains
|
||||
- Virtual alias domains
|
||||
3. Relay domains
|
||||
|
||||
Let us look at the first two in detail. The third one – relay domains – is not needed in this context.
|
||||
|
||||
## Canonical domains
|
||||
|
||||
Once upon a time users logged into servers using [Telnet](https://en.wikipedia.org/wiki/Telnet) or
|
||||
[SSH](https://en.wikipedia.org/wiki/Secure_Shell) to read their email in basic text-based mail clients. Their email
|
||||
addresses were often just a combination of their login name and the server name. Something like
|
||||
`mpauls@server17.biology.example.net`.
|
||||
|
||||
At least that made things pretty simple for the server administrator. He just accepted emails for any valid login user
|
||||
on that server. The mail server knew what to do. It checked if “mpauls” is found in /etc/passwd and then stored emails
|
||||
in “/var/mail/mpauls”.
|
||||
|
||||
If you wanted to do it that way…
|
||||
|
||||
1. You would need to create system accounts for all users who want to receive email. Every system user is a potential
|
||||
security risk because they can login to the server using a shell. If the user chooses a weak password you are
|
||||
essentially inviting attackers to harm your server.
|
||||
2. If you want to host many email accounts this becomes impractical.
|
||||
3. Postfix cannot distinguish the local domains. If you have three local domains “example.org”, “example.com” and
|
||||
“example.net” then all these email addresses would lead to the same mailbox: `john@example.org`, `john@example.com`
|
||||
and `john@example.net`. So you cannot use different domains for different purposes.
|
||||
|
||||
To tell Postfix which domain you consider local you list them in the “mydestination” configuration setting in your
|
||||
main.cf configuration file. Example:
|
||||
|
||||
```
|
||||
mydestination = example.org, example.com, example.net
|
||||
```
|
||||
|
||||
<Aside type="caution">
|
||||
A domain never belongs to more than one class. You would get a warning in your log file and Postfix will start
|
||||
behaving strangely. One domain can only have one purpose. This is a frequent mistake.
|
||||
</Aside>
|
||||
|
||||
It is common however to use the server’s hostname as a local domain. If your server is called “scully.example.com” you
|
||||
could set “mydestination = scully.example.com”. Or you just set it to “mydestination = localhost” if you rather want to
|
||||
use “scully.example.com” as a virtual domain. Some parts of your system may send emails to root@localhost so this is a
|
||||
sane setting.
|
||||
|
||||
## Virtual domains
|
||||
|
||||
This is the class of domains that will be our workhorse. Postfix allows us to add an unlimited number of domains that
|
||||
can receive emails. That’s what an ISP does and so will we.
|
||||
|
||||
Where do we put the list of domains? Postfix is pretty flexible. We could put it in a huge text file. We might use LDAP
|
||||
if we have some kind of user directory in our organization. But in this guide we will store that information in a simple
|
||||
MariaDB database.
|
||||
|
||||
Postfix handles hosted email addresses by checking two _mappings_:
|
||||
|
||||
- Virtual Aliases (“redirect this email address to another address”)
|
||||
- Virtual Mailboxes (“accept email for this email address”)
|
||||
|
||||
So much terminology. Time for an example, isn’t it? Let’s tell Postfix we want to have three email addresses:
|
||||
|
||||
- `john@example.org`
|
||||
- `jack@example.org`
|
||||
- `jack@example.com` _(note that this is another domain than example.org)_
|
||||
|
||||
Postfix only understands _mappings_. They are much easier than they sound. A mapping is just a table that has a left and
|
||||
a right column. Postfix always looks for stuff in the left column (_key_) to find more information in the right column
|
||||
(_value_). Like this:
|
||||
|
||||
| Virtual mailbox (key) | Virtual mailbox location on disk (value) |
|
||||
| --------------------- | ---------------------------------------- |
|
||||
| `john@example.org` | /var/vmail/example.org/john/Maildir |
|
||||
| `jack@example.org` | /var/vmail/example.org/jack/Maildir |
|
||||
| `jack@example.com` | /var/vmail/example.com/jack/Maildir |
|
||||
|
||||
So the left column lists the valid email addresses. And the right column is apparently some path on disk. Right, that’s
|
||||
all the magic. If an email for `jack@example.org` is received then Postfix will find the entry in the left column and
|
||||
can figure out where Jack’s mails are stored on disk.
|
||||
|
||||
Before checking if a specific email address is valid Postfix first checks if it is responsible for the domain at all.
|
||||
That’s done by this mapping:
|
||||
|
||||
| Virtualdomain (key) | Whateve (value) |
|
||||
| ------------------- | --------------- |
|
||||
| example.org | Kittens |
|
||||
| example.com | Puppies |
|
||||
|
||||
So there are two domains in the left column. But what is that in the right column? Kittens? Really? Well, the truth is:
|
||||
it doesn’t matter. It can be anything. So why is that column there at all? The reason is that _mappings_ always have two
|
||||
columns. Postfix wants a list of domains but it has no concepts for one-dimensional _lists_. So it uses this format and
|
||||
ignores the right column.
|
||||
|
||||
Essentially these two mappings are all we need. Add a little configuration and you can already receive emails. However
|
||||
Postfix provides another useful feature: _aliases_. An alias is a redirection (or _forwarding_) of one email address to
|
||||
one or more other addresses. Possible uses:
|
||||
|
||||
- forward postmaster@… for all your domains to one mailbox
|
||||
- create an email distribution list for a team or department
|
||||
- forward copies of all your emails to another address
|
||||
- redirect emails meant for a coworker who has left the organisation
|
||||
|
||||
As usual, Postfix expects a mapping for aliases. Time for another example:
|
||||
|
||||
| Step | Virtual email address (left) | Redirect to (right) |
|
||||
| :--- | :--------------------------- | :---------------------------------------- |
|
||||
| 1 | `postmaster@example.org` | `jack@example.com` |
|
||||
| 2 | `abuse@example.org` | `jack@example.com` |
|
||||
| 3 | `jack@example.com` | `jack@example.com`,`jacky@workaround.org` |
|
||||
| 4 | `sophie@example.net` | `hr@example.net` |
|
||||
| 5 | `hr@example.net` | `rick@example.net` |
|
||||
| | `hr@example.net` | `tina@example.net` |
|
||||
| 6 | `@example.org` | `jack@example.com` (DANGEROUS) |
|
||||
|
||||
Quite a lot happens here. How does Postfix interpret this table?
|
||||
|
||||
1. Redirect emails for `postmaster@example.org` to `jack@example.com`
|
||||
2. Redirect emails for `abuse@example.org` also to `jack@example.com`
|
||||
3. Keep a copy of an incoming email for `jack@example.com` in his mailbox and send another copy to
|
||||
`jacky@workaround.org.` Yes, you can use multiple email addresses seperated by commas.
|
||||
4. Sophie has left the organisation so her address is forwarded to HR.
|
||||
5. The HR team consists of two members: `rick@example.net` and `tina@example.net.` Every email to `hr@example.net` will
|
||||
be forwarded to each of them. This works similar to (3) but this time the target addresses are not seperated by
|
||||
commas but stored in different rows.
|
||||
6. Forward any email address of the _example.org_ domain to `jack@example.com.` We call this a _catch-all_ address.
|
||||
This is dangerous for two reasons. First it will send the poor Jack anything that is sent to the _example.org_
|
||||
domain if there is no user account for that address. (Yes, spammers guess addresses.) And second it disables a
|
||||
security check in Postfix that the destination address `jack@example.com` actually exists. Postfix may first receive
|
||||
the email and then be confused if the recipient is not reachable. That leads to backscatter and harms your
|
||||
reputation. But don’t worry. I will show you the proper way to use catch-all addresses later.
|
||||
|
||||
So basically this is the way that Postfix handles aliases:
|
||||
|
||||
1. Is the domain of the email address defined as a virtual domain?
|
||||
2. Get all rows of the table where the email address is found in the left column. Send a copy of the email to everyone
|
||||
mentioned in the right column. If the right column contains multiple email addresses separated by commas then split
|
||||
them first.
|
||||
3. If there was no such row then check again if there is an `@domain` row.
|
||||
4. Still nothing found? Then reject the email.
|
||||
|
||||
<Aside type="tip">
|
||||
Two email addresses are mandatory for every domain that you host. `postmaster@domain` and `abuse@domain`. These
|
||||
requirements are documented in RFC 521 and RFC 2142. Be sure to add aliases for them. If an email to those addresses
|
||||
would bounce, your domain reputation will suffer.
|
||||
</Aside>
|
||||
|
||||
## Database
|
||||
|
||||
Did you get the idea of mappings? Two columns in a table? Keys and values? Good. Now how do those mappings work when we
|
||||
want to put the information into an SQL database?
|
||||
|
||||
Basically a relational database works in rows and columns, too. So we can take the format of the tables shown above and
|
||||
put them into database tables. Let’s call these tables…
|
||||
|
||||
- virtual_domains (for the list of domains)
|
||||
- virtual_aliases (for the aliases mapping)
|
||||
- virtual_users (for the email users and their mailboxes respectively)
|
||||
|
||||
I won’t bore you with the SQL stuff right now. We will deal with that in a later chapter. Let’s just briefly cover how
|
||||
Postfix can get data from the database. We provide Postfix with _configuration files_ for that purpose. These files
|
||||
often have a “.cf” suffix (**c**onfiguration **f**ile).
|
||||
|
||||
Let’s take the virtual_users table for example. This is the content of a file that is located at
|
||||
`/etc/postfix/mysql-virtual-mailbox-maps.cf`:
|
||||
|
||||
```
|
||||
# Information on how to connect to your MySQL server
|
||||
user = someone
|
||||
password = some_password
|
||||
hosts = 127.0.0.1
|
||||
dbname = mailserver
|
||||
|
||||
# The SQL query string
|
||||
query = SELECT mailbox_path FROM virtual_users WHERE email_address='%s'
|
||||
```
|
||||
|
||||
The first four lines describe how the database can be connected to. You provide a database user and database password as
|
||||
well as the IP address or hostname of the SQL server and the name of the database.
|
||||
|
||||
A more interesting bit is the _query_ that is defined in the last line. Postfix still thinks in left and right columns.
|
||||
So it will run this query and replace the %s by the email address it is looking for – that’s what is expected in the
|
||||
left column. The database will then get all rows from the database table that match this criterion. If one row matches
|
||||
the query then the “SELECT mailbox_path” will return just the value of what Postfix would expect to be in the right
|
||||
column.
|
||||
|
||||
That’s all the magic that Postfix needs to talk to your SQL database. You tell Postfix how to connect to the database
|
||||
and how the data in the database table corresponds to the left and right columns.
|
||||
|
||||
Note that a lookup here must only return just one row from the database. Postfix must uniquely know where the mailbox
|
||||
path for a given user is. There are other mappings though where it’s allowed to have multiple right-hand side items for
|
||||
one left-hand side item – for example in virtual aliases.
|
||||
|
||||
To use the above configuration file you have to configure it in Postfix’s main.cf file:
|
||||
|
||||
```
|
||||
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual_mailbox_maps.cf
|
||||
```
|
||||
|
||||
If you find that this mapping is not working as you expected then the “postmap -q” command is your friend. You can ask
|
||||
Postfix what the right-hand side value for a given left-side value is. Say that you are interested in the mailbox_path
|
||||
for the email_address `john@example.org`:
|
||||
|
||||
```
|
||||
postmap -q john@example.org mysql:/etc/postfix/mysql-virtual_mailbox_maps.cf
|
||||
```
|
||||
|
||||
Postfix will then run the above SQL query with your “`john@example.org`” argument:
|
||||
|
||||
```
|
||||
SELECT mailbox_path FROM virtual_users WHERE email_address='john@example.org'
|
||||
```
|
||||
|
||||
The result should be:
|
||||
|
||||
```
|
||||
/var/vmail/example.org/john/Maildir
|
||||
```
|
||||
|
||||
In this guide we will use a slight variation of that SQL query:
|
||||
|
||||
```
|
||||
query = SELECT 1 FROM virtual_users WHERE email='%s'
|
||||
```
|
||||
|
||||
Postfix does not need to know where the mailbox is located on disk. We will not use Postfix to store the email to disk.
|
||||
Instead we will make Postfix hand over the email to Dovecot which stores the email. Dovecot has a couple of nice extra
|
||||
features (like server-side scripts) that we want to use. So Postfix just needs to know whether an email address is valid
|
||||
or not. We just return “1” because actually the right column is not considered anyway. Postfix just needs to know
|
||||
whether there is a database row or not.
|
||||
65
src/content/docs/ispmail-bookworm/60-installing-debian.mdx
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
---
|
||||
title: Installing Debian
|
||||
lastUpdated: 2023-09-25
|
||||
slug: ispmail-bookworm/install-debian-bookworm-on-your-server
|
||||
sidebar:
|
||||
order: 60
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
## Option 1: Get a server with Debian Bookworm pre-installed
|
||||
|
||||
If you rent a virtual server at your favorite hosting company it often comes with Debian pre-installed. If you have a chance to change the partitioning then consider the schema shown below. Otherwise feel free to skip this page.
|
||||
|
||||
## Option 2: Get a server with the older Debian Bullseye
|
||||
|
||||
Your hosting company does not offer Debian Bookworm yet but just Debian Bullseye? No big deal. Just change your /etc/apt/sources.list file to:
|
||||
|
||||
```
|
||||
deb http://deb.debian.org/debian bookworm main
|
||||
deb http://deb.debian.org/debian bookworm-updates main
|
||||
deb http://security.debian.org/debian-security bookworm-security main
|
||||
```
|
||||
|
||||
Run “apt update” and “apt dist-upgrade” and you are ready.
|
||||
|
||||
## Option 3: Install Debian yourself
|
||||
|
||||
You have a blank computer or virtual machine? Then simply install Debian Bookworm yourself. Get a boot medium from the [Debian website](https://www.debian.org/distrib/netinst). The smaller _network installer_ is sufficient – it will download all required packages directly from the internet. Make sure you choose ‘English’ as the installation language even if it is not your native language. If you have trouble with the server you will more likely find help when searching for English error messages on the internet.
|
||||
|
||||
Most of the installation is pretty straightforward. You don’t need me holding your hand. I strongly recommend that you use the _logical volume manager_ (LVM) for your partitions to stay flexible if your server grows. See my [article on LVM](https://workaround.org/understanding-lvm/) if you want some help understanding its concept.
|
||||
|
||||
## Partitioning
|
||||
|
||||
The only part that takes some patience is when it comes to partitioning your disk. Throughout this tutorial you will store your users’ emails in the /var/vmail partition. The MariaDB database will live in /var/lib/mysql and is usually pretty small even for a large number of email accounts – so that usually does not require a separate partition. Log files go to /var/log.
|
||||
|
||||
So my recommendation for partitioning:
|
||||
|
||||
- Boot partition, 300 MB, first partition of your disk (e.g. /dev/sda1), mounted at “/boot”
|
||||
- All the rest of the disk goes into one large LVM _physical volume_, second partition of your disk (e.g. /dev/sda2)
|
||||
- LVM _volume group_ consisting of that one _physical volume_
|
||||
- LVM _logical volume_ for “/”, 20 GB, formatted as ext4
|
||||
- LVM _logical volume_ for “/tmp”, 5 GB, formatted as ext4
|
||||
- LVM logical volume for “/var/vmail”, formatted as ext4, as much space for mailboxes as you want to start with
|
||||
|
||||
LVM is very flexible. You can enlarge your root partition or “/var/vmail”. You can even add another physical disk and join it into the _volume group_ to get more space. So just start small.
|
||||
|
||||
The Debian installer helps you set up LVM. In a nutshell these are the major steps:
|
||||
|
||||
- Partitioning method: “Manual”
|
||||
- Select your disk
|
||||
- Create a new partition table if you get asked
|
||||
- Create a new partition of “max” size as a _Primary_ partition
|
||||
- Set the partition to “Use as: physical volume for LVM”
|
||||
- Choose “Configure the Logical Volume Manager”
|
||||
- Create a volume group (call it “vg0” if you like)
|
||||
- Select the partition you created before (e.g. /dev/sda1)
|
||||
- Create the logical volumes for “root”, “tmp” and “mail”.
|
||||
- Edit each LV. Set its type to “ext4”. Set the mount points to
|
||||
- / for root
|
||||
- /tmp for tmp
|
||||
- /var/vmail for mail
|
||||
- Do not add swap space – your server will need very little RAM. You will get asked if you are serious. Yes, you are.
|
||||
|
||||
Write the changes to disk and continue with the rest of the installation. Just make sure that you install the “SSH server” component so that you can login from remote after the installation.
|
||||
|
|
@ -0,0 +1,183 @@
|
|||
---
|
||||
title: Install software packages
|
||||
lastUpdated: 2023-10-03
|
||||
slug: ispmail-bookworm/install-the-software-packages
|
||||
sidebar:
|
||||
order: 70
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
After the basic installation your server will reboot and allow you to login. You can either login at the console (if you have physical access to the system) or using SSH (_secure shell_).
|
||||
|
||||
<Aside type="tip" title="SSH as root with password login?">
|
||||
|
||||
By default the server will not allow logging in as “root” over SSH for security reasons. Instead use the non-root user you created during the installation. Then enter “su -” to become _root_. I recommend you use _SSH public key authentication_ which means creating an SSH key (if you don’t have one yet) using _ssh-keygen_ on your workstation and and _ssh-copy-id_ to copy your public key file from ~/.ssh/id\_rsa.pub file to ~/.ssh/authorized\_keys on the server. If logging in still does not work please check the /var/log/auth.log file on the server. You may have some permissions wrong.
|
||||
</Aside>
|
||||
|
||||
Unless you just installed the server it is a good idea to install all missing updates first:
|
||||
|
||||
```
|
||||
apt update
|
||||
apt upgrade
|
||||
```
|
||||
|
||||
## Packages
|
||||
|
||||
Let us install the necessary Debian packages to make it an actual mail server. Take a moment to read through this list of package – we will do the installation afterwards:
|
||||
|
||||
- mariadb-server
|
||||
The database service that will store information about your email accounts and domains. (If you haven’t heard of MariaDB yet, it is a fork of MySQL after being acquired by Oracle.)
|
||||
- postfix
|
||||
The MTA (mail transport agent) that speaks SMTP to receive and send emails.
|
||||
- postfix-mysql
|
||||
An extension that allows Postfix to get its information from a MySQL/MariaDB database.
|
||||
- dovecot-mysql
|
||||
The IMAP/POP3 mail server including an extension to query information from a MySQL/MariaDB database.
|
||||
- dovecot-pop3d _(optional)_
|
||||
An extension to Dovecot that allows users to fetch emails using the POP3 protocol. (_This is optional. Only few users nowadays still use POP3._)
|
||||
- dovecot-imapd
|
||||
An extension to Dovecot that allows users to access emails using the IMAP protocol.
|
||||
- dovecot-lmtpd
|
||||
Enables Dovecot to receive [LMTP](https://en.wikipedia.org/wiki/Local_Mail_Transfer_Protocol) connections. We will need it later for the internal transfer of emails from Postfix to Dovecot.
|
||||
- dovecot-managesieved _(optional)_
|
||||
An extension to Dovecot that allows users to define filter rules that are automatically run on the server when a new email arrives.
|
||||
- apache2 and php8.2
|
||||
The web server that powers the webmail interface. PHP is the scripting language that the Roundcube webmail software is written in.
|
||||
- adminer _(optional)_
|
||||
A web interface to manage your SQL database if you are not comfortable crafting SQL queries by hand.
|
||||
- rspamd
|
||||
A third-party software that deals with spam and handles automatic domain key (DKIM) signing.
|
||||
- redis-server
|
||||
A key-value store. It is a simple but fast kind of database service where Rspamd stores training data about spam and ham.
|
||||
- pwgen
|
||||
A tool to create random passwords.
|
||||
- swaks
|
||||
The _SWiss Army Knife of Smtp_. A utility to send emails through SMTP for testing purposes.
|
||||
- mutt
|
||||
A console-based program that can speak IMAP and also read Maildirs directly. Very helpful for testing the functionality of your mail server.
|
||||
- certbot
|
||||
A tool that talks to the LetsEncrypt certificate service to request and renew certificates. You will not need it if you have bought a certificate for your webmail server.
|
||||
- ca-certificates
|
||||
A set of certificates from common certificate authorities on the internet. It is required for the proper function of _wget_ for example.
|
||||
- fail2ban _(optional)_
|
||||
A daemon that tracks log files to recognize brute force attacks. It can block IP addresses of attackers. We will use it to defend against evil people trying to get into our mail server.
|
||||
|
||||
## pwgen
|
||||
|
||||
Let’s start with the _pwgen_ utility. It helps you create secure passwords. Unless you already have a tool for that…
|
||||
|
||||
```
|
||||
apt install -y pwgen
|
||||
```
|
||||
You will need a random passwords later to create a database user for the mail server. Just as an example: to create a secure password having a length of 20 characters you would run:
|
||||
|
||||
```
|
||||
pwgen -s 20 1
|
||||
```
|
||||
|
||||
That gets you a string like “W2EzNUFJzjEmA8tQT7A0”.
|
||||
|
||||
## MariaDB server
|
||||
|
||||
If you used MySQL before you may remember that you were forced to specify a password for the ‘root’ _database user_. That has changed with MariaDB in Debian – for the better. Now you can access the database server without any password if you are logged in as ‘root’ on the server. You might as well set a password but it is not necessary.
|
||||
|
||||
Go install the MariaDB server package:
|
||||
|
||||
```
|
||||
apt install -y mariadb-server
|
||||
```
|
||||
If all went well you can now run “mysql” and get a connection to your MySQL database:
|
||||
|
||||
```
|
||||
root@buster:~# mysql
|
||||
Welcome to the MariaDB monitor. Commands end with ; or \\g.
|
||||
Your MariaDB connection id is 30
|
||||
Server version: 10.3.18-MariaDB-0+deb10u1 Debian 10
|
||||
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
|
||||
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
|
||||
MariaDB [(none)]>
|
||||
```
|
||||
|
||||
Exit the SQL shell by typing “exit” or pressing CTRL-D.
|
||||
|
||||
## Postfix
|
||||
|
||||
Now on to the Postfix packages:
|
||||
|
||||
```
|
||||
apt install -y postfix postfix-mysql
|
||||
```
|
||||
When you get asked for the mail server configuration type please choose “Internet site”. Enter your own mail server name (the fully qualified domain name) or just press enter. The host name and domain does not need to match any of your email domains.
|
||||
|
||||
## Apache and PHP
|
||||
|
||||
To provide a webmail service you need the Apache web server software and the PHP scripting language support:
|
||||
|
||||
```
|
||||
apt install -y apache2 php8.2
|
||||
```
|
||||
|
||||
<Aside type="note">
|
||||
I have been asked several times why I endorse Apache. Why not use [Nginx](https://www.nginx.com/) instead? Well, Apache is still pretty common. And if you use PHP – which we need for the web mail software – there is no performance advantage of using Nginx anyway. If you prefer Nginx then go with it – PHP-FPM is your friend.
|
||||
</Aside>
|
||||
|
||||
## rspamd
|
||||
|
||||
Just like in previous versions of this guide we will deal with spam using _[rspamd](https://rspamd.com/)_. (The rspamd principal developer endorses his [own software packages](https://rspamd.com/downloads.html) but in this guide we will use the packages in Debian for stability.)
|
||||
|
||||
{/* TODO: Using the upstream packages has advantages regarding spam detection. */}
|
||||
|
||||
We also install Redis as a storage backend for Rspamd to store its training data about spam and ham.
|
||||
|
||||
```
|
||||
apt install -y rspamd redis-server
|
||||
```
|
||||
|
||||
## swaks
|
||||
|
||||
A very useful tool to test email delivery later is [SWAKS](https://www.jetmore.org/john/code/swaks/) (the _SWiss Army Knife for Smtp_):
|
||||
|
||||
```
|
||||
apt install -y swaks
|
||||
```
|
||||
|
||||
## mutt
|
||||
|
||||
This is a full-featured IMAP mail client. Think of it as the _vi_ of mail clients. It cannot display HTML but it is very helpful to test IMAP mail servers. And some hardcore users still prefer it over any other mail client.
|
||||
|
||||
```
|
||||
apt install -y mutt
|
||||
```
|
||||
|
||||
## certbot
|
||||
|
||||
You will want to use encrypted connections where possible. So you need a certificate for your services. The _certbot_ software helps you to request and manage [Let’s Encrypt](https://letsencrypt.org/) certificates automatically.
|
||||
|
||||
```
|
||||
apt install -y certbot
|
||||
```
|
||||
|
||||
## Dovecot
|
||||
|
||||
In addition to Postfix (that handles SMTP communication) you will need [Dovecot](https://dovecot.org/) to store received emails and provide IMAP (and optionally POP3) access for your users:
|
||||
|
||||
```
|
||||
apt install -y dovecot-mysql dovecot-pop3d dovecot-imapd dovecot-managesieved dovecot-lmtpd
|
||||
```
|
||||
|
||||
## Adminer
|
||||
|
||||
SQL databases are called like that because SQL (_structured query language_) is the way you talk to it. But as we are just puny humans let’s have a more user-friendly way to manage the database. I suggest _Adminer_ which is a tool similar to phpMyAdmin.
|
||||
|
||||
```
|
||||
apt install -y adminer
|
||||
```
|
||||
|
||||
## ca-certificates
|
||||
|
||||
To avoid errors when downloading files using _wget_ you should install the default set of certificates of common certificate authorities on the internet:
|
||||
|
||||
```
|
||||
apt install -y ca-certificates
|
||||
```
|
||||
337
src/content/docs/ispmail-bookworm/80-tls-certificates.mdx
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
---
|
||||
title: TLS certificate
|
||||
lastUpdated: 2023-10-03
|
||||
slug: ispmail-bookworm/creating-a-tls-encryption-key-and-certificate
|
||||
sidebar:
|
||||
order: 80
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
|
||||
<Aside type="tip" title="TLS is SSL">
|
||||
If you are unfamiliar with the abbreviation “TLS“: it is the name successor to SSL but works one the same principle.
|
||||
SSL 3.0 was just renamed to TLS 1.0 in 1999. Every new version of TLS deprecates unsafe encryption methods (ciphers)
|
||||
of previous versions. You will still find people calling it _SSL_.
|
||||
</Aside>
|
||||
|
||||
The internet is not a friendly place where you can trust people. If you send data over the internet there is a pretty
|
||||
good chance someone intercepts it. You don’t want that. Our best weapon against that is transport encryption. All you
|
||||
need is to create an encryption _key_ and a _certificate_ for Postfix (SMTP), Dovecot (IMAP/POP3) and Apache (HTTPS). If
|
||||
you are confused why that requires a key and certificate then please consider reading the Wikipedia about
|
||||
[public-key cryptography](https://en.wikipedia.org/wiki/Public-key_cryptography). The single most important detail here:
|
||||
nobody but you must have access to the private key.
|
||||
|
||||
<Aside type="caution">
|
||||
Do not use different certificates for Postfix and Dovecot. At least the MacOS mail client would refuse your connection
|
||||
with confusing error messages. You will be safe If you follow this guide.
|
||||
</Aside>
|
||||
|
||||
## What makes mail clients trust a certificate?
|
||||
|
||||
Your client’s operating system contains a list of “trusted” certificate authorities. Sometimes that list is located in a
|
||||
central location for all applications. And sometimes a browser comes with its own list. A certificate signed by any of
|
||||
those organization is fully trusted. Yes, your browser trusts companies you probably have never heard of. And so does
|
||||
your mail client if you use software like Thunderbird instead of using web mail. The tricky part is not the technology.
|
||||
Everyone with some knowledge about certificates can create their own authority. Mathematically your certificates are
|
||||
just as good as anyone else’s. The tricky part is to convince browser manufacturers to trust them and add your
|
||||
certificates to their trust list.
|
||||
|
||||
Those organization claim to do checks (in exchange for money) whether you are the righteous owner of a certain domain.
|
||||
These checks have sometimes failed so badly that they popped up in the news. Ten years ago a nerd
|
||||
[requested](https://bugzilla.mozilla.org/show_bug.cgi?id=647959) to get trusted by Firefox with his “Honest Achmed’s
|
||||
Used Cars and Certificates” authority. It was fun to watch the developers struggle with arguments why Achmed would be
|
||||
any worse than authorities like
|
||||
[Türktrust](https://arstechnica.com/information-technology/2013/01/turkish-government-agency-spoofed-google-certificate-accidentally/)
|
||||
or [Verisign](https://en.wikipedia.org/wiki/Verisign#2001:_Code_signing_certificate_mistake) (who failed so hard one
|
||||
must wonder why they are still in business).
|
||||
|
||||
In 2012 [two Mozilla employees](https://en.wikipedia.org/wiki/Let%27s_Encrypt#History) decided to do something about
|
||||
that. They came up with the idea of a non-profit certificate authority. The world needed encryption everywhere and
|
||||
people should no longer shy away from it because getting certificates cost money. Two years later the service was ready
|
||||
and everyone could get a free certificate valid for 90 days. An automated process renewed the certificate in time so
|
||||
there was no hassle. (They surely deserve a [donation](https://letsencrypt.org/donate/) for the many years of service to
|
||||
the internet community.)
|
||||
|
||||
## Where is the certificate used?
|
||||
|
||||
You guessed it, we will need our own certificate. It will be used in three places:
|
||||
|
||||
1. the webmail interface (driven by the Apache web server)
|
||||
2. Postfix (to encrypt SMTP communication where possible)
|
||||
3. Dovecot (to encrypt IMAP/POP3 communication and deny any unencrypted traffic)
|
||||
|
||||
A single certificate (per domain) can and should be used by all three pieces of software.
|
||||
|
||||
## Choosing a hostname for your mail server
|
||||
|
||||
Your mail server will be able to serve emails in many domains. But a certificate is usually issued for a single
|
||||
[fully-qualified domain name](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) like “mail.example.org”. The
|
||||
_common name_ is an important part of the certificate. That attribute must correspond to the host name you use during
|
||||
communication or else the certificate is rejected. Hmmm, many domains but only one name is allowed on a certificate? How
|
||||
do we deal with that?
|
||||
|
||||
### Option 1: a generic name and a single certificate
|
||||
|
||||
The simple solution is to take a generic name. The German ISP Hetzner for example uses the name “mail.your-server.de”
|
||||
for all customers and their domains. And there are many examples of such an approach. Experience shows that most users
|
||||
do not care about the hostname as long as they get their emails. So I would generally recommend this approach due to its
|
||||
simplicity.
|
||||
|
||||
### Option 2: multiple names/certificates & SNI
|
||||
|
||||
Some people however want to give their mail server as many names as they have domains. Suppose that you have three
|
||||
domains: example.org, example.net and example.com. Users of example.org can use mail.example.org while users of
|
||||
example.net use mail.example.net. As you can imagine this would not work if you had a single certificate. After all
|
||||
which name would you use as a _common name_ there? If you used “mail.example.org” then talking to your server by the
|
||||
name “mail.example.net” would lead to a certificate error.
|
||||
|
||||
So your mail server needs multiple certificates – one for each host name. And depending on how a user connects to your
|
||||
server you need to send the matching certificate. If the user wants to speak to “mail.example.org” then your server
|
||||
needs to send the certificate for “mail.example.org”. Fortunately that problem has been addressed by a technique called
|
||||
[Server Name Indication (SNI)](https://en.wikipedia.org/wiki/Server_Name_Indication). A client talks to the server and
|
||||
even before the encrypted connection is established it tells the server that it expects a certificate for the intended
|
||||
host name. The server can then load the matching certificate and initiate the encryption process.
|
||||
|
||||
SNI has long been a problem for mail servers. The mail user agent (e.g. Thunderbird) needs to support it as well as
|
||||
Postfix and Dovecot. Postfix has finally added SNI in version 3.4 so that we can use it.
|
||||
|
||||
While adding multiple host names needs extra work, it also has a benefit. If you want to move domains to other servers
|
||||
(e.g. when upgrading your server) you can move one domain at a time.
|
||||
|
||||
<Aside type="tip" title="More security?">
|
||||
If you wish to further enhance security against [man-in-the-middle
|
||||
attacks](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) you should get acquainted
|
||||
with [DNSSEC](https://en.wikipedia.org/wiki/Domain_Name_System_Security_Extensions) and [DANE](https://en.wikipedia.org/wiki/DNS-based_Authentication_of_Named_Entities).
|
||||
Also take a look at
|
||||
the [smtp\_dns\_support_level](http://www.postfix.org/postconf.5.html#smtp_dns_support_level) parameter in Postfix.
|
||||
</Aside>
|
||||
|
||||
## Preparing the Apache web server for HTTP
|
||||
|
||||
Let’s start with the web server. As an example I will assume that you want to offer a host name **webmail.example.org**
|
||||
to your users. Of course your server will have another name in a domain that you control. I will use that example
|
||||
throughout the tutorial though and keep that name printed in **bold** letters to remind you that you have to use your
|
||||
own host name.
|
||||
|
||||
Do you just want to play around with your new server and not use any real domain yet? No problem. Then use
|
||||
[nip.io](https://nip.io/). If your IP address is 1.2.3.4 then you can use 1.2.3.4.nip.io as a domain name that points to
|
||||
your server. (There used to be another domain xip.io for that purpose but it died in mid-2021.) Another service like
|
||||
that which also supports IPv6 is [sslip.io](https://sslip.io/).
|
||||
|
||||
If you have an actual domain then set up a DNS “A” and “AAAA” (if you use IPv6) record for that host name pointing to
|
||||
your server’s IP address.
|
||||
|
||||
First you need a web root directory for that host name:
|
||||
|
||||
```
|
||||
mkdir /var/www/webmail.example.org
|
||||
chown www-data:www-data /var/www/webmail.example.org
|
||||
```
|
||||
|
||||
Next you need to create a virtual host configuration file. Apache on Debian uses a neat system to manage virtual hosts:
|
||||
|
||||
- `/etc/apache2/sites-available/*.conf` contains the actual configuration files for each virtual host. Putting a file
|
||||
here does not enable that host though. That’s done in the next step. There are two configuration files by default.
|
||||
“000-default.conf” is a HTTP virtual host and “default-ssl.conf” is a HTTPS virtual host.
|
||||
- `/etc/apache2/sites-enabled/*.conf` contains symbolic links (“symlinks”) pointing to configuration files in the
|
||||
/etc/apache2/sites-available directory. Only \*.conf links in this directory will be loaded by Apache.
|
||||
|
||||
This technique allows you to enable and disable virtual hosts without having to destroy any configuration. Debian ships
|
||||
with the “a2ensite” (short for “apache2 enable site”) and “a2dissite” commands. In addition to some sanity checks those
|
||||
commands essentially create or remove symlinks between “sites-available” and “sites-enabled”.
|
||||
|
||||
<Aside type="tip">Apache configuration files must have a “.conf” suffix or else they will get ignored</Aside>
|
||||
|
||||
You may remove the default symlinks in `/etc/apache2/sites-enabled/*` unless you use them already.
|
||||
|
||||
Create a new virtual host configuration file /etc/apache2/sites-available/**webmail.example.org**-http.conf and make it
|
||||
contain:
|
||||
|
||||
<pre>
|
||||
<VirtualHost *:80> ServerName **webmail.example.org** DocumentRoot /var/www/**webmail.example.org**
|
||||
</VirtualHost>
|
||||
</pre>
|
||||
|
||||
The simple configuration makes Apache handle HTTP requests (on the standard TCP port 80) if a certain line in the
|
||||
request header from the browser reads “Host: webmail.example.org”. So the browser actually tells your Apache web server
|
||||
which server name it is looking for. That allows for multiple web sites on a single IP address. (Thanks to
|
||||
[Server Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication) as explained earlier this works well for
|
||||
HTTPS, too.)
|
||||
|
||||
Enable the site:
|
||||
|
||||
<pre>a2ensite **webmail.example.org**-http</pre>
|
||||
|
||||
You will be told:
|
||||
|
||||
```
|
||||
To activate the new configuration, you need to run:
|
||||
systemctl reload apache2
|
||||
```
|
||||
|
||||
Do that.
|
||||
|
||||
<Aside type="tip" title="Apache shows an error?">
|
||||
If after reloading/restarting the Apache web server you just get an error message like…
|
||||
|
||||
```
|
||||
Job for apache2.service failed.
|
||||
See “systemctl status apache2.service” and “journalctl -xeu apache2.service” for details.
|
||||
```
|
||||
|
||||
…then run “apache2ctl configtest” to find out why it is unhappy about your configuration."
|
||||
|
||||
</Aside>
|
||||
|
||||
Let’s check if the configuration works. Put a test file into your web root directory:
|
||||
|
||||
<pre>echo "Just a test" > /var/www/**webmail.example.org**/test</pre>
|
||||
|
||||
Now when you open the URL http://**webmail.example.org**/test in your browser you should see the text “Just a test”.
|
||||
|
||||
This is enough setup to make LetsEncrypt issue a certificate for you.
|
||||
|
||||
## Getting a LetsEncrypt certificate
|
||||
|
||||
Now you can use the _certbot_ tool to request an encryption certificate from LetsEncrypt. What will happen?
|
||||
|
||||
- certbot creates a _private key_ and a _certificate request_. It sends the _certificate request_ to the LetsEncrypt
|
||||
server.
|
||||
- the LetsEncrypt server replies with a _challenge_/_token_.
|
||||
- certbot puts that token into a file in the /var/www/**webmail.example.org**/.well-known/acme-challenge directory.
|
||||
- the LetsEncrypt server does an HTTP connection to `http://webmail.example.org/.well-known/acme-challenge/…` and
|
||||
expects to find that token. This verifies that you are in charge of the domain and the web server.
|
||||
- If all works well the LetsEncrypt server signs your _certificate request_ and thus creates the actual _certificate_.
|
||||
- certbot receives the certificate and puts it into /etc/letsencrypt/archive/**webmail.example.org**/
|
||||
|
||||
To get a certificate for your domain run:
|
||||
|
||||
<pre class="wrap">
|
||||
certbot certonly --webroot --webroot-path /var/www/**webmail.example.org** -d **webmail.example.org**
|
||||
</pre>
|
||||
|
||||
You can use multiple occurences of “-d” here to get a certificate valid for multiple domains. For example: “-d
|
||||
webmail.example.org -d something-else.example.org”. (See also:
|
||||
[https://eff-certbot.readthedocs.io/en/stable/using.html#webroot](https://eff-certbot.readthedocs.io/en/stable/using.html#webroot))
|
||||
|
||||
The first time you do that you will get asked for your email address so LetsEncrypt can send you reminders if your
|
||||
certificate would expire. You will also have to agree to their terms of service.
|
||||
|
||||
If everything worked well you should get output like:
|
||||
|
||||
```
|
||||
Requesting a certificate for webmail.example.org Successfully received certificate. Certificate is saved at:
|
||||
/etc/letsencrypt/live/webmail.example.org/fullchain.pem Key is saved at:
|
||||
/etc/letsencrypt/live/webmail.example.org/privkey.pem This certificate expires on 2024-01-02. These files will be
|
||||
updated when the certificate renews. Certbot has set up a scheduled task to automatically renew this certificate in
|
||||
the background.
|
||||
```
|
||||
|
||||
In /etc/letsencrypt/live/**webmail.example.org** you will find a couple of files now:
|
||||
|
||||
- cert.pem: the certificate file
|
||||
- chain.pem: the _chaining_ or _intermediate_ certificate. This certificate provides information how the LetsEncrypt
|
||||
certificates are linked to other known certificate authorities. It is generally a good idea to always send this
|
||||
certificate along with your own for clients who may not know LetsEncrypt properly yet.
|
||||
- fullchain.pem: this file contains a concatenation of the _cert.pem_ and the _chain.pem_. This is the preferred file to
|
||||
use when a piece of software asks where to find the _certificate_.
|
||||
- privkey.pem: the private key file. Keep it secret.
|
||||
|
||||
<Aside type="tip" title="Expires in 3 months?">
|
||||
LetsEncrypt issues certificates that are valid for 3 months. This is by design to get rid of old certificates quickly.
|
||||
The renewal feature of the certbot will automatically get a fresh certificate after 2 months. So you don’t have to
|
||||
worry about that. And if a renewal failed you will get an email from them reminding you to take a look.
|
||||
</Aside>
|
||||
|
||||
## Add HTTPS
|
||||
|
||||
Now that you have a valid certificate you can finally enable HTTPS for your web server. Create a new file
|
||||
/etc/apache2/sites-available/**webmail.example.org**-https.conf containing:
|
||||
|
||||
<pre>
|
||||
<VirtualHost *:443> ServerName **webmail.example.org** DocumentRoot /var/www/**webmail.example.org** SSLEngine
|
||||
on SSLCertificateFile /etc/letsencrypt/live/**webmail.example.org**/fullchain.pem SSLCertificateKeyFile
|
||||
/etc/letsencrypt/live/**webmail.example.org**/privkey.pem </VirtualHost>
|
||||
</pre>
|
||||
|
||||
This virtual host configuration looks suspiciously similar to the HTTP virtual host above. It just listens on port 443
|
||||
(standard port for HTTPS) instead of port 80. And it uses the “SSLEngine” that handles encryption and gets information
|
||||
about the certificate for your web server (that is shown to your users) and the private key (that the web servers uses
|
||||
to decrypt the user’s communication).
|
||||
|
||||
Enable the SSL module in Apache:
|
||||
|
||||
<pre>a2enmod ssl</pre>
|
||||
|
||||
Then enable the virtual host for HTTPS:
|
||||
|
||||
<pre>a2ensite **webmail.example.org**-https</pre>
|
||||
|
||||
And _restart_ the web server. A _reload_ is not sufficient this time because you added a module.
|
||||
|
||||
systemctl restart apache2
|
||||
|
||||
Now when you point your web browser to **webmail.example.org**, your browser should tell you that it trusts the web
|
||||
site’s certificate:
|
||||
|
||||

|
||||
|
||||
(Yes, sorry, this is not **webmail.example.org**. But I do not own the example.org domain and thus cannot get a valid
|
||||
certificate for it. This is my own site.)
|
||||
|
||||
So should you keep the HTTP virtual host? Yes. First for the HTTP->HTTPS redirection. And second to keep _certbot_
|
||||
working.
|
||||
|
||||
## Redirect HTTP to HTTPS
|
||||
|
||||
Sometimes users forget to enter https://… when accessing your webmail service. So they access the HTTP web site. We
|
||||
obviously don’t want them to send their password over HTTP. So we should redirect all HTTP connections to HTTPS.
|
||||
|
||||
One exception though. Let’s Encrypt will use HTTP to verify your challenge token. So we need to serve files at
|
||||
http://**webmail.example.org**/.well-known/acme-challenge/… directly while redirecting all other requests to HTTPS. You
|
||||
can accomplish that by putting these lines inside the <VirtualHost> section of your
|
||||
`/etc/apache2/sites-available/webmail.example.org-http.conf` file:
|
||||
|
||||
```
|
||||
RewriteEngine On
|
||||
RewriteCond %{REQUEST_URI} !.well-known/acme-challenge
|
||||
RewriteRule ^(.*)$ https://%{SERVER_NAME}$1 \[R=301,L\]
|
||||
```
|
||||
|
||||
This requires the _rewrite_ module to be enabled in Apache. That is simple though:
|
||||
|
||||
```
|
||||
a2enmod rewrite
|
||||
systemctl restart apache2
|
||||
```
|
||||
|
||||
So now entering http://**webmail.example.org** will redirect you to https://**webmail.example.org**.
|
||||
|
||||
## Automatic certificate renewal
|
||||
|
||||
The *certbot* package automatically adds a timed job that runs twice a day at random times. The random part is important
|
||||
to avoid millions of server hammering the LetsEncrypt service at the same second.
|
||||
|
||||
<Aside type="tip" title="Systemd timer instead of a Cron job">
|
||||
This job is not a classic Cron job but instead latches into systemd. You can find the _timer_ definition in
|
||||
the `/lib/systemd/system/certbot.timer` file. That timer triggers the renewal service defined
|
||||
in `/lib/systemd/system/certbot.service`. To find out if the service has run and when the next occurence will be, run
|
||||
“systemctl status certbot.timer”. There is also a Cron file at /etc/cron.d/certbot. Don’t be confused. This job will
|
||||
not do anything due to the “`-d /run/systemd/system`” condition that checks if systemd is installed. And Debian
|
||||
nowadays uses systemd.
|
||||
</Aside>
|
||||
|
||||
So the renewal already happens automatically. Should it fail then LetsEncrypt start sending you reminder emails that
|
||||
your certificate should be renewed. That’s a clear sign that something went wrong with the automatic renewal.
|
||||
|
||||
There is one puzzle piece missing though. Even if the renewal worked it will only update the certificate files. But the
|
||||
software components – Postfix, Dovecot and Apache – will not notice the change. So we need to add a so called
|
||||
_post-hook_ to certbot that triggers a restart of all processes thereafter.
|
||||
|
||||
For that purpose edit the /etc/letsencrypt/cli.ini file and add:
|
||||
|
||||
```
|
||||
post-hook = systemctl restart postfix dovecot apache2
|
||||
```
|
||||
|
||||
Well done. You have implemented Let’s Encrypt for all your services now. Let’s go on.
|
||||
219
src/content/docs/ispmail-bookworm/90-prepare-database.mdx
Normal file
|
|
@ -0,0 +1,219 @@
|
|||
---
|
||||
title: Preparing the database
|
||||
lastUpdated: 2023-10-03
|
||||
slug: ispmail-bookworm/prepare-the-database
|
||||
sidebar:
|
||||
order: 90
|
||||
---
|
||||
|
||||
import { Aside } from "@astrojs/starlight/components";
|
||||
import { Tabs, TabItem } from '@astrojs/starlight/components';
|
||||
|
||||
Now it’s time to prepare the MariaDB database that stores the information that controls your mail server. In the process you will have to enter [SQL](http://en.wikipedia.org/wiki/SQL) queries – the language of relational database servers. You may enter them in a terminal window using the ‘mysql’ command. But if you are less experienced with SQL you may prefer using a web interface. That’s what you installed _[Adminer](https://www.adminer.org/)_ for.
|
||||
|
||||
<Aside type="tip" title="Why not PostgreSQL?">
|
||||
Several comments show that some system administrators prefer the PostgreSQL database engine over MariaDB. Count me in. So why is this guide still using MariaDB/MySQL? Mainly for historical reasons. I want to spare you any unneccessary changes when migrating to a new version. If you are familiar with PostgreSQL however you can easily adapt this guide to PostgreSQL. There are dovecot-pgsql and postfix-pgsql packages. And Adminer works with PostgreSQL as well.
|
||||
|
||||
It may be worth using SQLite instead. Because on small _family-and-friends_ servers that would be totally sufficient. Let us see if it will be added to this guide at a later time.
|
||||
</Aside>
|
||||
|
||||
|
||||
## Setting up Adminer
|
||||
|
||||
Basically Adminer is just a couple of PHP files served from your Apache web server. The setup is simple. Edit your /etc/apache2/sites-available/**webmail.example.org**-https.conf file and put this line anywhere between the <VirtualHost> and the </VirtualHost> tags:
|
||||
```
|
||||
Alias /adminer /usr/share/adminer/adminer
|
||||
```
|
||||
Reload the Apache process:
|
||||
```
|
||||
systemctl reload apache2
|
||||
```
|
||||
|
||||
<Aside type="danger" title="Security warning">
|
||||
Having an SQL admin interface publicly available on your web site is an invitation for internet scoundrels to do bad things. Consider protecting the /adminer location by an additional password. The Apache documentation shows you how to do that. Or use a less obvious path than “/adminer” in the Alias.
|
||||
</Aside>
|
||||
|
||||
You will not be able to login yet. The only available database user is ‘root’, but it is only usable from the shell by default – not over a network.
|
||||
|
||||
|
||||
## Generate two random passwords
|
||||
|
||||
In this section you will create the basic database “mailserver” and two users. One user (“mailadmin”) will be able to change the data in the database and is meant for you. The other user (“mailserver”) can only read from the database and is meant for the server processes.
|
||||
|
||||
Use the _pwgen_ tool to create two random passwords for these users:
|
||||
```
|
||||
pwgen -s1 30 2
|
||||
```
|
||||
Take a note of the passwords or store them somewhere safe.
|
||||
|
||||
|
||||
## Create the ‘mailserver’ database
|
||||
|
||||
This step is simple. Connect to the database using the ‘mysql’ command:
|
||||
```
|
||||
mysql
|
||||
```
|
||||
You should see the MariaDB prompt that allows you to enter further SQL commands:
|
||||
```
|
||||
MariaDB [(none)]>
|
||||
```
|
||||
|
||||
Now you are expected to speak SQL. To create a new database for our needs enter:
|
||||
|
||||
```
|
||||
CREATE DATABASE mailserver;
|
||||
```
|
||||
You will be told that your query was OK and that one new row was added.
|
||||
|
||||
|
||||
## Create the database users
|
||||
|
||||
Now you have an empty database. Let us give the “mailadmin” database user the required privileges to manage it.
|
||||
|
||||
You are still connected to the database, right? To create a user with full permissions enter this SQL command. Please use the **first** password you just generated instead of mine:
|
||||
```
|
||||
grant all privileges on mailserver.* to 'mailadmin'@'localhost' identified by 'gefk6lA2brMOeb8eR5WYaMEdKDQfnF';
|
||||
```
|
||||
|
||||
Also create the read-only user that will grant Postfix and Dovecot database access later (use your **second** random password here).
|
||||
|
||||
```
|
||||
grant select on mailserver.* to 'mailserver'@'127.0.0.1' identified by 'x893dNj4stkHy1MKQq0USWBaX4ZZdq';
|
||||
```
|
||||
|
||||
<Aside type="note" title="127.0.0.1 versus localhost">
|
||||
Wait a minute. Why is there “127.0.0.1” instead of “localhost” in the second SQL command? Is that a typo? No, it’s not. Well, in network terminology those two are identical. But MariaDB (and Oracle’s MySQL) distinguishes between the two. If you initiate a database connection to “localhost” then you talk to the socket file which lives at /var/run/mysqld/mysqld.sock on your server. But if you connect to “127.0.0.1” it will create a network connection talking to the TCP socket on port 3306 on your server. The difference is that any process on your server can talk to 127.0.0.1. But the socket file has certain user/group/other permissions just like any other file on your file system. Postfix will be restricted to its /var/spool/postfix directory and cannot by default access that socket file. So by using 127.0.0.1 we circumvent that limitation.
|
||||
|
||||
When you use _Adminer_ you will have to use ‘localhost’ as a database server when using the ‘mailadmin’ user but ‘127.0.0.1’ when using the ‘mailserver’ user.
|
||||
</Aside>
|
||||
|
||||
Now you can use _Adminer_ to log in using the _mailadmin_ account and the **first** password:
|
||||
|
||||

|
||||
|
||||
You should get logged in and see the “mailserver” database:
|
||||
|
||||

|
||||
|
||||
## Creating the database tables
|
||||
|
||||
Do you remember that I introduced three Postfix _mappings_ earlier? One for _virtual domains_, one for _virtual aliases_ and another for _virtual users_? Each of the mappings needs a database table that you will create now. Feel free to use _Adminer_. I will however also show the SQL statement to create the tables that you can enter on the ‘mysql’ command-line tool. Below you can click on either \[Adminer\] or \[SQL\] to choose.
|
||||
|
||||
The first table to create is…
|
||||
|
||||
## virtual\_domains
|
||||
|
||||
This table just holds the list of domains that you will use as _virtual\_mailbox\_domains_ in Postfix.
|
||||
|
||||
<Tabs syncKey="sql">
|
||||
<TabItem label="Adminer">
|
||||
| **Column** | **Purpose** |
|
||||
| ---------- | -------------------------------------------------------------------------------- |
|
||||
| id | A unique number identifying each row. It is added by the database automatically. |
|
||||
| name | The name of the domain you want to receive email for. |
|
||||
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
USE mailserver;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `virtual_domains` (
|
||||
`id` int(11) NOT NULL auto_increment,
|
||||
`name` varchar(50) NOT NULL,
|
||||
PRIMARY KEY (`id`)
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
### virtual\_users
|
||||
|
||||
The next table contains information about your users. Each mail account takes up one row.
|
||||
|
||||
<Tabs syncKey="sql">
|
||||
<TabItem label="Adminer">
|
||||
| **Column** | **Purpose**|
|
||||
| ---------- | --- |
|
||||
| domain\_id | Contains the number of the domain’s id in the _virtual\_domains_ table. This is called a _[foreign key](https://en.wikipedia.org/wiki/Foreign_key)_. A “delete cascade” makes sure that if a domain is deleted that all user accounts in that domain are also deleted to avoid orphaned rows. |
|
||||
| email | The email address of the mail account.|
|
||||
| password | The hashed password of the mail account. It is prepended by the [password scheme](https://doc.dovecot.org/configuration_manual/authentication/password_schemes/). By [default](https://doc.dovecot.org/configuration_manual/authentication/password_schemes/) it is `{BLF-CRYPT}` also known as _bcrypt_ which is considered very secure. Previous ISPmail guides used `{SHA256-CRYPT}` or even older crypt schemes. Prepending the password field the hashing algorithm in curly brackets allows you to have different kinds of hashes. So you can easily migrate your old passwords without locking out users. Users with older schemes should get a new password if possible to increase security. |
|
||||
| quota | The number of bytes that this mailbox can store. You can use this value to limit how much space a mailbox can take up. The default value is 0 which means that there is no limit. |
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS `virtual_users` (
|
||||
`id` int(11) NOT NULL auto_increment,
|
||||
`domain_id` int(11) NOT NULL,
|
||||
`email` varchar(100) NOT NULL,
|
||||
`password` varchar(150) NOT NULL,
|
||||
`quota` bigint(11) NOT NULL DEFAULT 0,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `email` (`email`),
|
||||
FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
### virtual\_aliases
|
||||
|
||||
The last table contains forwardings from an email address to other email addresses.
|
||||
|
||||
<Tabs syncKey="sql">
|
||||
<TabItem label="Adminer">
|
||||
| **Field** | **Purpose** |
|
||||
| --- | --- |
|
||||
| id | A unique number identifying each row. It is added by the database automatically. |
|
||||
| domain\_id | Contains the number of the domain’s id in the virtual\_domains table again. |
|
||||
| source | The email address that the email was actually sent to. In case of catch-all addresses (that accept any address in a domain) the source looks like “@example.org”. |
|
||||
| destination | The email address that the email should instead be sent to. |
|
||||
</TabItem>
|
||||
<TabItem label="SQL">
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS `virtual_aliases` (
|
||||
`id` int(11) NOT NULL auto_increment,
|
||||
`domain_id` int(11) NOT NULL,
|
||||
`source` varchar(100) NOT NULL,
|
||||
`destination` varchar(100) NOT NULL,
|
||||
PRIMARY KEY (`id`),
|
||||
FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
|
||||
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
|
||||
```
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
As described in the section about domain types there can be multiple targets for one source email address. You just would need to insert several rows with the same source address and different destination addresses that will get copies of an email. Postfix will consider all matching rows.
|
||||
|
||||
|
||||
## Example data to play with
|
||||
|
||||
Too much theory so far? I can imagine. Let’s populate the database with an `example.org` domain, a `john@example.org` email account and a forwarding of `jack@example.org` to `john@example.org`. We will use that information in the next chapter to play with.
|
||||
|
||||
To add that sample data just run these SQL queries:
|
||||
|
||||
```sql
|
||||
REPLACE INTO mailserver.virtual\_domains (id,name) VALUES ('1','example.org');
|
||||
|
||||
REPLACE INTO mailserver.virtual\_users (id,domain\_id,password,email)
|
||||
VALUES ('1', '1',
|
||||
'{BLF-CRYPT}$2y$05$.WedBCNZiwxY1CG3aleIleu6lYjup2CIg0BP4M4YCZsO204Czz07W',
|
||||
'john@example.org');
|
||||
|
||||
REPLACE INTO mailserver.virtual\_aliases (id,domain\_id,source,destination)
|
||||
VALUES ('1', '1', 'jack@example.org', 'john@example.org');
|
||||
```
|
||||
|
||||
Do you wonder how I got the long cryptic password? I ran…
|
||||
|
||||
```
|
||||
doveadm pw -s BLF-CRYPT
|
||||
```
|
||||
|
||||
…to create a secure hash of the simple password “summersun”. Once you have installed Dovecot you can try that yourself but you will get a different output. The reason is that the passwords are [salted](https://en.wikipedia.org/wiki/Salt_\(cryptography\)) to increase their security.
|
||||
|
||||
Remember to remove that sample data before you go live with your mail server. Thanks to the _delete cascade_ you just need to remove the virtual\_domain. The alias and the mailbox will be deleted automatically. This would be the SQL query you should run before taking your mail server into production:
|
||||
|
||||
```sql
|
||||
DELETE FROM mailserver.virtual_domains WHERE name='example.org';
|
||||
```
|
||||
|
After Width: | Height: | Size: 80 KiB |
|
After Width: | Height: | Size: 37 KiB |
|
After Width: | Height: | Size: 43 KiB |
|
After Width: | Height: | Size: 47 KiB |
|
After Width: | Height: | Size: 17 KiB |
BIN
src/content/docs/ispmail-bookworm/images/managing-ima.jpg
Normal file
|
After Width: | Height: | Size: 282 KiB |
|
After Width: | Height: | Size: 13 KiB |
|
After Width: | Height: | Size: 38 KiB |
|
After Width: | Height: | Size: 16 KiB |
|
After Width: | Height: | Size: 16 KiB |
|
After Width: | Height: | Size: 19 KiB |
|
After Width: | Height: | Size: 22 KiB |
|
After Width: | Height: | Size: 11 KiB |
|
After Width: | Height: | Size: 16 KiB |
|
After Width: | Height: | Size: 15 KiB |
|
After Width: | Height: | Size: 55 KiB |
|
After Width: | Height: | Size: 21 KiB |
|
After Width: | Height: | Size: 69 KiB |
|
After Width: | Height: | Size: 14 KiB |
|
After Width: | Height: | Size: 68 KiB |
BIN
src/content/docs/ispmail-bookworm/images/tls-browser-bar.png
Normal file
|
After Width: | Height: | Size: 17 KiB |
|
After Width: | Height: | Size: 41 KiB |
|
After Width: | Height: | Size: 19 KiB |
|
After Width: | Height: | Size: 51 KiB |