Add failure logging/email notification in backup.sh; harden /opt/backup perms

This commit is contained in:
Christoph Haas 2026-09-19 22:20:31 +00:00
parent d06a9eac7d
commit 68aae4715d
5 changed files with 57 additions and 5 deletions

3
.gitignore vendored
View file

@ -1,3 +1,6 @@
# Secrets — never commit # Secrets — never commit
backup.env backup.env
encryption-key.pem encryption-key.pem
# Runtime artifacts
backup.log

View file

@ -11,13 +11,14 @@ Everything lives in `/opt/backup/`. The connection details and credentials
| File | Purpose | | File | Purpose |
|------|---------| |------|---------|
| `backup.env` | Shared config: server, datastore, namespace, token, key (sourced by both scripts) | | `backup.env` | Shared config: server, datastore, namespace, token, key, `NOTIFY_EMAIL` (sourced by both scripts) |
| `backup.env.sample` | Template — copy to `backup.env` and fill in real values | | `backup.env.sample` | Template — copy to `backup.env` and fill in real values |
| `setup.sh` | Install the client and create the encryption key (run once as root) | | `setup.sh` | Install the client and create the encryption key (run once as root) |
| `backup.sh` | Full backup of `/` with client-side encryption | | `backup.sh` | Full backup of `/` with client-side encryption |
| `restore.sh` | FUSE-mount the encrypted backup to browse/restore files | | `restore.sh` | FUSE-mount the encrypted backup to browse/restore files |
| `/opt/backup/encryption-key.pem` | Client-side encryption key (store a copy off-host!) | | `/opt/backup/encryption-key.pem` | Client-side encryption key (store a copy off-host!) |
| `/opt/backup/exclude` | Paths excluded from the backup | | `/opt/backup/exclude` | Paths excluded from the backup |
| `backup.log` | Runtime log written by `backup.sh` (created on each run) |
## Setup ## Setup
@ -53,6 +54,31 @@ chmod 600 backup.env
4. The client encrypts everything locally before uploading. The PBS only ever 4. The client encrypts everything locally before uploading. The PBS only ever
stores ciphertext and cannot read the data. stores ciphertext and cannot read the data.
The client's output is piped through `tee` into `backup.log` (the file in the
table above). The script captures the client's exit status (via
`set -o pipefail`), so a failed `proxmox-backup-client` run is detected even
though its output went through `tee`.
## Failure notification
If `backup.sh` fails it exits non-zero and, when a mailbox is configured, emails
the failure. Set it in `backup.env`:
```
NOTIFY_EMAIL="admin@example.com"
```
Empty (the default) disables mail. There is no `mail`/`mailx` on the hosts, so
the script sends via Postfix directly with:
```
... | sendmail -t
```
The mail body contains the timestamp and the contents of `backup.log`. This only
covers `backup.sh` failures — a *missed* run (host down, cron removed) is not
detected, so monitor the PBS side too.
Encryption is client-side. The key file has no password (`kdf: none`), so Encryption is client-side. The key file has no password (`kdf: none`), so
**possession of the key file alone is enough to decrypt** — protect it. **possession of the key file alone is enough to decrypt** — protect it.

View file

@ -27,3 +27,6 @@ ENCRYPTION_KEY_FILE="/opt/backup/encryption-key.pem"
# --- Local hostname (used as backup-id and archive-name prefix) --- # --- Local hostname (used as backup-id and archive-name prefix) ---
HOSTNAME="$(hostname)" HOSTNAME="$(hostname)"
# --- Failure notification (email sent on backup failure; empty = disabled) ---
NOTIFY_EMAIL="admin@example.com"

View file

@ -10,6 +10,7 @@ source "$SCRIPT_DIR/backup.env"
# ============================================================================== # ==============================================================================
BACKUP_LABEL="${HOSTNAME}-root" BACKUP_LABEL="${HOSTNAME}-root"
EXCLUDE_FILE="/opt/backup/exclude" EXCLUDE_FILE="/opt/backup/exclude"
LOG_FILE="${SCRIPT_DIR}/backup.log"
# ============================================================================== # ==============================================================================
# EXECUTION # EXECUTION
@ -27,9 +28,24 @@ while IFS= read -r line; do
EXCLUDE_ARGS+=(--exclude "$line") EXCLUDE_ARGS+=(--exclude "$line")
done < "$EXCLUDE_FILE" done < "$EXCLUDE_FILE"
proxmox-backup-client backup "${BACKUP_LABEL}.pxar:/" \ if proxmox-backup-client backup "${BACKUP_LABEL}.pxar:/" \
--keyfile "$ENCRYPTION_KEY_FILE" \ --keyfile "$ENCRYPTION_KEY_FILE" \
--ns "$BACKUP_NS" \ --ns "$BACKUP_NS" \
"${EXCLUDE_ARGS[@]}" "${EXCLUDE_ARGS[@]}" 2>&1 | tee "$LOG_FILE"; then
echo "Backup completed successfully!"
echo "Backup completed successfully!" else
echo "Backup FAILED — see $LOG_FILE" >&2
if [ -n "${NOTIFY_EMAIL:-}" ]; then
{
echo "To: $NOTIFY_EMAIL"
echo "From: root@${HOSTNAME}"
echo "Subject: [pbs-backup] FAILED on ${HOSTNAME}"
echo
echo "The Proxmox backup on ${HOSTNAME} failed at $(date -R)."
echo
echo "--- backup log ---"
cat "$LOG_FILE"
} | sendmail -t
fi
exit 1
fi

View file

@ -107,6 +107,10 @@ EOF
systemctl daemon-reload systemctl daemon-reload
systemctl enable --now pbs-backup.timer systemctl enable --now pbs-backup.timer
# 8. Hardening: deny read/write/execute to "others" for everything in /opt/backup
# (including the directory itself), so config and backups are not world-readable.
chmod -R o-rwx "$BACKUP_DIR"
echo echo
echo "Done. Fill in the credentials, then run ${BACKUP_DIR}/backup.sh (or wait for the timer):" echo "Done. Fill in the credentials, then run ${BACKUP_DIR}/backup.sh (or wait for the timer):"
echo " ${EDITOR:-vi} ${ENV_FILE} # server/datastore/namespace/token" echo " ${EDITOR:-vi} ${ENV_FILE} # server/datastore/namespace/token"