Self-Hosting Email

Self-hosting email is the canonical free-software project that used to be easy and is now actively hard. The mail server itself — Postfix, Dovecot, a spam filter, a database for users — has not gotten more difficult. The internet around it has. By 2017, an email server that does not authenticate itself thoroughly through SPF, DKIM, and DMARC, and does not run from an IP with a clean reputation, will silently disappear into the spam folders of the only providers that matter.

This post is about what that fight actually looks like in 2017, what stack to deploy, and whether the project is still worth the effort.

Why the difficulty is not in the mail server

Postfix has been a stable, well-documented MTA for fifteen years. Dovecot is the IMAP server that approximately everyone runs. SpamAssassin or rspamd handles inbound spam filtering. OpenDKIM signs your outbound mail. None of these are hard to install on a Debian 8 Jessie box. The configurations are documented in dozens of places and the defaults are reasonable.

The difficulty is everywhere else. It is in your DNS records, your IP reputation, your reverse DNS, your TLS certificates, your IPv6 hygiene, and most of all in the increasingly elaborate dance of proving to Gmail and Outlook.com that your domain is who it claims to be and is not a spam factory.

If you skip any of that, your mail does not bounce. It is silently filed as spam, and you do not find out until someone tells you they never got the message. This is the failure mode that makes self-hosted email frustrating: not catastrophic failures you can fix, but slow attrition you cannot easily detect.

The minimum viable stack

For a single domain, single-user-or-family setup, the stack on Debian 8 Jessie is:

• Postfix as the SMTP server (inbound and outbound).
• Dovecot as the IMAP/POP3 server, with LMTP delivery from Postfix.
• OpenDKIM for DKIM signing of outbound mail.
• SpamAssassin or rspamd for inbound spam scoring.
• Fail2ban to throttle the brute-force attempts that arrive within hours of opening port 25.
• Let's Encrypt for TLS on SMTP (port 587 submission, port 465 SMTPS, and port 993 IMAPS).
  Approximate install:

1apt-get install postfix postfix-pcre dovecot-imapd dovecot-lmtpd \
2                opendkim opendkim-tools spamassassin spamc fail2ban

Postfix's interactive configurator handles the basics. The pieces that matter and that the configurator does not set up for you are the four DNS records below, the OpenDKIM signing pipeline, and the TLS configuration.

The four DNS records that decide your deliverability

Without these, no amount of clean configuration will save you.

MX record. Points your domain to your mail server's hostname. Standard, almost everyone gets this right.

1example.com.    IN MX    10 mail.example.com.
2mail.example.com. IN A   203.0.113.10

SPF (Sender Policy Framework). A TXT record on your domain that lists which IPs are authorized to send mail for it. Strict policies (-all) tell receivers to reject mail from any other IP. Soft fail (~all) says "treat suspiciously." Use -all once you are confident your sending infrastructure is enumerated.

1example.com.    IN TXT   "v=spf1 mx -all"

DKIM (DomainKeys Identified Mail). A public key published in DNS, used by recipients to verify that the message was signed by your mail server's private key. OpenDKIM generates the keys; you publish the public half.

1opendkim-genkey -s 2017a -d example.com -b 2048

This creates 2017a.private (deploy to OpenDKIM) and 2017a.txt (publish to DNS). The selector (2017a here) lets you rotate keys later without breaking validation of older signed mail; pick a new selector annually.

12017a._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjAN..."

DMARC (Domain-based Message Authentication, Reporting and Conformance). A TXT record that tells receivers what to do with mail that fails SPF or DKIM, and where to send aggregate reports.

1_dmarc.example.com. IN TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com; ruf=mailto:dmarc@example.com; fo=1"

Start with p=none for two weeks while you read the aggregate reports and confirm legitimate sources are passing. Then move to p=quarantine, then p=reject once you are confident. Going straight to p=reject from a fresh setup is how you discover at 9 AM that your monitoring system has been silently failing to deliver alerts for a week.

Reverse DNS and the IP reputation problem

Two things separate "mail server" from "spam source" in the eyes of large receivers.

Reverse DNS (PTR record). The IP your mail leaves from must reverse-resolve to a hostname under your domain, and that hostname must forward-resolve back to the same IP. Most VPS providers let you set the PTR for your assigned IP. Set it. Verify it with dig -x 203.0.113.10. A missing or generic reverse DNS (the kind that says 203-0-113-10.cloud-provider.example) is the single most common reason new mail servers fail Gmail's heuristics.

IP reputation. This is the harder problem. Large cloud providers — AWS EC2, DigitalOcean, Vultr, Hetzner Cloud — have blocks of IPs that have been used to send spam, and the major receivers blocklist large chunks of them by default. If your mail leaves from an IP previously used by a spammer, your mail goes to the spam folder regardless of how clean your config is.

Solutions, in increasing order of effort:

1. Pick a small, reputable VPS provider with clean IP allocation. Hetzner's dedicated server line and OVH's mid-range offerings both tend to give cleaner IPs than the big-three clouds.
2. Check the IP against mxtoolbox.com/blacklists.aspx before deploying. If it is on Spamhaus, Barracuda, or SORBS, request delisting before sending mail, or get a different IP.
3. Use a smarthost relay for outbound mail. Services like Mailgun, SendGrid, or Amazon SES will relay mail for you at low volume for free, and they have spent years building IP reputation. Postfix can be configured to use them as relayhosts for outbound only — you keep inbound on your own server, and outbound goes through a paid relay. This sidesteps the reputation problem entirely at the cost of dependence on a third party.
   The third option is increasingly popular and is, honestly, the pragmatic choice for most people in 2017. Pure self-hosting of outbound is a project; self-hosting inbound while relaying outbound is an afternoon.

TLS, ports, and modern submission

Three ports are relevant in 2017:

• Port 25 — server-to-server SMTP, opportunistic STARTTLS. Inbound from the world arrives here.
• Port 587 — submission, mandatory STARTTLS, authenticated. Your mail clients connect here.
• Port 465 — SMTPS, implicit TLS. Long deprecated, recently un-deprecated by RFC 8314 (released January 2018, but the practice was returning by 2017). Some clients only support 465; offering both 587 and 465 is the safe choice.
  Use Let's Encrypt to obtain a certificate for mail.example.com and configure both Postfix and Dovecot to use it. Test the result against testssl.sh:

1testssl.sh --starttls smtp mail.example.com:25
2testssl.sh --starttls smtp mail.example.com:587
3testssl.sh mail.example.com:993

A 2017-clean configuration disables SSLv2, SSLv3, and TLSv1.0, prefers TLSv1.2 with modern cipher suites, and has reasonable Diffie-Hellman parameters (at least 2048-bit, ideally generated locally with openssl dhparam).

What the project actually costs

Plan for:

• A weekend of initial setup if you've done it before, two weekends if you haven't.
• A few hours per quarter on maintenance: certificate renewals (automated but worth verifying), package updates, log review.
• Occasional emergencies when something on the upstream side changes — a major receiver tightens their DMARC enforcement, a blocklist adds your provider's range, a CA expires its intermediate.
• Approximately three to ten euros a month for the VPS, depending on provider and specs.
• A non-zero amount of social cost when family members or colleagues ask why their messages to you keep landing in your spam folder.
  That last one is real. Some recipients' mail servers will spam-fold your mail no matter how clean you are, because their administrator has an aggressive bayesian filter trained on a corpus that thinks small-domain mail is suspicious. There is nothing you can do about this from your side. The fix is on theirs.

Whether to do it

Self-hosting email in 2017 is a different proposition than it was in 2010. Then it was easy and uncommon. Now it is hard and rare. The reasons to still do it have not changed:

• You own the system end to end.

• No third party can read, sell, or scan your mail.

• You learn how email actually works, which is increasingly rare knowledge.

• You are not one terms-of-service update away from losing your address.
  The reasons against have grown:

• The deliverability fight is real and ongoing.

• The mail server is not the only thing you now operate; you are operating SPF, DKIM, DMARC, reverse DNS, and a growing pile of authentication infrastructure.

• The major providers have, by accident or by design, made things hard for small operators.
  The honest answer in 2017: self-host inbound mail because that is fully under your control, relay outbound through a reputable service if your IP reputation is questionable, and accept that the dream of a pure self-hosted setup is harder than it was. The principles still hold. The execution requires more skill than it used to.

If you do nothing else after reading this: set up DMARC reporting on whatever email infrastructure you currently use, even if it is Gmail. The aggregate reports will tell you who is sending mail claiming to be from your domain. You will be surprised.