How to Set Up Your Own Mail Server on a VPS
Self-hosted email has a reputation problem. Half the internet calls it a pleasant weekend project, the other half insists nobody should attempt it after 2015. Both camps are describing the same thing from different distances.
The arithmetic usually starts the conversation. Hosted business email is billed per person: Google Workspace Business Starter sits at $7 per user per month on an annual commitment and $8.40 month to month, and Microsoft 365 Business Basic matches that $7 after its July 2026 pricing update. Twenty-five people on a starter plan comes to roughly $2,100 a year, and none of it buys capacity you control. A VPS capable of serving those same twenty-five mailboxes costs a fraction of that, and the figure stays flat whether you hire five people or fifty.
Cost is rarely the whole reason. The rest comes from procurement questionnaires, which now routinely ask where message data is stored, who administers it, and what happens to the archive when a vendor changes terms.
What follows is the practical version: what decides success before you install anything, how the installation actually goes, and where self-hosted mail tends to go wrong.
Four Things Decide Whether This Works
Most failed mail servers fail for reasons that have nothing to do with the software running on them.
The IP address needs a clean history. Receiving networks score the address, not your intentions. A recycled address that previously belonged to a spam operation starts you in a hole, and climbing out takes weeks.
Outbound port 25 has to be reachable. Mail servers hand messages to each other on that port and nowhere else. A significant number of US providers disable it by default.
DNS has to be complete and internally consistent. Five records, all agreeing with each other. Miss one and your mail technically works while quietly landing in spam folders.
Somebody has to maintain it. Updates, certificate renewals, backup verification. Half an hour a month, but it has to be somebody’s half hour.
Get those four right and the rest is a Tuesday afternoon of configuration. Get any of them wrong and no amount of tuning will rescue the result.
What Sits Inside the Box
A mail server is not a single program. It is a handful of services with separate jobs, each listening on its own port, which is exactly why firewall rules and provider policies matter so much here.
| Service | Port | Job |
|---|---|---|
| SMTP, server to server | 25 | Receives mail from the rest of the internet and hands your outgoing mail to other servers |
| SMTP submission | 587 and 465 | Accepts authenticated mail from your own users and applications |
| IMAP | 993 | Serves mailboxes to clients while messages stay on the server |
| POP3 | 995 | Downloads messages to one device and clears them from the server |
| HTTP and HTTPS | 80 and 443 | Webmail, admin panel, TLS certificate issuance |
The useful detail sits in the first two rows. Port 25 carries traffic between servers and never involves a password, which is precisely why compromised machines abuse it and why providers restrict it. Ports 587 and 465 always require authentication, so nobody blocks those.
One component lives outside the server entirely. An MX record in your domain’s DNS announces which host accepts mail for you. To see where a domain currently sends its mail, dig MX example.com +short answers in a second, and the number in front of each hostname is a preference value, with lower numbers tried first.
Sizing the Server and Vetting the Provider
Hardware requirements follow the stack you choose. Mailcow, the most feature-complete option, documents 6 GB of RAM plus 1 GB of swap and 20 GB of disk before a single message is stored, and suggests 8 GB once you pass five users. Leaner setups run happily on 1 or 2 GB. Storage for the mailboxes is a separate calculation, driven by how much history your team keeps.
Provider selection deserves more scrutiny than the hardware spec, and three questions cover it.
Is outbound port 25 available? This is where projects die quietly. DigitalOcean blocks SMTP on Droplets for new accounts and directs customers to open a support ticket, with roughly sixty days of account history expected first. Several other US providers apply comparable rules. Establish this before you build anything, because a server that cannot reach port 25 cannot deliver mail at all.
Will they publish a PTR record for you? Reverse DNS belongs to whoever owns the IP address. A provider that will not configure it leaves your mail permanently second-class.
Where are the data centers? Relevant for latency, more relevant for the compliance conversation that probably prompted the project.
Settle all three before you build anything. On Serverspace, reverse DNS is configured by support on request and data centers span several regions including the US. Docker can also be selected during server creation, which removes a stage from the walkthrough below.
A small requirement causes outsized trouble here: keep time synchronization running. Clock drift breaks signature validation and time-based authentication codes in ways that are genuinely unpleasant to diagnose.
Picking the Stack
Three families of software cover nearly every deployment.
Building from parts means Postfix for transport, Dovecot for mailbox access, and separate decisions about filtering, storage, and webmail. Total control, and every upgrade becomes yours to plan.
Packaged Docker stacks ship those same components pre-wired with an admin interface on top. This is what most teams actually want.
Single-binary servers represent the newer approach: one process, one configuration file, dramatically less memory.
| Option | Composition | RAM floor | Admin interface | Where it fits |
|---|---|---|---|---|
| Postfix with Dovecot | Transport and mailbox access only, everything else chosen by you | from 512 MB | None | Requirements that packaged stacks get in the way of |
| Mailcow | Full suite including Rspamd filtering, antivirus, SOGo groupware | 6 GB and 1 GB swap | Yes | Replacing hosted business email outright |
| Mailu | Similar components, fewer of them, MIT licensed | roughly 1.5 GB | Yes | Small teams with no groupware requirement |
| Stalwart | SMTP, IMAP, JMAP and filtering inside one Rust binary | roughly 200 MB under load | Yes | Low-memory servers and modern protocol support |
| docker-mailserver | Postfix and Dovecot containerized, no database, files only | roughly 1 GB | None | Infrastructure kept in version control |
| Mail-in-a-Box | Prescriptive bundle, single installation command | roughly 1 GB | Yes | One domain, one administrator, minimal fuss |
Mailcow carries the walkthrough below. Its documentation is the most thorough of the group, the panel handles domains and mailboxes without opening a config file, and message signing and filtering arrive already configured.
From Empty Server to Working Mailbox
Five stages, roughly forty minutes if DNS propagation cooperates.
Stage one: name the machine first. Choose the hostname before anything else, typically mail.yourdomain.com, publish an A record for it, and apply the same name to the server with hostnamectl set-hostname. The hostname, the A record, and the reverse record you will request later all have to agree. Receiving servers compare them, and mismatches explain a large share of spam-folder complaints.
Stage two: Docker. Install Docker Engine and the Compose plugin from the official repository for your distribution, then confirm both respond to --version. If your provider offers Docker as a deployment option, this stage disappears entirely.
Stage three: generate the configuration. Clone the mailcow-dockerized repository into /opt and run its generate_config.sh script, which asks for the mail hostname and time zone and writes mailcow.conf. Two settings in that file are worth knowing immediately: SKIP_CLAMD=y and SKIP_FTS=y switch off the antivirus scanner and full-text indexing, the two components the maintainers name as the heaviest consumers of memory. On a 4 GB machine, disabling them is the difference between a responsive server and one that swaps.
Stage four: firewall, then launch. Allow 25, 80, 443, 465, 587, 993 and 995 through ufw. Before starting the stack, run ss -tlpn to confirm nothing already occupies those ports, since most distributions ship a minimal mail transfer agent that claims 25. Then bring the containers up with docker compose up -d. The first run pulls a lot of images and takes several minutes.
Stage five: log in and lock down. The panel answers over HTTPS at your mail hostname. Change the default administrator password before anything else, then add your domain and your first mailbox. The panel displays a DKIM key at that point, so leave the tab open, because it goes into DNS next.
Connecting a client means IMAP on 993 with SSL and submission on 587 with STARTTLS, and the username is always the full address. Send a message out, send one back in, and if either direction fails, docker compose logs postfix-mailcow explains why.
The DNS Layer That Makes or Breaks Delivery
Your server works now. Whether anyone receives what it sends is decided in DNS.
MX routes incoming mail to your hostname.
SPF declares which addresses may send as your domain. A single-server record reads v=spf1 mx a ip4:your.ip.here -all, and the closing -all is the part that gives it teeth.
DKIM attaches a cryptographic signature to outgoing messages. Mailcow generates the value and you publish it as a TXT record under the selector shown in the panel. When DKIM fails validation, the cause is almost always a mistyped selector or a key truncated on its way into a DNS form.
DMARC tells receivers how to treat messages that fail the first two checks. Publish v=DMARC1; p=none; with a reporting address and leave it alone for a few weeks. Enforcement applied on day one silently strangles mail from every other system that sends as your domain, and billing platforms are usually the first casualty.
PTR maps the IP address back to your hostname, and you cannot publish it yourself, because the reverse zone belongs to your provider. Serverspace handles it through a support request: supply the address and the hostname, and it takes effect within a few hours. Records for the domain itself can be managed from the control panel once the domain is delegated to the platform name servers, and available configurations are listed on the VPS page.
Whatever value goes into PTR has to match the name your server announces when it connects outward. That single correspondence carries more weight with spam filters than anything else discussed here.
What Receiving Providers Expect
Gmail and Yahoo tightened their sender rules considerably, and the baseline now applies to everyone rather than only bulk senders.
Every sender needs SPF or DKIM in place, TLS on the connection, and a complaint rate below 0.3%. Cross 5,000 messages a day to Gmail addresses and DMARC plus matching forward and reverse DNS become mandatory. The grace period closed in November 2025, when Gmail started issuing temporary and permanent rejections instead of quietly filtering non-compliant messages.
Beyond the checklist, two habits matter. Start slowly, because a fresh address with no sending history that suddenly emits thousands of messages looks statistically identical to a compromised host. And test before you trust: mail-tester scores a sample message and names what is missing, which beats reading your own logs.
The Honest Trade-off
Arguments in favor:
- pricing decouples from headcount, so growth stops being a line-item negotiation;
- data-residency questions in security reviews get a specific answer instead of a link to somebody’s compliance page, which helps under SOC 2, HIPAA, and CCPA;
- aliases, shared mailboxes, and retention rules cost nothing and carry no seat limits;
- no vendor can reprice or deprecate your email.
Arguments against:
- maintenance never ends, and a neglected mail server degrades faster than a neglected web server;
- reputation management lands on you, and blocklist removal moves at somebody else’s pace;
- the archive is only as safe as your last verified restore;
- mail wants its own machine, isolated from anything else facing the public internet.
A hybrid model is worth considering. Keep mailboxes and storage in-house, route outbound through a dedicated sending service, and you trade a measure of independence for somebody else’s reputation team.
Where It Genuinely Pays Off
Teams past twenty people. Per-seat pricing overtakes the cost of a capable VPS well before that point, while the features most offices touch stay identical.
Organizations answering data questions. Auditors and enterprise customers ask where email lives. Owning the server produces a short, verifiable answer.
Applications that send. Receipts, password resets, and notifications leaving your own infrastructure are cheaper to scale and far easier to trace when one goes missing.
Per-vendor addressing. A unique address for every service, all forwarding into one mailbox, turns the eventual spam into evidence of who leaked the list.
Separate domains under one roof. A subsidiary, a spun-out product, or a department with its own retention policy gets a mail domain without a second subscription.
Where Deployments Go Wrong
| Symptom | Underlying cause | What to do |
|---|---|---|
| Incoming mail works, outgoing vanishes | Outbound port 25 is filtered by the provider | Confirm the block, request an exception, or relay through an authenticated service on 587 |
| Everything lands in spam despite valid SPF and DKIM | PTR missing, or pointing somewhere other than the announced hostname | Have reverse DNS set to the exact name the server uses when it connects out |
| Mail from the CRM and billing tools stops arriving | DMARC moved to enforcement before those systems were aligned | Roll back to p=none, read the aggregate reports, align each sender, then tighten |
| DKIM signatures fail validation | Selector misspelled, or the key truncated when pasted | Compare the published record against the panel character for character |
| Mailboxes empty after a container rebuild | Message data was never in a persistent volume | Move it to named volumes and schedule backups you periodically restore from |
| Constant swapping and sluggish webmail | Full suite deployed on a 2 GB instance | Add memory, or disable the antivirus scanner and full-text indexing |
| Clients disconnect overnight with certificate errors | Automatic renewal stopped working some weeks earlier | Automate renewal and alert on the expiry date instead of checking by hand |
Before You Point the MX Record
Everything above can be verified before a single colleague depends on it, which is the entire advantage of migrating deliberately.
Five checks:
- A test message scores cleanly in mail-tester with no critical findings.
- PTR resolves to the same hostname the server announces.
- Backups of the mail volume run on a schedule, and you have restored from one successfully at least once.
- Certificate renewal is automated and monitored.
- Old MX entries are ready to be removed rather than left behind as accidental fallbacks.
Then change the MX record and watch the logs for an hour. Problems surface immediately or not at all.
For the machine underneath, configurations are available in a few minutes at Serverspace, and the build described here runs comfortably on a mid-range instance.
