Sending Email from SmartOS

Sending email is complicated today.  Unsolicited spam is easily as rampant as it has ever been, which has led to legal precedents, clever countermeasures, and ultimately high technical requirements just to reliably have your email end up in the right inbox.

Setting up a simple email server just isn't that simple anymore.  An entire industry has sprung up around the reliable delivery of email, and most sane people will offload their email delivery to one of many third parties that specialize in delivering transactional email, or in a pinch, even Google.

There are costs, however, in service fees, increased sending latency and in surrendering control to a third party, and while email server hosting is certainly more complex than it has been, it can still be economically done in-house.

This guide will focus on setting up Postfix on a dedicated SmartOS Zone (base-64-lts, v15.4.1) for use as a send-only email server, suitable for sending automated messages from a small to medium sized project directly to recipients.  In addition to Postfix, we will also be covering the following sub-topics:

• DKIM
• SPF
• DMARC
• Reverse DNS
• Postfix TLS
• Message Rate Limiting
• Monitoring
• Archiving

Beyond this guide, I recommend reading Postfix Documentation.  I found it's insight to be invaluable while researching for this article, and I have certainly not done it justice here.

SmartOS Zone Configuration

For reference, I'm operating on a dedicated zone that has the following configuration:

{
  "alias": "cospix.pdx0.mx1",
  "hostname": "cospix-pdx0-mx1",
  "brand": "joyent",
  "image_uuid": "088b97b0-e1a1-11e5-b895-9baa2086eb33",
  "max_physical_memory": 1024,
  "cpu_cap": 100,
  "quota": 1000,
  "delegate_dataset": true,
  "nics": [{
    "nic_tag": "cospix0",
    "ips": [ "ipv4/netmask", "addrconf" ],
    "gateways": [ "ipv4" ]
  }],
  "resolvers": [ "8.8.8.8", "8.8.4.4" ]
}

Quota

You should always set a zone quota that can easily handle the volume of email you intend on storing.  To calculate this intelligently, you should have an idea of what your email volume numbers are, specifically:

• The volume of email you send in a month (in GB/mo, assume 5GB)
• The volume of email you receive in a month (in GB/mo, assume 10GB)
• The rate at which your sending and receiving volumes are changing each month (with 1.0 being no change, assume 1.1 or 110% for each)
• Maximum period of time that the mail server will be unable to deliver messages (5 days)
• When you expect to replace this email server (in months, assume 24)
• How long you want to archive email for (in months, assume 36)

While we could go to a second-order model, the reality is a bit too unpredictable for anything beyond linear interpolation in this context.  This is also why you shouldn't forecast much beyond 24 months.

Notice: These numbers have nothing to do with email per user but instead absolute email volume across an organization, and should be measured, not assumed like we have done here.  Worst case you can also guess low and increase quota later on.

Consider that a simple send-only email server only has to hold (outgoing) messages in its mail queue until they've been delivered.  The worst case for this getting too large is if Postfix is unable to deliver messages for a period of time.  Based on our assumptions above, we can calculate this number as the product of the volume of messages sent a month, the fraction of maximum downtime per month, and the rate of growth after the server's lifetime.  That number comes out as roughly 8.2GB:

\[ S_{queue} = 5GB \cdot \frac{5}{30} \cdot 1.1^{24} \approx 8.2GB \]

A quota of 10GB should be appropriate for the Postfix queue in a send-only mail server.

If you intend on locally archiving email on this server, you will need to expand the zone quota to handle the added data storage requirements, which can be roughly calculated as the product of the monthly outgoing email volume and the definite integral of the rate of growth of outgoing volume across the months you wish to store your data for.  With our model, that comes out being roughly 1600GB.

\[ S_{archive} = 5GB \cdot \int^{36}_{0} 1.1^{x}dx \approx 5GB \cdot 314 \approx 1.6TB \]

If you intend on also receiving email through this server, you will need to account for storage requirements of the individual email boxes, as well as additional archive storage (if you are also locally archiving email).  Consider that a large number of users may wish to store their email for years, perhaps even indefinitely.  Since we can't easily model user's deletion habits we can just assume they will equal data storage requirements of the incoming archive, which comes out as roughly 6400GB.

\[ S_{incoming} = 2 \cdot 10GB \cdot \int^{36}_{0} 1.1^{x} dx \approx 2 \cdot 10GB \cdot 314 \approx 6.4TB \]

Adding these three sizes up gives us an estimated storage requirement of over eight terabytes, which is well beyond our available zones pool capacity.  There is a silver lining though, as the exercise has made it fairly clear as to how quickly email data can accumulate, and where it should be allocated.  The incoming message volume (6400GB) is actually going half to the incoming mailboxes (3200GB) and half to the incoming archive (3200GB), while the outgoing archive volume is exclusively archive (1600GB).  We can assume that the archive should be slightly larger than (specifically, the difference of the outgoing email archive) the incoming mailboxes dataset size.  That information will become useful in the next section.  For this example, we are going to limit our installation to 1TB for email.

Notice: If you are locally archiving email through this zone, you may want to increase max_physical_memory to allow for efficient compression of daily archives.  I recommend a minimum of 1GB.

Delegated Dataset

Dataset delegation is a double-edged sword.  On one hand, it's easy to limit your maximum email queue size with a quota set on a dataset mounted under /var/spool/postfix; a feature that will likely appeal to larger installations.  On the other hand, it introduces added complexity to the zone.  If you plan on limiting your operation to sending only, you may choose to not use dataset delegation here with relatively little consequence.

Notice: You should always use dataset delegation if you intend on receiving or archiving email through this zone.

If you did decide to enable dataset delegation, perform the following additional operations within the zone to store your Postfix queue in it's own dedicated dataset:

# UUID=`sysinfo | json UUID`
# zfs set mountpoint=none zones/$UUID/data
# rmdir -p /zones/$UUID /var/mail/\:saved /var/spool/postfix/etc
# zfs create -o quota=980G -o mountpoint=/var/mail zones/$UUID/data/mail
# zfs create -o quota=10G -o mountpoint=/var/spool/postfix \
zones/$UUID/data/mail/postfix

These commands will remove the default data mount point as well as the directory cruft that it (and other things) produces.  Next it will create a general mail ZFS dataset as an umbrella for the rest of our operations, and finally, it will create the postfix dataset for the postfix mail spool.  The quota size of 990G assumes a zone quota of 1000G (leaving 20G for non-email related files).

Dedicated IP Address

While you can share an IP address with several other zones behind a NATing router, or even share a single SmartOS Zone, you should ideally run your email server on its own stable, dedicated IP address.  This becomes a serious factor as your email volume increases past the capability of a single host, as many spam mitigation techniques use spikes in the number of messages coming from a given IP address (past a minimum threshold) as a method of detecting new spam-generating hosts, and the task of migrating your sending email IP addresses should not be taken lightly:

Configuring Postfix

Postfix comes pre-installed and configured on all modern SmartOS zone images, and only needs a few simple tweaks to get working as we'll need it to (thanks, Joyent!)  Postfix also comes with a wonderful command-line configuration viewing and modifying tool called postconf, which we will be using to alter our Postfix configuration rather than editing main.cf directly.

# postconf -e <directive>=<value> [<directive>=<value>...]

Multiple (space separated) directive-value pairs can be provided to postconf and it will do the hard work of searching the configuration file for an existing instance or appending a value to the end.  Values which have spaces in them can be enclosed in quotations, and postfix variable prefixes should be escaped (ie: $ turns into \$).  It can also be used without the -e flag to view configured directives.

Let's start by setting a few basic configuration directives.

# postconf -e myhostname=mx.cospix.net \
mynetworks="\$myhostname, 127.0.0.0/8, [::1/128]"

The myhostname parameter allows us to explicitly set the name that Postfix will use when identifying itself to other software agents.  While this could be set to any domain name, it's best to set it to a name within a domain that you control (especially if you intend on receiving email with this service).  Common examples are mx.cospix.net, mx1.cospix.net, pdx.mx.cospix.net, etc.

The mynetworks parameter allows us to specify which hosts and networks should be considered local.  By default, local hosts are allowed to send email with no authorization checks.

Note: Many Postfix guides recommend setting the myorigin parameter to be $mydomain instead of (my preference of) its default of $myhostname.  While this only makes sense to me on the medium scale, where your local (UNIX) users will be the ones sending email.  Feel free to choose either way.

Additionally, the postmaster account should be aliased to an account you check regularly.  As postmaster is already redirected to root, set the following parameter:

/opt/local/etc/postfix/aliases

root: your@email.address

Run newaliases before continuing.

# newaliases

Using a Relay Service

If you're planning on using a local relay host to forward all of your outgoing mail through, you can complete this section (on Postfix configuration) and safely ignore the rest of this guide, as the relay host you use will be the one scrutinized by other agents.

To have Postfix relay email through an intermediate host, set the following parameter:

# postconf -e relayhost="[hostname]:port"

Where hostname is the hostname or IP address of your relay email server, and :port is the optional port, which can be omitted in the case of a default port.

Using an Authenticated Relay Service

If you're using an authenticated email relay service such as any of the ESPs listed above, you will need to authenticate your connection to their servers.  Fortunately, this is easy to do with Postfix.  Set the following parameters:

# postconf -e smtp_sasl_auth_enable=yes \
smtp_sasl_password_maps=hash:smtp_passwd \
smtp_sasl_security_options=noanonymous

The smtp_passwords file stores a set of tuples containing a hostname, port, username, and password, suitable for login.  Below is the example contents of this file.

/var/db/postfix/smtp_passwd

[hostname] username:password
[hostname]:port username:password

Run postmap before you continue.

# postmap /var/db/postfix/smtp_passwd

Important: Also read and follow the Postfix TLS section below, as you will be otherwise be sending your authentication credentials in plain text.

Providing Unauthenticated Relay Service

If you are using this zone to relay messages for other hosts on your local network (see the above sub-section), you should additionally include the hosts you will be relaying for in mynetworks.  We will not be going over how to set up authenticated relaying as that would be well outside the scope of the rest of this guide.  Set or change the following parameters:

# postconf -e inet_interfaces="1.2.3.4, localhost" \
mynetworks="\$myhostname, 127.0.0.0/8, [::1/128], 1.2.3.0/24"

Where 1.2.3.4 is your email server's local network address and 1.2.3.0/24 is the network segment that you will be relaying for.  Note: This should only include hosts you have direct control over.  If you're in a situation where you need to listen to all network addresses to relay traffic (such as only having one network address), then set the following configuration:

# postconf -e inet_interfaces=all

The only reason to run with an open SMTP port on the public Internet is to receive or forward email.  If you only intend on using this server to send messages (and never receive them), you should also include a firewall rule to block incoming connections to port 25.

/etc/ipf/ipf.conf

block in on net0 proto tcp from any to any port = 25

Deny any incoming traffic on net0 intended for tcp/25, since this zone isn't doing any routing, this is sufficient.

Be sure to enable IPFilter when you're done.

# svcadm enable ipfilter

Enable and test

After you're done with configuration, start Postfix.

# svcadm enable postfix

We're going to use socat to test email, as it gives us the most feedback as to what's happening.  Client input and server responses can be distinguished by the fact that all server responses are prefixed with a 3 digit code (whereas no client input is).

# socat - TCP:localhost:smtp
220 mx.cospix.net ESMTP
EHLO mx.cospix.net
250-mx.cospix.net
250-PIPELINING
250-SIZE 51200000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
MAIL FROM: <root@cospix.net>
250 2.1.0 Ok
RCPT TO: <your@email.address>
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
Subject: Test Email
This is a test email
.
250 2.0.0 Ok: queued as 5B69E5E
QUIT
221 2.0.0 Bye

I recommend performing this send test after completing each section of this guide to verify that everything is functioning as intended.  If you are using this email host to relay messages, it's recommended to additionally test at least once from where the messages originate from.

Configuring DKIM

Domain Keys Identified Email is a system for generating cryptographically secure message digests for email passing through a Mail Transfer Agent (MTA) that are verifiable via public keys stored in DNS.  In short, it allows an email administrator to provide additional assurance that a message is valid.

Messages that have a valid DKIM signature are treated as less suspect than messages that do not, so we will want to use DKIM to wrapper our outgoing email.  Because of its approach, it requires modification to both our MTA and the DNS records of domains we will be sending email for to function.

Let's start by installing OpenDKIM to our zone.

# pkgin in opendkim

Next, let's let Postfix know about OpenDKIM.  Set the following configuration parameters.

# postconf -e milter_protocol=2 milter_default_action=accept \
smtpd_milters=unix:/var/db/opendkim/opendkim.sock \
non_smtpd_milters=unix:/var/db/opendkim/opendkim.sock

Since we're using domain sockets instead of TCP sockets, we will need to add Postfix to OpenDKIM's group so that Postfix and OpenDKIM can communicate.

# usermod -G mail,opendkim postfix

Next, set the entire contents of the OpenDKIM configuration file to the following:

/opt/local/etc/opendkim.conf

Canonicalization        relaxed/simple
ExternalIgnoreList      refile:/var/db/opendkim/TrustedHosts
InternalHosts           refile:/var/db/opendkim/TrustedHosts
KeyTable                refile:/var/db/opendkim/KeyTable
SigningTable            refile:/var/db/opendkim/SigningTable
Socket                  local:/var/db/opendkim/opendkim.sock
Syslog                  Yes
UMask                   007

Populate the TrustedHosts table.  These are hosts which OpenDKIM will sign messages from.

/var/db/opendkim/TrustedHosts

127.0.0.1
::1

Create a default 2048-bit key for our domain.

Note: If you have multiple domains, the below commands should be repeated per domain.

# mkdir -p /var/db/opendkim/keys/cospix.net
# opendkim-genkey -b 2048 -d cospix.net -D /var/db/opendkim/keys/cospix.net -r

Once you're done, lock everything down to minimal permissions.  The revocation of group permission (note the 0s in the group column) is due to the inclusion of Postfix in the OpenDKIM system group and it's solely to maintain compartmentalization between the services.

# chown -R opendkim:opendkim /var/db/opendkim
# chmod 400 /var/db/opendkim/*
# chmod 710 /var/db/opendkim
# chmod 500 /var/db/opendkim/keys /var/db/opendkim/keys/*/
# chmod 400 /var/db/opendkim/keys/*/*

Next, we will tell OpenDKIM about the keys we just generated.

Note: Each domain should have its own line, the following example is a single line:

/var/db/opendkim/KeyTable

default._domainkey.cospix.net cospix.net:default:/var/db/opendkim/keys/cospix.net/default.private

/var/db/opendkim/SigningTable

*@cospix.net default._domainkey.cospix.net

Now we can start OpenDKIM and refresh Postfix.

# svcadm enable opendkim
# svcadm restart postfix

Finally, we will want to publish each DKIM public key in DNS.  Fortunately, this is really easy to do as OpenDKIM generates each record for us.

/var/db/opendkim/keys/cospix.net/default.txt

default._domainkey      IN      TXT     ( "v=DKIM1; k=rsa; s=email; "
"p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOymcU1ODODcKUathqCXzDgQPV0YhhWUOg+mNVkKzymZ6iE0+/0UrqPXCpOW9hfHXuFrWR74wzPZlhkJknPh80D57Omd1w7hVnmUyRFTEAKpNkpolO4BbFNs+CUTO6oYbPO4s8dRzJen8M39U4aA0bUuyO9slXMf4Vz4kszSuGnwIDAQAB" )  ; ----- DKIM key default for cospix.net

Create a DNS TXT record based on this information.  Your standing DNS records should look something like this when you're done:

# dig +short txt default._domainkey.cospix.net
"v=DKIM1\; k=rsa\; s=email\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDOymcU1ODODcKUathqCXzDgQPV0YhhWUOg+mNVkKzymZ6iE0+/0UrqPXCpOW9hfHXuFrWR74wzPZlhkJknPh80D57Omd1w7hVnmUyRFTEAKpNkpolO4BbFNs+CUTO6oYbPO4s8dRzJen8M39U4aA0bUuyO9slXMf4Vz4kszSuGnwIDAQAB"

Now would be a great time to send another test message.  You should see the following additional headers in your test email:

...
Authentication-Results: mx.google.com;
   dkim=pass header.i=@cospix.net;
...
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cospix.net;
    s=default; t=1469938684;
    bh=VAgC5MVP54VmvlcGQaMT2ZdmokXkhMKi/RNSpcUu2qw=;
    h=Subject:Date:From;
    b=pbkdhvCXZyeOwq3iyFTDcMo+guaFhCUrT7amMHPP5MNA7COz1BkFPSrMZNbGjmlwp
     p0Ohh62YqmWucnZzJjY0AnLM6Dgz/WeL9K1YX1COd3Ns28GX9fzMBlLLZMnJKW+xQ8
     j/1+5CyRsfvkcGXLYWLg+vwzLZbTWdrwPmynfiV0=
...

Adding an ADSP record

Besides using DNS to distribute public keys, DKIM can inform receivers on policy about how email should be handled from the domain in the context of DKIM.  Specifically, what to do with unsigned or unverifiable emails.  While this has been depreciated in favor of using DMARC records, it's still worth it to implement, as it's a single TXT record under the _adsp sub-domain immediately subordinate to the target domain, and will be checked anyways by ADSP compliant implementations.

The contents of the TXT record are a key-value pair with a key that can only be dkim and one of three possible values:

• unknown: (default) Some, most, or all email from the domain will be signed.
• all: All email from the domain will be signed.
• discardable: All email from the domain will be signed, and if a signature is missing or invalid, the message is to be silently discarded.

The first value is a waste of time and the last value is a bit draconian, so we'll opt for the middle value.

Your dig output should look like this when you're done.

# dig +short txt _adsp._domainkey.cospix.net
dkim=all

Configuring SPF

The Sender Policy Framework allows email administrators to incorporate a host access control list into the DNS records for a given domain.  Emails that originate from a host that is on this whitelist are considered less suspect by receiving MTAs, so we will want to ensure that we have a valid SPF record for our sending domains which includes our sending MTAs.

This is an example of a valid SPF record:

# dig +short txt cospix.net
"v=spf1 mx include:_spf.google.com -all"

All SPF version 1 records begin with v=spf1 and have a space separated list of one or more of the following parameters:

• ALL: Matches everything, often used at the end of a list in the negative form (-all) to black-list all addresses not explicitly referenced.
• A: Matches a domain's A or AAAA records.
• IP4: Matches an explicit IPv4 address or IPv4 address range (ip4:1.2.3.4 or ip4:1.2.3.0/24).
• IP6: Matches an explicit IPv6 address or IPv6 address range (ip6:bad0::1 or ip6:bad0::/64).
• MX: Matches the domain's MX records.
• PTR: Really shouldn't be used anymore.
• INCLUDE: Includes the SPF list from another domain (include:_spf.google.com).

Additionally, each element of the list can be qualified with the following prefixes:

• +: Pass.  This can be omitted (it's the default qualifier).
• ?: Neutral.  This omits the item it's qualifying from its list.
• ~: Softfail.  This is a cross between neutral and fail, usually, results in a message being accepted but tagged suspicious.
• -: Fail.  Messages from this host should be rejected.

Note that the use of the SPF record type has been deprecated in favor of using TXTand that it applies directly to the domain that it's describing.

All SPF records should end with -all, specifically set them up as white-lists, then simply include the parameters you need.

Say that you want to only allow messages from the mail exchangers:

@ IN TXT "v=spf1 mx -all"

Only allow email from the host:

@ in TXT "v=spf1 a -all"

Only allow email from a subnet:

@ in TXT "v=spf1 ip4:1.2.3.0/24 -all"

Only allow email from the mail exchangers and google:

@ in TXT "v=spf1 mx include:_spf.google.com -all"

My recommendation here is to only make your SPF as permissive as it needs to be, which configuration that is depends on your specific situation.

Configuring DMARC

Domain-based Message Authentication, Reporting, and Conformance is more of a policy protocol rather than a message or host authentication mechanism.  It allows the administrators of a sending domain to recommend a policy when dealing with messages that fail validation checks (SPF and DKIM) as well as providing a method of feedback to the administrators of the sending domain.

As with SPF, this protocol lives in DNS, specifically as a TXT record under the _dmarc subdomain.

# dig +short txt _dmarc.cospix.net
"v=DMARC1\; p=none\; rua=mailto:postmaster@mx.cospix.net"

All DMARC version 1 records are formatted as semicolon separated labeled parameters, with the first two, v and p being required. A possibly relevant list follows:

• v: Version, currently, only v:DMARC1.
• p: Policy; one of none, quarantine or reject.  Recommendation on how to handle messages that fail SPF or DKIM.  It is best to start with none and set rua/ruf below and then change to quarantine or reject after you know that you will not incur false positives.
• sp: Sub-policy; one of none, quarantine or `reject.  Same as policy but it applies to sub-domains.
• pct: Percent; 0-100.  Percentage of email that is checked by DMARC.  This should always be 100% (default).
• rua: Failure Report Aggregation; Email URI (mailto:dmarc-rua@cospix.net). This is where aggregate reports on messages that fail authentication are sent.
• ruf: Failure Report Forensics; Email URI (mailto:dmarc-ruf@cospix.net).  This is where sample messages that fail authentication are forwarded to.  It's recommended to not use for long periods of time unless you're okay with copious volumes of highly suspicious email.
• adkim: DKIM Identifier Alignment; one of r (relaxed) or s (strict).  If your emails adhere to strict or relaxed DKIM.  Given our above configuration, this should be r here (default).
• aspf: SPF Identifier Alignment; one of r (relaxed) or s (strict).
• rf: Report Format; one of afrf (Authentication Failure Reporting Format) or iodef (Incident Object Description Exchange Format).  Best to leave on default, unless it isn't.
• ri: Reporting Interval; 1-4294967295.  The number of seconds between single aggregate reports.  86400 is the number of seconds in one 24-hour period and is the default value.

For now, our DMARC record is recommending a policy of permit and report back to our postmaster account:

_dmarc IN TXT "v=DMARC1; p=none; rua=mailto:postmaster@cospix.net"

Reverse DNS

It's considered proper form that the hostname reported by the MTA match the reverse lookup of the IP address that's establishing the connection.  Most MTAs will suspect messages being sent from a host that doesn't report as the same host it reverse resolves to.  This means that you should also set a reverse DNS record for your email host which matches the hostname you set in Postfix's myhostname parameter.

Unfortunately, the specifics of this are outside the scope of this guide, but usually, involve a call to someone who can modify PTR records for your network(s).  When you're done, dig should report your public IP address like this:

# dig +short -x 173.164.88.234
mx.cospix.net.

Postfix TLS

Postfix supports Transport Layer Security on connections made both to and from it, and can be made to opportunistically encrypt or even require encrypted connections between other software agents.

While this would normally always be a good thing, there may be reasons why enabling TLS is not what you want to do, either because of performance or security concerns, as Postfix TLS is based on OpenSSL.

This section would be rather short if we stopped here though.

Since this guide is focused on sending email from Postfix, this section will only focus on the role Postfix plays as a "client" (connecting to other listening MTAs), and forgo all of the server TLS configuration as described in the original document.  We will be specifically describing a simple configuration of opportunistic DANE Client TLS Authentication, as it's fast becoming nearly ubiquitous for email systems across the public Internet.

It's best not to configure Postfix SMTP client certificates unless one or more servers you connect to require that you present a certificate.  Client certificates are not usually needed and can cause complications in situations that work well without them.

While we could configure Postfix Server-side TLS here, it doesn't make sense to as the only connections this server should see are from localhost or over secured network segments, making TLS a complete waste of time and effort at this time.

Set the following configuration directives and refresh postfix.

# postconf -e smtp_dns_support_level=dnssec \
smtp_tls_security_level=dane
# svcadm refresh postfix

To test this, send another email and verify the existence of encryption data within the received header (not all email services provide this header, but Gmail does).  It should look something like this.

(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);

While it's technically against specification to use anything with more solid security assurances, and DANE appears to be getting solid traction, it still may be worthwhile to read the Postfix guide on Client TLS: DANE.

Message Rate Limiting

Message Rate Limiting allows an email administrator to limit the rate at which messages are sent to local or remote destinations (other MTAs), usually by constraining the number of simultaneous connections to a destination domain or by introducing a delay between each message sent to the same destination domain.  Historically this was used to maintain good performance on heavily used email systems, but it's more common use today is as the primary tool for maintaining good sending reputations for your email server's IP address.

Extremely large ESPs such as Google, Yahoo, and Hotmail/MSN keep track of the number of messages sent by your domain to theirs and will start temporarily rejecting incoming messages if your message volume exceeds a certain base unpublished rate.  This is the first line of defense against the mass of unsolicited messages that batter their email servers.  In addition to message volume, it's theorized that ESPs also track the following:

• Average daily, weekly, and monthly message delivery rates.
• Number of messages that were blocked due to exceeding the base (or modified) rate.
• Number of messages that were addressed to no valid recipients.
• Number of messages that failed their DKIM signature.
• Number of messages that failed SPF.
• Number of messages that were sent from an MTA that reported a different hostname as it reverse resolved to.
• Number of messages that had viruses attached.
• Number of messages that failed heuristic spam detection.
• Number of messages marked as spam by users.

One or more of these data points are used to calculate your IP address' sending reputation, with a better sending reputation allowing the IP address to send messages at a higher than base rate, while a worse sending reputation limits the IP address to send messages at a lower than base rate, which makes it more difficult to change.

The art of "warming up an IP address" for sending email is really just slowly adjusting the destination rate delays (and possibly changing email content to avoid being marked as spam) to manipulate your sending reputation with the ESPs.

Limit small projects too

It may be tempting to ignore setting up message rate limiting if you're starting a small project, instead relying on the natural organic growth of your user base to drive a natural organic growth of your email volume.

Just one high-profile mention of your project has the potential to destroy any sender reputation you may have amassed, preventing emails from being sent at a time where you should be capitalizing on converting potential users.

Warming up new email servers

If you've outgrown your small project network and are moving your project to new email servers, it's considered a best practice to divert an incrementally increasing volume of email to your new servers instead of just jumping in, which would likely cause significant shock to your new IP address sending rates, damaging the as of yet un-warmed sending reputations of your new IP address(es).

Basic Configuration

Set the following configuration directives and refresh postfix.

# postconf -e smtp_destination_recipient_limit=10 \
smtp_destination_rate_delay=300s
# svcadm refresh postfix

The first parameter, smtp_destination_recipient_limit sets the maximum number of recipients per message delivery.  If the number of recipients (of the same domain) in the original message exceeds this number than the original message will be duplicated as many times as required for the remaining recipients.  Increasing this number improves email efficiency, at a possible risk of exceeding the rate limits of the recipient domain.  This number should never be reduced below 2, as it changes the behavior of smtp_destination_rate_delay to act on each recipient instead of on each domain.

The second parameter, smtp_destination_rate_delay determines how much time should pass between individual deliveries to the same destination, or in this case, same domain.  A value of 1s here would limit your email server to 86400 messages in a 24 hour period, or 2.6 million messages a month, which may be fine for a well-established reputation, but a new IP address should start with something much more conservative, such as 300s (which would allow for 288 messages a day, 8640 messages a month) and ramp up, ideally by monitoring your email queue and shaving off fractions of the remaining rate delay based on the queue length.

While 288 messages a day may seem like nothing, consider that this is per receiving domain, so that's 288 messages to gmail.com accounts, 288 messages to yahoo.com accounts, etc, and after just a day you should be able to drop down to 240s and then 180s without raising any alarm.  Just remember to slow down after that, as that the relationship between rate delay and emails per period isn't linear, and that the total email volume approaches infinity as the rate delay approaches zero.

The smtp_destination_rate_delay parameter should probably only be set to zero on an outbound SMTP server if you're using the advanced configuration below, with limits of at least 1s for the major ESPs.  If you need to send more email volume than that, it's best to use multiple sending IP addresses, each with a 1s rate delay.

If you do decide to completely disable rate delays, I recommend doing so only after establishing a long history of using 1s and only if your mail queues never fully empty through a 24 hour period.

Note: Several other guides online recommend tweaking the *_initial_destination_concurrency and *_destination_concurrency_limit parameters.  These parameters limit the maximum number of simultaneous connections and are completely independent of rate delay.  Changing the values of these parameters is completely unnecessary in the context of rate limiting, and can lead to severe overall performance degradation.

Advanced Configuration

Since IP sender reputation is calculated independently by each of the major ESPs, you may find that you're able to send emails at vastly different rates to different ESPs.  Fortunately, you can capitalize on this with Postfix by specifying different recipient limits, rate delays, and concurrency limits on a per domain basis.

We will also use smtp_destination_recipient_limit and smtp_destination_rate_delay as defaults for domains that we haven't explicitly categorized.

Append the following parameters to the end of the named Postfix configuration files:

/opt/local/etc/postfix/master.cf

smtp-aggr unix - - n - - smtp
smtp-nice unix - - n - - smtp

We're defining separate transports to allow us to easily specify different parameters in main.cf.

/opt/local/etc/postfix/transports

gmail.com smtp-aggr:
yahoo.com smtp-nice:
hotmail.com smtp-aggr:

Map each domain to use our non-standard transport.

# postconf -e transport_maps=hash:/opt/local/etc/postfix/transport \
smtp_destination_recipient_limit=20 \
smtp_destination_rate_delay=10s \
smtp-aggr_destination_recipient_limit=100 \
smtp-aggr_destination_rate_delay=1s \
smtp-nice_destination_recipient_limit=10 \
smtp-nice_destination_rate_delay=300s

This configuration allows us to specify a blend of rate delays based on the receiving domain.  Feel free to adjust as you need.

Finally, postmap the transport file and reload postfix.

# postmap /opt/local/etc/postfix/transport
# svcadm refresh postfix

Followup

For a more complete view of what is possible with Postfix rate limiting and performance tuning, I recommend taking a quick look through the official Postfix Performance Tuning document, which is where my configuration recommendations came from.

Additionally, if your interest is in maximizing your reputation with the major ESPs, I recommend reading each of their best practices guides for bulk email senders:

• Google
• Yahoo
• Microsoft
• AOL

Monitoring

I recommend monitoring both internal queue status and external indications of IP sender reputation on your shiny new email server.

Qshape

While mailq is nice if you want to see individual messages, I find qshape invaluable for assessing the health of your outgoing message queues.

# qshape
                 T  5 10 20 40 80 160 320 640 1280 1280+
         TOTAL  20  6  8  5  1  0   0   0   0    0     0
     gmail.com   9  3  1  4  1  0   0   0   0    0     0
     yahoo.com  10  2  7  1  0  0   0   0   0    0     0
   hotmail.com   1  1  0  0  0  0   0   0   0    0     0

Instead of individual messages, qshape displays aggregates: how long a number of messages have been waiting in queue, by default the active and incoming queues.  From this information, we can tell the following:

• We have 9 emails waiting to be sent to gmail.com, of which:
• 3 have been waiting for less than 5 minutes
• 1 has been waiting for 5-10 minutes
• 4 have been waiting for 10-20 minutes
• 1 has been waiting 20-40 minutes
• Based on this information, messages to gmail.com should have their rate delay reduced, as waiting a minimum of 20 minutes for an email is kind of ridiculous today.
• We have 10 emails waiting to be sent to yahoo.com, of which only one has been in queue for 10-20 minutes.  This is much more acceptable but is still a good candidate for the reduction of rate delay.
• We have 1 email waiting to be sent to hotmail.com, which has been waiting in queue for less than 5 minutes.  This should be perfectly acceptable.

Additionally, you can change bucket counts and times changed with the -b and -t options, respectively:

# qshape -b 7 -t 1
              T  1  2  4  8 16 32 32+
       TOTAL  0  0  0  0  0  0  0   0

To get a better idea of what can be done with qshape, I recommend reading Postfix Bottleneck Analysis.

Log Analysis suites

Beyond interactive command line tools, I don't discourage installing a monitoring solution for your email service as well, such as the popular pflogsumm which generates daily reports on the total numbers and bytes of messages received, delivered, forwarded, bounced and rejected, as well as sender and recipient host/domain breakdowns, etc.  An incomplete list of such suites is available on the Postfix Add-on Software page under Logfile Analysis.

Spam Lists and Sender Reputation

I also recommend periodically checking your sending IP address reputation on Sender Score and Sender Base, while this isn't as immediate as grepping your logs for 421 errors, it does give you a general idea of what ESPs think of your IP address.

Archiving

You may have very specific corporate or legal requirements to archive your sent and received email.  After spending several hours researching the specific email retention requirements in the US (and Canada, the UK, EU, etc) I am convinced that no one person can tell you exactly how long you should legally retain email for.  A rough estimate is 2-3 years, but in certain industries, it can be as long as 35 years.  Specific email retention policies should come from people who know what the legal requirements for your company are, and they are likely to change as time goes on.

Additionally, if your organization is currently under subpoena, your email archives are likely under legal hold, which means you can't delete them under after the legal proceedings have passed.

We can facilitate all of this easily in Postfix by appending an email address to the BCC header of all traversing emails, which will send a copy of the email to the specified address.

# postconf -e always_bcc=mailarchive@localhost

If you're using a single SMTP server, I recommend using a local address as messages can be locally archived more efficiently than they can by being sent to a remote archive address online.

If you've already made the jump to multiple SMTP servers, or prefer having a well-structured auditing process, I recommend using an archival solution such as Plier and simply forwarding messages directly to plier, as it yields an overall simpler configuration.  If you are using plier, disregard the remainder of this section.

Storing a local email archive

Since we're doing this on SmartOS, we will want to set up a dedicated ZFS Dataset to store our email archive.  The following commands will create the dataset and set the proper ownership and permissions:

# UUID=`sysinfo | json UUID`
# DIR=/var/mail/archive
# zfs create -o quota=500G zones/$UUID/data/mail/archive
# chown nobody $DIR
# chmod 700 $DIR

Next up, we will configure Postfix to send messages intended for mailarchive@localhost to the mail archive directory for local delivery and refresh postfix to enact our changes.

# echo "mailarchive: $DIR/current" >> /opt/local/etc/postfix/aliases
# newaliases
# svcadm refresh postfix

Archive retention

By specifying a (nonexistent) filename (instead of a directory), we tell Postfix that we want it to store email locally in mbox format instead of maildir format.  Since we are only appending to this file, we can periodically move it to a new filename, compress it, and delete it after it's fallen out of our retention period.  If we are under legal hold, we simply continue to move and compress but refrain from deleting our archives.  For this use-case scenario, mbox is quite a bit more efficient than maildir, however, it is a bit more complicated to deal with (since we're compressing daily blocks of email and automatically deleting old ones without trusting file dates)

This script is available on Github.

/root/scripts/mail_rotate.sh:

#!/bin/sh

# Simple bash script that rotates and compresses mbox formatted mailboxes
# and deletes old ones

HOLD=hold
DAYS=1096
CURRENT=current
DIR=/var/mail/archive
CP=xz
CL=9

# Set the year and date for the file we're working with (yesterday's)
date "+%Y %Y-%m-%d" -d yesterday | read YEAR DATE

# Only move and compress the mbox file if it exists
if [ -f $DIR/$CURRENT ]; then

        # Create the year directory if it doesn't exist
        if [ ! -e $DIR/$YEAR ]; then
                mkdir $DIR/$YEAR
        fi

        # Find a suitable destination filename
        DST=$DIR/$YEAR/$DATE
        if [ -e $DST ] || [ -e $DST.$CP ]; then
                SUFFIX=0
                while [ -e $DST.$SUFFIX ] || [ -e $DST.$SUFFIX.$CP ]; do
                        SUFFIX=`expr $SUFFIX + 1`
                done

                DST=$DST.$SUFFIX
        fi

        # Move and compress the file
        mv $DIR/$CURRENT $DST
        $CP $DST -$CL
fi

# Only delete old archive files if the hold file isn't present
if [ ! -e $DIR/$HOLD ]; then
        # Remove all files with modification times longer than $DAYS
        find $DIR/ -type f -mtime +$DAYS -exec rm "{}" \;

        # Remove all empty directories
        rmdir --ignore-fail-on-non-empty $DIR/[0-9][0-9][0-9][0-9]
fi

Enable execution and include it in the crontab:

# chmod u+x /root/scripts/mail_rotate.sh
# crontab -e

Add the following line to your crontab and save:

0 0 * * * /root/scripts/mail_rotate.sh

Conclusion

I hope that this guide has illustrated to you that while sending email has certainly gotten more involved, it can still be easily set up and managed, just like any other service.  Next month we will be exploring what goes into receiving email.