Making Sure Your Email Gets Delivered

In an age where somewhere between 80% and 97% of all e-mail delivery attempts are attempting to deliver a message which is unwanted by the recipient, it is hardly surprising that most e-mail service providers have implemented various stringent methods of keeping as much of this spew as possible out of their users’ inboxes.

Unfortunately, since no automated means of determining whether a message is “spam” can be 100% accurate, the filtering that providers perform can result in legitimate mail (defined here as any e-mail that the recipient wants to receive) being improperly classified—with consequences ranging from delayed e-mail to outright rejection.

This guide is intended to detail some of the methods that our customers have found effective in making their e-mail look less like spam, which increases their chances of getting their e-mail delivered. The various approaches we recommend are not alternatives; we generally recommend a multi-pronged deployment of multiple approaches (possibly sequentially, to measure the effectiveness of each), to maximize your chances of success. In some cases, however, some approaches may not be appropriate for a particular customer.

Hopefully it goes without saying that no e-mail should be sent to any person who has not affirmatively registered their willingness to accept the e-mails they are sent. We treat spam complaints very seriously, and we will not hesitate to block the e-mail, or terminate the service, of any customer who sends e-mail that the recipient does not wish to receive.

Using a Dedicated IP

Odds are, you’re using our default outbound e-mail cluster (your relay SMTP server is smtp.eyXX.engineyard.com, or just smtp) for your outbound mail server. As we cover in more detail elsewhere , this cluster is shared by all the customers on that cluster, with most customers using a single source IP address for their e-mail deliveries.

Having lots of e-mail going out over a single IP address doesn’t pose any deliverability limitations in and of itself, but since not every customer is completely attentive to their e-mail “hygiene”, in rare cases one of these shared servers will get put on somebody’s naughty list. When that happens, mail from your slices to that destination may be delayed or rejected.

We’ve crafted a rather neat solution that allows you to utilize our shared mail systems while retaining your own dedicated sending IP. This insulates you (and your sending reputation) from being associated with other customers’ e-mail practices.

What this means, in simple terms, is that you can be sure that whatever problems you’re having delivering e-mail is caused by your own actions, and not some other customer. This is an important first step in making sure that your e-mail gets delivered, as once you know that the problem is with your own e-mail, it’s easier to make changes for the better.

We recommend that anyone who is sending a large amount of mail, or who needs to be maximally certain that their e-mail reaches it’s recipient, should send out through a dedicated IP allocated for their sole use.

If you would like to have a dedicated IP, please lodge a support ticket and we’ll get it setup for you.

Sender Protection Framework (SPF)

One of the things that spammers routinely do is forge other people’s addresses as the sender of their spam. This distracts recipients from the true source of the spam, ensures that the spammers don’t get walloped by the bounces that come back, and generally makes them even more obnoxious.

SPF is a method of telling the world where e-mail is expected to legitimately come from for a domain, so that recipients can filter out e-mail that fails these checks, and reduce the impact of forged From lines.

In and of itself, SPF is not actually an anti-spam measure—a spammer can just as easily configure SPF records for a throw-away domain they use in their sender headers. However, SPF records are seen by some providers as prima facie evidence of legitimacy, and it can certainly reduce the chances of a spammer degrading the positive reputation of your sending domain, and it lowers the amount of “backscatter” that you’ll receive (bounces from e-mail that had your e-mail address as the sender).

SPF isn’t a solution for any customer that sends e-mail from arbitrary domains, however we generally don’t recommend completely obscuring the source of your e-mail. You should put the address you wish to “appear” to come from as the From address, and then put your own “bounce handling” address (see Bounce Handling, below) in the Sender header.

At a technical level, SPF is a DNS record that lists the hosts and addresses that are allowed to send e-mail for a given domain. MTAs can use this information to “sanity check” that an e-mail with a certain sender address is coming from an expected place, and not somewhere else.

The receiving MTA takes the domain of the sender address, prefixes it with _spf. and looks up the TXT data associated with that FQDN. For example, if an e-mail was received claiming to be from “freddy@example.com”, the name looked up would be _spf.example.com.

There’s a special syntax that is used in the returned data to describe the places that e-mail should be coming from. The full syntax of the SPF records isn’t really important here.

In general, if you’re routing your e-mail out through our shared e-mail infrastructure, and haven’t had a dedicated IP address allocated, then you’ll want to put the following into your SPF record: re include:_spf.eyNN.engineyard.com

Where “eyNN” is the cluster ID of the cluster you’re on.

If you’ve got your own dedicated IP, then you’ll want to instead add something like this: re ip4:A.B.C.D

Where “A.B.C.D” is the IP address that you’ve been allocated for your outgoing e-mail.

If you’re transitioning between the shared outgoing IP and a dedicated one, you can always put both records in, and if you’ve got other places that e-mail can come from for your domain (such as your local ISP, or a webmail provider) then you should ask them what to add to your SPF record so that the mail coming from those other locations doesn’t get flagged as being a problem.

If we host your DNS, all you have to do is lodge a ticket and we’ll update your DNS zone to suit your needs. Otherwise, use the management interface provided by your DNS hosting provider (or edit the zone files if you host it yourself) to include the appropriate record.

DomainKeys

DomainKeys is a different way to solve approximately the same problem as SPF—that of sender forgery. It also shares some of the same deficiencies, in that it doesn’t directly stop spam, and that it doesn’t work if you’re sending e-mail from arbitrary domains. The most high-profile mail recipient that uses DomainKeys is Yahoo! Mail, and if you’re having any problems delivering to Yahoo!, we usually recommend implementing DomainKeys as a first step.

Like SPF, DomainKeys is also based around using DNS, but it works quite differently in the details. DomainKeys has the added property of “signing” several fields in the e-mail, to prevent modification in-transit or at the destination.

With DomainKeys, the sending e-mail server holds a private key that it uses to hash certain headers in outgoing e-mail, while the public portion of the key is published in a special DNS record. The calculated hash is placed in a special header in the outgoing e-mail.

Receiving mail clients and MTAs can then lookup your public key and use it to verify the hash. This provides 2 benefits, first the recipient knows the message was sent by an authorized server for your domain (otherwise the sending MTA wouldn’t know the private key to calculate the correct signature) and second that the signed parts of the message were not altered in transit.

The exact implementation of DomainKeys can be a bit tricky; the best option is to lodge a support ticket and our staff will get things underway. If we host your DNS, we can do everything for you; otherwise, we will give you a public key which you will need to add to your domain by whatever means is appropriate.

Content Analysis

One of the most common (and most powerful) means of detecting spam is to do analyze the content and headers of an e-mail to try and find patterns of behavior that have a high probability of occurring in spam but a low probability of occurring in legitimate e-mail.

Unfortunately, many providers use proprietary means of analyzing the content of e-mails, reasoning that if spammers know how content is analyzed, they will be able to reformat their spam to get around it.

However, there is a Free Software product, SpamAssassin, which incorporates a lot of the same checks as most proprietary content analysis systems. If your e-mail gets a low SpamAssassin score, then it’s likely that you’ll pass through most proprietary content filters without too much hassle.

Rather than setup a local copy of SpamAssassin just to check your e-mail, we recommend using http://www.brandonchecketts.com/emailtest.php. The added bonus of this particular service is that it also checks that your SPF and DomainKeys records are correctly configured, which is handy.

There’s not a lot of general assistance we can provide with analyzing the content of the e-mails you send. However, if you’re stuck with any part of the process, please raise a support ticket and we’ll do whatever we can.

Sender Address Validity

It’s a really, really simple thing, but it’s amazing how often it’s missed in the excitement of rolling out a new application:

Make sure that the sender address of your e-mails will accept return e-mail.

This is crucially important, for two reasons:

• You need to have somewhere to accumulate bounce messages (see below);
• Many e-mail providers will check that the sender address is valid before accepting your e-mail, on the (reasonable) assumption that an address which doesn’t want to accept a return e-mail is far, far more likely to be dodgy (and the provider won’t be able to send a bounce anywhere useful if something goes wrong later).

Yep, it’s a simple thing to do, and it’s an equally simple thing to forget. Don’t be one of the forgetters.

Bounce Handling

A common trait of spammers is that they tend to be quite persistent, attempting to send the same e-mail to the same recipient dozens of times, or randomly “guessing” usernames in the hope that eventually they’ll pick a valid user and get an e-mail sent. This may seem pointless, but it’s amazing what becomes feasible when you’ve got a few thousand zombie machines which can just try random combinations for days on end…

As a result of all this, the big e-mail providers frown upon mail senders who send a lot of e-mail that bounces. One or two “nobody’s home!” responses won’t get you in any trouble, but when you consistently (and persistently) send e-mail to invalid addresses, your IP address is entirely likely to end up being blocked for extended periods of time.

How do you prevent this? Well, there are a number of things that need to be done:

• Validate e-mail addresses as you receive them. You should be doing this as a matter of course, to ensure that you’re not sending e-mail to an address that doesn’t belong to the person who entered the address into your application (known as Confirmed opt-in), but you can do a number of simpler checks for syntactic correctness and apparent deliverability at the time the address is first entered, so you can give immediate feedback if the user hasn’t entered their address correctly.
• Ensure that your e-mail has a valid sender address.
• Process the bounce e-mails that accumulate in the mailbox of your sending address. Unfortunately, what “processing bounces” means differs between applications, depending on why you’re sending the e-mail in the first place. If you were sending out invites, blacklisting the e-mail address from being sent any further e-mails (for a reasonable time period, like a month, if not permanently). On the other hand, if you’re sending some sort of newsletter, then a bounce means that the recipient no longer exists, and should be removed from your database of recipients.

• We can help you make sure that your bounces have somewhere to go with our incoming e-mail servers, just put in a ticket.

Further Reading

• An Introduction to SPF
• DomainKeys primer