The Case of the Missing Emails

If you attempted to reach me using the address associated with this website between mid-June and mid-September of this year — including responding to an email I sent, or using the contact form — it is likely that I never received your message. If you were fortunate you may have received an error letting you know your communication did not go through, but if you use Gmail you may have been blocked from seeing any failure notice for the same reason you were blocked from seeing my email. This post explains what happened and why, so you can make sure it doesn’t happen to you.

Toward the end of summer, in the middle of an otherwise forgettable day, in the middle of an otherwise forgettable week, I decided to send a few otherwise forgettable emails from the Ditchwalk email account. Sending emails from this site is not something that transpires on a daily basis, but from time to time a thought takes hold and what can a site owner do? I refuse to be silenced by myself.

As usual I logged into the web application for my Ditchwalk email account, then composed and sent several emails one after the other. While looking through the inbox, however, I noticed that it had been quite a while since I received any incoming mail. And by quite a while I was surprised that it had actually been several months — or had it?

Try as I might I could not remember if I had received any recent emails which I subsequently deleted. Checking the trash and spam folders also didn’t help because both were empty. And the fact that they were empty didn’t help because I may have emptied them myself, instead of waiting for them to be emptied by my email provider.

Wracking my mind I tried to recall the specifics of any recent emails. Complicating matters was the fact that I did remembered getting emails at other addresses, including other addresses tangentially related to this site. I also remembered getting recent emails about this site at other addresses, but try as I might I could not remember getting any emails at the specific address associated with this website, and the more I thought about it the more that seemed a little weird.

The reason it did not seem a lot weird — and thus galvanize my attention — was because I hadn’t been posting much lately, and as a consequence I hadn’t been sending out many emails from my Ditchwalk account. I did dash off a random now and then, and looking through my sent folder reminded me of those messages, but few if any of them required a response from the recipients, so it wasn’t like I was expecting any specific replies. Still, the more I thought about it the more uncomfortable I became not because I didn’t have recent message in my inbox, but because I didn’t have any recent messages in my spam folder. Real people get busy and have better things to do than reply to my crackpot emails, but sending out incessant spam is pretty much the only thing spammers do.

While it might have been the case that I had simply fallen off the major spam lists, or my email provider was doing a sterling job of catching spam before it ever got to my account, the fact that I couldn’t specifically remember getting any recent spam — or deleting the contents of my spam folder — seemed suspicious. Having experienced email glitches in the past, however, I knew that the diagnostic process was exceedingly simple. Open up both the email account for Ditchwalk and a separate, reliable account using another host, then send test messages back and forth to see what was or was not getting through.

And so I did exactly that, sending one test email from Ditchwalk to an active Gmail address, and an email from that Gmail address to the Ditchwalk email account. And then I waited. And waited. And…waited.

Without a Trace
We have all had email messages fail to arrive at a recipient’s address, or an anticipated message fail to arrive at our own address. In almost every instance, however, the problem is not with the email system itself, but the result of an erroneous, bad or mistyped address. On those rare occasions when there is a problem with the underlying networking, we also expect that notification will be made in the form of a server-level message announcing that a sent message cannot be delivered.

What I did not anticipate when I sent my test messages to the same two addresses was that neither would ever arrive at their intended destination, yet no error messages would be triggered. How was that even possible? After following up myself on every explanation I could think of — including checking spam folders to make sure the messages were not being improperly flagged — I was completely stumped, and finally reached out to the tech support people at my email provider.

In short order the tech support contact spotted a misconfiguration in one of the domain records for my website, and the appropriate changes were made. The specific file in question was called the MX record, where ‘MX’ stands for ‘mail exchanger’ — which was just the problem I was having. At that point I simply had to wait for those updated records to be propagated across the web, which would take anywhere from a few hours to a day.

In order to help me track and visualize the propagation process, the tech support contact pointed me to this site, where I could input my site name, select the appropriate record, then ping a list of sites around the world. Sure enough, in less than an hour several of the servers in the United States reported the update, and shortly thereafter I received an incoming email from someone I had contacted in my latest batch of outgoing messages. (Fortuitously, that person also took the time to note that they had been getting error messages on their end in the recent past, thus confirming my suspicions that the problem was on my end.)

After obsessively and pointlessly checking the propagation progress throughout the day, I went to bed assuming the problem was solved and everything would be functional when I woke. If only!

The Gmail Gorilla
Although propagation had yet to take place on a few far-flung web servers, the next morning I dutifully sent new test emails to and from the address in question. While I was able to send and receive at non-Gmail addresses, however, and I was able to receive messages from Gmail addresses, Gmail still refused to accept messages from the address associated with this site. The good news was that Gmail was now throwing an error message on failed delivery. The bad news was that no matter how I tried I couldn’t understand what the message was telling me.

Once again I contacted tech support for my email provider and reported my confusing findings. After again digging into my DNS records the tech support contact spotted another misconfiguration in what is called a TXT file, which in this case related to the MX record that was fixed the day before. Specifically and crucially, some email handlers — including monsters like Google — require that an SPF record accompany the DNS information for each email sender. And to be clear that’s a good thing, because it helps combat spam not only at Gmail but across the internet.

Precisely because Gmail is the internet’s email gorilla, however, if you are not compliant with Gmail’s requirements you are going to have a hard time reaching anyone who uses Google’s email services. Sure enough, when I checked the then-current value for my SPF record it was incorrect, and with tech support’s help I made the requisite change. (You can see the current SPF settings for Ditchwalk here.) After a few more hours the new change began taking effect and I was able to send emails to first one, then other Gmail addresses. What I still did not understand was how those DNS records became misconfigured in the first place, but the Gmail error messages I had been unable to interpret ultimately told the tale.

The Ghost in the Machine
By the time I realized that something was amiss with the email address associated with this site, I had forgotten that earlier in the summer — sometime in early June — I had contacted my internet service provider (ISP) to report random email error messages when attempting to contact a few small websites. In response, the tech support contact at my ISP checked my DNS records, identified the problem, then updated those records. And that would have worked, except….

What I have not yet mentioned is that unlike most small site owners, I do not use the email service provided by my ISP. Instead, I use a combination email provider and registrar to keep track of my website registrations, and that was important for two reasons. First, when your email and site registration is hosted by a different company than your ISP, you effectively have two different groups of DNS records, but only one sets of files is active. And at my email provider I had seen the message telling me that multiple times while trying to diagnose my problems, but it never quite sank in:

This domain is using third-party nameservers. DNS records added here won’t have an effect.

Second, when I originally contacted my ISP during the summer — which was the right move on my part — they did attempt to fix the records in question. Unfortunately, what I failed to tell them is that they weren’t actually my email provider. Meaning they fixed my TXT/SPF file as if they were providing my hosting, instead of providing the value for my actual host.

Only after the MX record and the TXT/SPF records were properly configured at my ISP was the problem resolved. Unfortunately, while both errors were in effect, every email sent from my Ditchwalk address had no validity at any Gmail address, and emails sent to my Ditchwalk address disappeared into a void because the MX record was incorrect. (When everything was resolved a few emails pre-dating my fixes did arrive in my inbox, because they finally had a proper address to reach.)

Testing Testing
If you’re savvy about DNS records some of the above explanation may not make sense, but that’s only because some of it still doesn’t make sense to me. What I do know is that any messages that were sent to the email address associated with my site during a three-month span were simply lost, and they’re not coming back. (If you received an error message when trying to contact me, feel free to re-send if you still have the outgoing message in your sent folder.)

I consider myself reasonably savvy about tech, but it’s still incredibly easy to find myself out of my depth on relatively mundane matters. That said, the fault for this protracted email disruption ultimately rests with me because nothing prevented me from including testing while making sure everything is properly updated. Going forward I intend to regularly test not only the email address for this site, but others as well, because even when you think everything is stable things might change.

— Mark Barrett