Which DNS name is checked for TLS on a mailserver?

allo asked:

Let’s say i have these records:

  • A reverse.somedomain:
  • A mail.somedomain:
  • A mail.mailserverdomain:
  • MX somedomain: mail.somedomain
  • PTR reverse.somedomain

A MTA connects mail.somedomain for delivering mail to somedomain and gets a certificate for mail.mailserverdomain presented, and the MTA presents its hostname as mail.mailserverdomain in HELO.

Is this a valid TLS session or would this mean a certificate for a wrong hostname (expecting somedomain or reverse.somedomain)?

My answer:

Surprise! For a TLS negotiation with a mail server, typically the subject name is not checked at all. Much of the encryption between mail servers is entirely opportunistic, using only self-signed certificates, often with a subject name of localhost.

Unfortunately, it’s not really possible to do much better than this, as it is possible to MitM the connection and prevent the encryption negotiation from ever taking place. And it’s not possible to prevent that attack until DNSSEC (or an equivalent) is fully deployed and everyone requires proper certificates of everyone else. This is unlikely to happen before we retire.

If your email must be encrypted end to end, you’ll need to do it yourself.

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.