Ben Holness asked:
I run a server that sends emails through smtp.mailgun.org. I locked down the server to only allow outgoing SMTP on port 587 to smtp.mailgun.org.
The IP address for this domain recently changed and as a result my system stopped sending emails. As part of trying to figure out what had stopped working, I became aware of the downsides of using domain names in iptables rules as well as the fact that iptables does a lookup when you run the rule and saves the IP address at the time the rule is created, thus IP address changes in domains are not reflected in iptables.
From mailgun’s status page, they say
Our infrastructure upgrades will result in more frequent changes to these IP addresses in the future and customers should ensure that they are only using our supported hostnames when connecting to the Mailgun service.
I was not notified of an IP address change, and I do not know if I will be notified of future changes.
What’s the best way to manage this situation? Should I write a script that regularly checks the IP address for smtp.mailgun.org and updates iptables when it changes? (I am aware of the security risks in doing this, but perhaps they are worth taking).
A quick look at the current IP addresses for smtp.mailgun.org reveals that they’re all on Amazon AWS. You ought to expect these to change frequently, and so hard coding them in the firewall becomes tricky at best. You may well lose outgoing mail even with a dynamic script, as the IP address may change and you try to send mail before the next run of the script. This is called a race condition. There’s not much you can do about it except redesign, and that’s what I’ll recommend…
As for egress firewalling, I really don’t have much concern about outgoing traffic to port 587. All such traffic has to authenticate to the remote mail server anyway, so it’s not a significant concern for me. For mail I worry about outgoing traffic to port 25; if you’re using a service like mailgun, there should be no such traffic, and if there is, it’s almost certainly because you’ve been compromised.
Some things you can consider instead are:
Allow any outgoing TCP traffic to destination port 587. As explained above, this is fairly low risk, but you can reduce the risk further by…
Run your web app under a unique user ID, and then allow any outgoing TCP traffic to destination port 587 for only that user ID. (Use the
-m ownermatch.) This reduces the risk even further, by stipulating that only the web app’s user can initiate such traffic. Note that if you do that, other users won’t be able to make this outgoing connection, not even root (but root can alter the firewall rules to make it possible).
You can make your setup more robust by running a local-only mail server configured to use mailgun as a smarthost, relaying all mail it receives (as it only listens locally, this should only come from your web app). Then you allow only the mail server’s user ID to make outgoing connections to port 587 as above. This gives you the additional benefit of not losing mail because mailgun was unreachable at the time your web app tried to send mail. If mailgun is unreachable, either due to network issues or misconfigured firewall or any other reason, your web app will log an error and lose the mail, but the local mail server will queue the mail and deliver it when service is restored. You then configure the web app to deliver locally.
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.