I am maintaining a server to send email mailings (no spam, of course).
The current speed of this machine is approx. 2000 emails per hour.
(edit: actually I did a test mailing today, removing the throttling and sending a mailing to 2500 recent + active subscribers. It took approx. 1 hour and 45 minutes to get this mailed out.)
My boss pointed out that he is not satisfied with that, as he saw companies like Mail Chimp and the like where you can send thousands of emails within a couple of seconds / minutes. And they do go out, of course, as you get immediate replies, opens, etc.
My question is, what exactly is needed to achieve such a sending speed?
I mean, of course you can add to the hardware and build a more and more complex system of servers that all send out your mail, etc. And of course it is also a matter of having a clean list (no unknown hosts, etc. as they all eat up the resources of the server)
But aside from that, I am sure there must be other ways to improve this. Anyone can give some overview on this?
As requested in the comments, here are some more specifics on what hardware is being used as well as sending behavior:
Type of server
Operating system: CentOS Linux 5.11 Kernel and CPU: Linux 2.6.18-400.1.1.el5 on i686 Processor: Intel Core2 Duo CPU E7500 @ 2.93GHz, 2 cores CPU load averages: 1.07 (1 min) 1.18 (5 mins) 0.65 (15 mins) CPU usage: 4% user, 1% kernel, 56% IO, 38% idle Real memory: 1.49 GB used, 1.94 GB total Virtual memory: 1.13 GB used, 3.91 GB total Local disk space: 55.10 GB used, 219.71 GB total
Postfix version 2.3.3
Average size of emails
Of the recent mailings, the largest one I found was just below 20k. On average I can say it's probably between 8k and 10k per message.
hdparm -t /dev/sda /dev/sda: Timing buffered disk reads: 336 MB in 3.01 seconds = 111.64 MB/sec dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync 16384+0 records in 16384+0 records out 1073741824 bytes (1.1 GB) copied, 23.9512 seconds, 44.8 MB/s
Measurements of various metrics
CPU load - see above disk time - ??? RAM usage - see above bandwidth usage - below data from "iftop -n" over a time period of ca. 5 minutes while sending mail. TX: cum: 3.43MB peak: 1.16Mb rates: 5.36Kb 99.5Kb 137Kb RX: 1.01MB 120Kb 2.06Kb 38.0Kb 32.3Kb TOTAL: 4.44MB 1.28Mb 7.42Kb 137Kb 169Kb
Some maillog data:
in case this is of value, here are a couple of lines from the maillog: Mar 4 14:00:32 mailserver postfix/smtp: 6C419107802A: to=<firstname.lastname@example.org>, relay=mx.example.com[184.108.40.206]:25, delay=1.6, delays=0.05/0/0.14/1.4, dsn=2.0.0, status=sent (250 OK id=1YT8ud-0004fe-Rn) Mar 4 14:00:32 mailserver postfix/qmgr: 6C419107802A: removed Mar 4 14:00:32 mailserver amavis: (26392-01-46) lookup (score_sender), 1 matches for "email@example.com", results: "."=>[Amavis::Lookup::RE=ARRAY(0xaa7f358),HASH(0xac5f891)] Mar 4 14:00:32 mailserver amavis: (26392-01-46) lookup_re("firstname.lastname@example.org"), no matches Mar 4 14:00:32 mailserver amavis: (26392-01-46) query_keys: email@example.com, myself@, mailserver.com, .mailserver.com, .com, . Mar 4 14:00:32 mailserver amavis: (26392-01-46) lookup_hash(firstname.lastname@example.org), no matches Mar 4 14:00:32 mailserver amavis: (26392-01-46) lookup (score_sender<email@example.com>) => undef, "firstname.lastname@example.org" does not match Mar 4 14:00:32 mailserver amavis: (26392-01-46) SpamControl: calling spam scanner Mar 4 14:00:32 mailserver amavis: (26392-01-46) spam_scan: DSPAM not available, skipping it Mar 4 14:00:32 mailserver amavis: (26392-01-46) timer set to 320 s for SA (was 480 s) Mar 4 14:00:32 mailserver amavis: (26392-01-46) calling SA parse, SA version 3.2.5 Mar 4 14:00:33 mailserver amavis: (26392-01-46) CALLING SA check Mar 4 14:00:33 mailserver postfix/smtp: A5341207802D: to=<email@example.com>, relay=mx.example.com[220.127.116.11]:25, delay=1.7, delays=0.15/0/0.14/1.4, dsn=2.0.0, status=sent (250 OK id=1YT8ue-0005BY-5x) Mar 4 14:00:33 mailserver postfix/qmgr: A5341107802D: removed Mar 4 14:00:34 mailserver postfix/smtp: C30371078144: to=<firstname.lastname@example.org>, relay=mx.example.com[18.104.22.168]:25, delay=1.8, delays=0.05/0/0.13/1.6, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued on mx.example.com as 1425474034-NECyl5JAc9-0Xx8wjwN)
Running a mailing list on a postfix server myself, I expect to see 2,000 messages get processed at least once (i.e. they might be deferred) before you can finish making a cup of coffee.
Your system appears to be running spam and virus scans on outgoing mail. While it’s a good idea to scan incoming mail, it’s probably not so good an idea to scan outgoing mail, especially if the source of that mail is already well-controlled. We can see that this is adding several seconds to the delivery of each message, and that it is also driving up your disk I/O dramatically.
I would reconfigure Postfix to stop scanning outgoing mail. If you have a need to scan outgoing mail normally, e.g. for the people in your office sending mail from their computers, then set up a dedicated mail server specifically for handling the outbound mailing list traffic.
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.