serving mp3s to mobile devices is flooding nginx with partial requests

drumfire asked:

I am serving mp3s with a minimalistic nginx server. What I see in my log files is that there are a lot of requests, in particular from AppleCoreMedia and sometimes Android useragents, that flood the server with short requests. Sometimes they keep requesting to download the same partial content for a very long time; sometimes more than an hour. For example:

"GET /somefile.mp3 HTTP/1.1" 206 33041 "AppleCoreMedia/ (iPhone; U; CPU OS 5_1_1 like Mac OS X; en_us)"
"GET /somefile.mp3 HTTP/1.1" 206 33041 "AppleCoreMedia/ (iPhone; U; CPU OS 5_1_1 like Mac OS X; en_us)"
"GET /somefile.mp3 HTTP/1.1" 206 33041 "AppleCoreMedia/ (iPhone; U; CPU OS 5_1_1 like Mac OS X; en_us)"

I also get a lot, but not as much, of these:

"-" 400 0 "-" 
"-" 400 0 "-"

The IP addresses are always from clients that start downloading shortly after that request, usually they have roughly the same UserAgent as in the first example.
emphasized text
I have enabled server throttling and connection limits in nginx to limit the huge amount of log entries from equivalent IPs at least somewhat.

There was a performance issue when I saw the same behaviour on the previous server that used Apache. I installed nginx on a better server then moved the site. When Apache could not handle more connections from the increasing number of clients effectively that server was ddossed. There was no bandwidth issue with already connected clients and I don’t know if the already connected clients were using more than one connection at a time.

Please tell me:

  • Are clients that appear to get stuck on a download a Bad Thing™

I heard people say their mobile bandwidth use was much higher than they could account for. I’m thinking this type of client behaviour can account for that. And costs us more bandwidth too.

  • Which up to date alternatives exist out there that can handle serving this type of data better than plain HTTP?
  • Useful general insights for someone who just came into this field straight out of the late 90s. 🙂

My answer:

Second issue first:

"-" 400 0 "-"

This generally means that the client opened a Keep-Alive connection, but then disconnected after receiving a response to its previous request. Since the web server was expecting something, it logs a bad request (HTTP 400). These are mostly harmless.

Now, what is the problem with iPhones downloading partial content? Is it just the log files being large? You can turn off logging for specific requests using nginx directives. If it’s causing a performance problem, or if the clients aren’t able to access the resources, then maybe you should start worrying. But you didn’t explicitly mention either of these.

(All that said, that’s strange behavior from an iPhone; it wastes both network resources and battery life, things mobile devices have in short supply.)

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.