Find whats using all my servers bandwidth

Benno asked:

TLDR: how can I dig deeper into my Mac mini OSX server to determine what processes are consuming so much bandwidth, or where all the inbound traffic is coming from?

Resource caching is on, all resources are minified or images crushed, our pages are less bandwidth consuming then the website last month (www.vulytrampolines.com), and our traffic in analytics is the same.

Back story: We have 2x mac mini servers running our website/staging/databases etc. Since moving from a dedicated server in America to a colocation place in the city, our bandwidth consumption has somehow quadrupled. There is a database replication process setup between the two, as well as DNS and various websites (e.g. large files, database tasks, intranet packages etc are on staging server, website and database are on production server)

Our staging server has had 7GB of inbound traffic in 3 DAYS. Does anyone know how to check for inbound traffic sources to see potentially where this consistent stream of inbound traffic at a 200k connection is coming from? We have no idea. We aren’t sending it files at all, the only thing that should be active is SSH and a database replication process. netstat [see below] shows we have about 20+ established and 30+ close_weight connections to port 625. We haven’t the foggiest how this could be happening.

The annoying thing is, webstats show we haven’t used anywhere near 11.66GB for HTTP traffic (it says we used 22GB in the last month, yet our outbound traffic was recorded as over 100GB). Database stats say we haven’t used anywhere near enough bandwidth to cause the issue either.

This is our staging server, venus1 (it has been like this for a few weeks):
enter image description here

This is our production server, venus2:
enter image description here

netstat -anp tcp output on venus1. Most of the foreign addresses are from our work IP addresses. 11211 is memcached.

tcp4       0      0  122.99.117.18.49712    204.93.223.143.80      ESTABLISHED
tcp4       0      0  122.99.117.18.11211    122.99.117.18.49711    ESTABLISHED
tcp4       0      0  122.99.117.18.49711    122.99.117.18.11211    ESTABLISHED
tcp4       0     52  122.99.117.18.22       59.167.152.67.56106    ESTABLISHED
tcp4       0      0  122.99.117.18.11211    122.99.117.18.49588    ESTABLISHED
tcp4       0      0  122.99.117.18.49588    122.99.117.18.11211    ESTABLISHED
tcp4       0      0  122.99.117.18.49410    122.99.117.19.3306     ESTABLISHED
tcp4       0      0  122.99.117.18.5432     122.99.117.18.58704    ESTABLISHED
tcp4       0      0  122.99.117.18.58704    122.99.117.18.5432     ESTABLISHED
tcp4       0      0  127.0.0.1.58699        *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      110.142.234.238.62055  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.71.206.61838    ESTABLISHED
tcp4       5      0  122.99.117.18.625      203.206.171.34.61741   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.61270  ESTABLISHED
tcp4       0      0  127.0.0.1.54           *.*                    LISTEN
tcp4       0      0  122.99.117.18.53       *.*                    LISTEN
tcp4       0      0  127.0.0.1.53           *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      110.142.234.238.63980  ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.55282   ESTABLISHED
tcp46      0      0  *.80                   *.*                    LISTEN
tcp4       0      0  *.*                    *.*                    CLOSED
tcp46      0      0  *.443                  *.*                    LISTEN
tcp4       0      0  *.*                    *.*                    CLOSED
tcp4       0      0  122.99.117.18.625      122.99.117.19.50766    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.63981    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.60214  ESTABLISHED
tcp4       0      0  122.99.117.18.625      27.33.32.204.65196     ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.60274  ESTABLISHED
tcp4       0      0  122.99.117.18.625      122.99.117.19.53201    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.59662   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.49869    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.53827    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.64678  ESTABLISHED
tcp4       0      0  122.99.117.18.625      122.99.117.19.52810    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.62510  ESTABLISHED
tcp4       0      0  122.99.117.18.625      122.99.117.19.49909    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.56096  ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.53247   ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.62051   ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.93.92.59123     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.57173    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.49352   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.64362  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.59772     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.59528    CLOSE_WAIT
tcp4       0      0  *.3306                 *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      27.33.32.204.56812     ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.52987  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.50598    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.63339  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.63283  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.61312    ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.52045  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      122.99.117.19.49172    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.50501   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.93.92.56042     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.55882     ESTABLISHED
tcp4       0      0  122.99.117.18.311      58.111.93.92.55875     ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.58776   ESTABLISHED
tcp6       0      0  *.5432                 *.*                    LISTEN
tcp4       0      0  *.5432                 *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      58.111.93.92.52692     ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.57121   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.54673     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.53915     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.52109     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.51807     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.65049     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.93.92.64442     ESTABLISHED
tcp4       0      0  122.99.117.18.311      203.206.171.34.51628   ESTABLISHED
tcp4       0      0  122.99.117.18.625      203.206.171.34.51594   ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.62597     ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.62454     ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.58088     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.57305     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.53724     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.62224     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.62064     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.58236     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.51320   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.51297   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      203.206.171.34.50864   CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.49800  CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      27.33.32.204.50894     ESTABLISHED
tcp4       0      0  122.99.117.18.625      59.167.152.67.50411    ESTABLISHED
tcp4       0      0  122.99.117.18.625      27.33.32.204.54446     ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.51680     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.60797     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.60729     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.49209    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.63371     CLOSE_WAIT
tcp4      81      0  122.99.117.18.625      113.128.44.66.3842     CLOSE_WAIT
tcp4      11      0  122.99.117.18.625      213.57.189.27.55646    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      110.142.234.238.53655  ESTABLISHED
tcp4       0      0  122.99.117.18.625      110.142.234.238.53644  ESTABLISHED
tcp4       0      0  122.99.117.18.625      58.111.79.42.52146     CLOSE_WAIT
tcp4       0      0  127.0.0.1.8005         *.*                    LISTEN
tcp46      0      0  *.8009                 *.*                    LISTEN
tcp46      0      0  *.8080                 *.*                    LISTEN
tcp4       0      0  122.99.117.18.625      58.111.79.42.50716     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.49872    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.63218     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.62471     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.64758     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.64646     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56788     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56770     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56017     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.53131     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.52519     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.51215     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.51131     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.57058    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      59.167.152.67.56711    CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.50975     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.57209     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.54753     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.60786     CLOSE_WAIT
tcp4       0      0  122.99.117.18.625      58.111.79.42.56174     CLOSE_WAIT
tcp4       0      0  *.11212                *.*                    LISTEN
tcp6       0      0  *.11212                *.*                    LISTEN
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49167        ESTABLISHED
tcp4       0      0  127.0.0.1.49167        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  122.99.117.18.5218     122.99.117.18.49166    ESTABLISHED
tcp4       0      0  122.99.117.18.49166    122.99.117.18.5218     ESTABLISHED
tcp46      0      0  *.5268                 *.*                    LISTEN
tcp46      0      0  *.5218                 *.*                    LISTEN
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49163        ESTABLISHED
tcp4       0      0  127.0.0.1.49163        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49162        ESTABLISHED
tcp4       0      0  127.0.0.1.49162        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  127.0.0.1.5348         127.0.0.1.49161        ESTABLISHED
tcp4       0      0  127.0.0.1.49161        127.0.0.1.5348         ESTABLISHED
tcp4       0      0  127.0.0.1.5348         *.*                    LISTEN
tcp4       0      0  *.11211                *.*                    LISTEN
tcp6       0      0  *.11211                *.*                    LISTEN
tcp4       0      0  *.88                   *.*                    LISTEN
tcp6       0      0  *.88                   *.*                    LISTEN
tcp6       0      0  *.2000                 *.*                    LISTEN
tcp4       0      0  *.2000                 *.*                    LISTEN
tcp6       0      0  *.4190                 *.*                    LISTEN
tcp4       0      0  *.4190                 *.*                    LISTEN
tcp4       0      0  *.464                  *.*                    LISTEN
tcp6       0      0  *.464                  *.*                    LISTEN
tcp6       0      0  *.25                   *.*                    LISTEN
tcp4       0      0  *.25                   *.*                    LISTEN
tcp4       0      0  *.749                  *.*                    LISTEN
tcp6       0      0  *.749                  *.*                    LISTEN
tcp4       0      0  *.22                   *.*                    LISTEN
tcp6       0      0  *.22                   *.*                    LISTEN
tcp4       0      0  *.5900                 *.*                    LISTEN
tcp6       0      0  *.5900                 *.*                    LISTEN
tcp4       0      0  *.625                  *.*                    LISTEN
tcp6       0      0  *.625                  *.*                    LISTEN
tcp4       0      0  127.0.0.1.631          *.*                    LISTEN
tcp6       0      0  ::1.631                *.*                    LISTEN
tcp4       0      0  *.311                  *.*                    LISTEN

My answer:


Based on the close correlation between the traffic stats on the staging and production servers, it appears your two servers are talking to each other.

Port 625 is a management port used by Apple products. From your netstat, it appears that one machine (.19) is connecting to the other (.18) on port 625. From some Google searches, it appears that Mac OS X uses port 625 for Workgroup Manager (which was replaced in Mountain Lion with Profile Manager).

It also appears that a wide variety of other machines on the Internet are connecting to port 625 as well, probably as break-in attempts. You should firewall your server as soon as possible to prevent intrusion.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.