Load Balancer –> Varnish –> Nginx

ddavtian asked:

Been scratching my head for a few hours now and wanted to see if anyone can help.

1) I have a load balancer with 6 servers in the back end.

2) The back end servers are Nginx and to get the real IP addresses of the visitors, all I have to do is the following in each Nginx install and I am able to get the real client IP address of each visitor.

 set_real_ip_from 192.168.255.0/24; <-- to handle the load balancer IP
 real_ip_header X-Forwarded-For;

3) Now I have installed Varnish in front of each Nginx running on 127.0.0.1 doing caching and for some reason now Nginx doesn’t see the real client Ip addresses anymore coming from LoadBalancer –> Varnish –> Nginx

It’s printing the following:

IP address:
192.168.255.9 <– this should be the real client IP address and not the 192.168 (assuming the load balancer IP address is being printed)

More detailed host address:
192.168.255.9

Many thank if you can help.

Dave

UPDATE:

Without Varnish in the equation, I have the following LB –> NGINX and within NGINX the
following existis

set_real_ip_from  192.168.255.0/24;
real_ip_header X-Forwarded-For;

When NGINX logs the remote_addr, the first entry below prints the real client IP address

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
              '$status $body_bytes_sent "$http_referer" '
              '"$http_user_agent" "$http_x_forwarded_for"';


213.205.234.x - - [05/Sep/2012:09:42:08 -0700] "GET /2011/10/28/chicken-and-apples-in- honey-mustard-sauce/ HTTP/1.1" 200 18283 "-" "Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; GT-I9100 Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30" "213.205.234.x"

With Varnish in the equation LB –> Varnish –> NGINX

And within NGINX I switched the set_real_ip_from to point to 127.0.0.1

set_real_ip_from  127.0.0.1;
real_ip_header X-Forwarded-For;

$remote_addr in NGINX doesn’t print the real client IP address:

192.168.255.9 - - [05/Sep/2012:09:46:41 -0700] "GET /2012/09/03/stuffed-baked-potatoes-deconstructed/ HTTP/1.1" 200 18159 "-" "Mozilla/5.0 (Linux; U; Android 2.2.1; en-us; ADR6400L Build/FRG83D) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1" "69.255.125.x, 192.168.255.9"

As you can see from above the $remote_addr being printed is the Load Balancer’s IP address: 192.168.255.9 instead of the client’s remote_addr. Although the “$http_x_forwarded_for”‘ is printing the correct address I guess: “69.255.125.x, 192.168.255.9”. My goal is to have the $remote_addr hold the correct IP address instead

Thanks
Dave

UPDATE:

Here’s my default.vcl from Varnish, commented out the part mentioned by Shane, here’s the current output from the access log from NGINX

127.0.0.1 - - [05/Sep/2012:11:16:43 -0700] "GET /wp-content/plugins/wp-pagenavi/pagenavi-css.css?ver=2.70 HTTP/1.1" 304 0 "http://mobilefoodblog.com/2011/10/28/chicken-and-apples-in-honey-mustard-sauce/" "Mozilla/5.0 (Linux; U; Android 2.3.5; en-us; SCH-I405 Build/GINGERBREAD) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1" "67.168.209.192, 192.168.255.9"


    # This is a basic VCL configuration file for varnish.  See the vcl(7)
    # man page for details on VCL syntax and semantics.
    # 
    # Default backend definition.  Set this to point to your content
    # server.
    # 

    backend default {
         .host = "localhost";
         .port = "8080";
    }

    sub detect_device {
      # Define the desktop device
      set req.http.X-Device = "desktop";

      if (req.http.User-Agent ~ "iP(hone|od)" || req.http.User-Agent ~ "Android" || req.http.User-Agent ~ "iPad") {
        # Define smartphones and tablets
        set req.http.X-Device = "smart";
      }

      elseif (req.http.User-Agent ~ "SymbianOS" || req.http.User-Agent ~ "^BlackBerry" || req.http.User-Agent ~ "^SonyEricsson" || req.http.User-Agent ~ "^Nokia" || req.http.User-Agent ~ "^SAMSUNG" || req.http.User-Agent ~ "^LG") {
        # Define every other mobile device
        set req.http.X-Device = "other";
      }
    }

    acl purge {
            "localhost";
    }

    sub vcl_recv {
       call detect_device;

         # if (req.restarts == 0) {
         #  if (req.http.x-forwarded-for) {
         #      set req.http.X-Forwarded-For =
         #          req.http.X-Forwarded-For ", " client.ip;
         #  } else {
         #      set req.http.X-Forwarded-For = client.ip;
         #  }
         # }

       if (req.request == "PURGE") {
            if (!client.ip ~ purge) {
                 error 405 "Not allowed.";
            }
            return(lookup);
       }

    if (req.url ~ "^/$") {
          unset req.http.cookie;
        }
    }

    sub vcl_hit {
            if (req.request == "PURGE") {
                    set obj.ttl = 0s;
                    error 200 "Purged.";
            }
    }

    sub vcl_miss {
        if (req.request == "PURGE") {
                    error 404 "Not in cache.";
        }

        if (!(req.url ~ "wp-(login|admin)")) {
                            unset req.http.cookie;
        }

        if (req.url ~ "^/[^?]+.(jpeg|jpg|png|gif|ico|js|css|txt|gz|zip|lzma|bz2|tgz|tbz|html|htm)(?.|)$") {
               unset req.http.cookie;
               set req.url = regsub(req.url, "?.$", "");
        }

        if (req.url ~ "^/$") {
               unset req.http.cookie;
        }
    }

    sub vcl_fetch {
            if (req.url ~ "^/$") {
               unset beresp.http.set-cookie;
            }

        if (!(req.url ~ "wp-(login|admin)")) {
               unset beresp.http.set-cookie;
        }
    }

    sub vcl_hash {
         set req.hash += req.url;
         if (req.http.host) {
             set req.hash += req.http.host;
         } else {
             set req.hash += server.ip;
         }

         # And then add the device to the hash (if its a mobile device)
         if (req.http.X-Device ~ "smart" || req.http.X-Device ~ "other") {
           set req.hash += req.http.X-Device; 
         }

         return (hash);
    }

My answer:


It sounds like your load balancer is not passing the client IP address in any header. Since you didn’t say what you were using for the load balancer, it’s impossible to give a specific solution, but in general you want to configure the load balancer to place the client IP address in one of the headers (such as X-Forwarded-For). You could also designate a custom header for this if you wanted.

If your load balancer is setting X-Forwarded-For, then I’d configure your Varnish servers to set a header other than X-Forwarded-For, so that you have full visibility of both the client’s IP address and which server handled the request.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.