CentOS – Nginx logs got deleted – unrecognized system restart, why? How to prevent?

Pikk asked:

I am using CentOS Linux release 7.0.1406 (Core).

The last time I logged in to SSH of the server was on April 20. Everything was working fine.

Today I logged in once again to check if anything new in the error.log of my websites. I do it periodically. But today there was a surprise:

[root@myserver nginx]# ls -la
total 104840
drwx------ 2 nginx nginx     4096 Apr 30 03:19 .
drwxr-xr-x 7 root  root      4096 May  3 03:20 ..
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 access.log
-rw-r--r-- 1 root  root  17956729 Apr 30 03:19 access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 awstats.site1.net.access.log
-rw-r--r-- 1 root  root      5229 Apr  2 14:21 awstats.site1.net.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 awstats.site1.net.error.log
-rw-r--r-- 1 root  root      4654 Apr  2 14:21 awstats.site1.net.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site1.net.access.log
-rw-r--r-- 1 root  root     26082 Apr 20 21:12 devel.site1.net.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site1.net.error.log
-rw-r--r-- 1 root  root     46743 Apr 20 21:14 devel.site1.net.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site2.pl.access.log
-rw-r--r-- 1 root  root      1652 Apr 24 06:28 devel.site2.pl.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 devel.site2.pl.error.log
-rw-r--r-- 1 root  root       237 Feb 28 21:32 devel.site2.pl.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 error.log
-rw-r--r-- 1 root  root    596623 Apr 30 02:38 error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site1.net.access.log
-rw-r--r-- 1 root  root  83764451 Apr 30 03:18 site1.net.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site1.net.error.log
-rw-r--r-- 1 root  root    147462 Apr 29 21:36 site1.net.error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site3.com-access.log
-rw-r--r-- 1 root  root    177285 Apr 30 03:14 site3.com-access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site3.com-error.log
-rw-r--r-- 1 root  root     27929 Apr 28 23:16 site3.com-error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 panel.site4.com-access.log
-rw-r--r-- 1 root  root      1963 Apr 25 22:22 panel.site4.com-access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 panel.site4.com-error.log
-rw-r--r-- 1 root  root       488 Apr 13 14:21 panel.site4.com-error.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site2.pl.access.log
-rw-r--r-- 1 root  root   4485845 Apr 30 03:12 site2.pl.access.log-20150430.gz
-rw-r--r-- 1 web   nginx        0 Apr 30 03:19 site2.pl.error.log
-rw-r--r-- 1 root  root     61613 Apr 30 01:36 site2.pl.error.log-20150430.gz

As you can see, the .log files were 0KB!!! But there was a plenty of data there. It just… flew away.

I also noticed that with last, there was a strange reboot I was not aware of:

reboot   system boot  2.6.32-042stab08 Wed Apr 29 20:41 - 15:09 (8+18:27)

Now I changed back the owner/group to nginx and it looks like the logs are once again populating.

EDIT:

Here is my nginx.conf:
user web;
worker_processes 2;
pid /var/run/nginx.pid;

events {
        worker_connections 768;
        multi_accept on;
}

http {
        rewrite_log off;
        ##
        # Basic Settings
        ##
        client_max_body_size 20m;
        sendfile off;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

#       log_format  main  '$remote_addr $host $remote_user [$time_local] "$request" '
#                          '$status $body_bytes_sent "$http_referer" "$request_time"';

        log_format main     '$remote_addr - $remote_user [$time_local] "$request" '
                            '$status $body_bytes_sent "$http_referer" '
                            '"$http_user_agent" "$http_x_forwarded_for"';

        include /etc/nginx/mime.types;
        default_type application/octet-stream;



        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;

        fastcgi_buffer_size  16k;
        fastcgi_buffers      16  16k;
        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log main;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##


        gzip             on;
        gzip_disable "msie6";
        gzip_min_length  1000;
        gzip_proxied     expired no-cache no-store private auth;
        gzip_types       text/plain application/xml text/css text/js text/xml application/x-javascript text/javascript application/json application/xml+rss;

        ##
        # nginx-naxsi config
        ##
        # Uncomment it if you installed nginx-naxsi
        ##
        #include /etc/nginx/naxsi_core.rules;

        ##
        # nginx-passenger config
        ##
        # Uncomment it if you installed nginx-passenger
        ##

        #passenger_root /usr;
        #passenger_ruby /usr/bin/ruby;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

Here is the output of: ps axu | grep log

root        86  0.0  0.0  34636   848 ?        Ss   Apr29   0:05 /usr/lib/systemd/systemd-logind
root       541  0.0  0.0   9512   588 ?        S    Apr29   0:01 dovecot/log
mysql      593  0.6  5.7 1675596 179496 ?      Sl   Apr29  89:36 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
root     30100  0.0  0.0   8988   900 pts/1    S+   20:31   0:00 grep --color=auto log

I have few questions:

  1. I don’t remember if there were .gz files. But now there are. How/where can I check if there is some rule somewhere that says that it should gzip each logfile?
  2. What do you think happened? Is there anything else I can check to find the root cause of that issue
  3. Is there a way to prevent such things happening in future?
  4. Is there a way to recover the logs that disappeared?

My answer:


Here is the problem:

user web;

Nginx expects to run as user nginx, and all the associated scripts like logrotate expect this as well. This should never be changed unless you know exactly what you’re doing and every possible implication. Revert it to the default:

user nginx;

View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.