Can't delete corrupt files on Linux

quickshiftin asked:

So I got hit by a script kitte… Fortunately the box is Ubuntu and was able to replace w/ binaries from a comparable system, however,

Some of the files I couldn’t delete, and am still stumped on this. The hijacked files are sitting in the /_bin directory which is writeable by root.

nathan@db-0:~$ ls -ld !$
ls -ld /_bin
drwxr-xr-x 2 root root 4096 Mar 12 18:00 /_bin

Ok, those are the perms on the directory, now for the files within:

nathan@db-0:~$ ls -l /_bin
total 268
-rwxr-xr-x 1 root root  39696 Nov 19 22:25 ls
-rwxr-xr-x 1 root root 119800 Mar 31  2012 netstat
-rwxr-xr-x 1 root root 101240 Dec 12  2011 ps

Now when I try to delete one of these files (as root):

root@db-0:/home/nathan# rm /_bin/ls
rm: cannot remove `/_bin/ls': Operation not permitted

Or if I try to delete the entire _bin directory (again as root):

root@db-0:/home/nathan# rm -rf /_bin
rm: cannot remove `/_bin/ls': Operation not permitted
rm: cannot remove `/_bin/netstat': Operation not permitted
rm: cannot remove `/_bin/ps': Operation not permitted

So how can I delete these files?

Edit:

Sure enough the immutable bit has been set, however, removing it does not let me delete the files.

root@db-0:/home/nathan# lsattr /_bin
s---ia--------- /_bin/ls
s---ia--------- /_bin/netstat
s---ia--------- /_bin/ps

root@db-0:/home/nathan# chattr -R -i /_bin
root@db-0:/home/nathan# lsattr /_bin
s----a--------- /_bin/ls
s----a--------- /_bin/netstat
s----a--------- /_bin/ps

root@db-0:/home/nathan# rm -rf /_bin
rm: cannot remove `/_bin/ls': Operation not permitted
rm: cannot remove `/_bin/netstat': Operation not permitted
rm: cannot remove `/_bin/ps': Operation not permitted

Also verified /_bin doesn’t have immutable bit:

root@db-0:/home/nathan# lsattr -d /_bin
--------------- /_bin

My answer:


Most likely the attacker has set the immutable attribute on the files and directory. This is commonly done by rootkits to make cleanup more difficult.

To confirm this, try:

lsattr /_bin

To remove the immutable attribute, use:

chattr -R -i /_bin

You’ll also want to clear the a and s attributes, since these may affect your ability to remove the files.

chattr -R -i -a -s /_bin

See the chattr man page for a full explanation of what all the attributes are and what they do.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.