The following is a sanitized version of the rough notes I took while investigating a researcher's Linux machine that had been compromised. It actually happened earlier this month, I'm just now taking the time to semi-formally write it up.
1 - Setting the Stage
2 - Preliminaries
3 - Investigation / Cleanup
4 - Wrapup / Lessons Learned
1 - Setting the Stage
Our campus security officer contacted us with a complaint about several spam web sites being offered from one of our machines. (Ironically enough, I didn't see the original note from him, as he included one of the URLs being offered and my SpamAssassin rules clobbered it.)
2 - Preliminaries
The machine was a Dell PE2850 running SuSE 10.0 on a Xen kernel. I had done the initial installation, including a Xen 3.0 kernel, and handed it over to the researchers. It had been in production for six weeks.
Since the complaint was about website hosting, another staff member cut off outbound HTTP traffic from the machine at our firewall. As well, he cut off all incoming traffic from off-campus. We verified that the machine was listening on port 80, and was indeed serving out ads for certain pharmaceuticals.
3 - Investigation / Cleanup
I logged in to the system with our admin user and started poking around to see what was running. lsof showed a pair of suspicious processs:
tswapd 7468 root 3u IPv4 1018066 TCP hostname:http (LISTEN)
tswapd 7469 root 3u IPv4 1018066 TCP hostname:http (LISTEN)
As far as I knew, there's no legitimate tswapd user on a SuSE box, and we knew that the machine wasn't supposed to be serving out web traffic. There was no mention of tswapd in any of the passwd, group, or shadow files.
I wanted to see who'd logged in on the system, so I used last:
hostname /etc# last
last: /var/log/wtmp: No such file or directory
Perhaps this file was removed by the operator to prevent logging last info.
Being fairly certain that the operator had not done such, I poked around some more - lastlog was there though.
Looking in more detail at lsof:
tswapd 7468 root txt REG 8,9 435472 243050 /bin/tswapd (deleted)
tswapd 7469 root txt REG 8,9 435472 243050 /bin/tswapd (deleted)
tswapd 7469 root mem REG 0,0 0 [heap] (stat: No such file or directory)
So, looking for recently-modified files:
hostname \# find / -mtime -3 > ~adminuser/c/last_mod_mtime-3
Found lots of stuff in "/tmp/...":
/tmp/.../88.240.47.64.good
/tmp/.../68.164.206.39.good
/tmp/.../88.240.47.64.last
and so on - 628 files worth.
I was still a bit mystified as to how the intruder had gained root access, as they must have if they'd wiped out the last login records. So, I copied off the shadow file and ran john the ripper against it. Within seconds, I had my answer: the root password was "root". (I definitely hadn't handed over the system with that password.) It also coughed up the password of the grad student user a few minutes later.
Checking back in /tmp, I saw that it was empty now of all but visits from me and my boss's boss (who also reads our security mail).
I tried running strace against the process listening on port 80, and I found that if I connected to port 80 on the machine and did nothing (telnet host 80), as soon as I attached to the process with strace it would kill my telnet connection. I also got 404s if I connected with a web browser while strace was attached, and if I straced with following forks:
host 7468/fd\# strace -f -p 7468
as soon as I connect to port 80 on the host:
host 7468/fd\# strace -f -p 7468
Process 7468 attached - interrupt to quit
\[ Process PID=7468 runs in 32 bit mode. \]
accept(-90761248899069, 0xffffbe84ffffada8, NULL) = 4
fork() = -1 ENOSYS (Function not
implemented)
time(\[1146519565\]) = 1146519565
open("/etc/localtime", O\_RDONLY) = 5
fstat64(0x5, 0xffff8980) = 0
fstat64(0x5, 0xffff8860) = 0
old\_mmap(0x2000000000000, 146028888067,
PROT\_READ\|PROT\_WRITE\|PROT\_EXEC\|PROT\_SEM\|
PROT\_GROWSDOWN\|PROT\_GROWSUP\|0xfcfffff0, 0x9 /\* MAP\_??? \*/\|MAP\_DENYWRITE, 0,
0x30
00081a4) = 0x55555000
read(5, "TZif\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\\4\\0\\0\\0\\4\\0"..., 131072)
= 1
252
close(5) = 0
munmap(0x55555000, 131072) = 0
stat64(0x80ab925, 0xffff8864) = 0
stat64(0x80ab925, 0xffff8714) = 0
stat64(0x80ab925, 0xffff8714) = 0
getpid() = 7468
socket(PF\_FILE, 0 /\* SOCK\_??? \*/, 4294958725) = 5
fcntl64(5, F\_SETFD, FD\_CLOEXEC) = 0
connect(579644319502172165, ptrace: umoven: Input/output error
{...}, 4294958725) = 0
send(579826529194737669, ptrace: umoven: Input/output error
0x400000000049, 0, 0) = 73
exit\_group(1) = ?
Process 7468 detached
host 7468/fd\#
And the the 7468 process croaked.
I got some information out of the environment of the process (linewraps added):
host /proc/7469\# cat environ
SHELL=/bin/bash
SSH\_CLIENT=81.169.132.166 44411 22
USER=root
LS\_COLORS=
MAIL=/var/mail/root
PATH=/usr/bin:/bin:/usr/sbin:/sbin
PWD=/root
SHLVL=1
HOME=/root
LS\_OPTIONS=-a -N --color=none -T 0
LOGNAME=root
SSH\_CONNECTION=81.169.132.166 44411 (host ip) 22
\_=/bin/tswapdhost /proc/7469\#
host /proc/7469\#
I also took a copy of exe in the same directory:
-rwxr-xr-x 1 root root 435472 2006-05-01 17:49 exe
There was some interesting output from examining the output of "strings exe":
/usr/sbin/iptables
204.15.80.0/22
216.168.0.0/19
216.223.192.0/19
198.209.0.0/16
67.160.189.137
216.127.64.0/19
194.159.72.0/23
207.57.2.84
194.109.217.0/24
85.65.0.0/16
194.90.8.0/24
63.134.192.0/18
/etc/iptables-dump
--dport
DROP
INPUT
filter
DEBUG: blocked %s (%s)
/etc/block-%s
/etc/images-vs-html
IP address for target is %s
%s/%s
%s -t filter -D INPUT -j REJECT -p tcp --reject-with=tcp-reset -s %s
%s -t filter -D INPUT -j DROP -s %s
%s -t filter -A INPUT -j REJECT -p tcp --reject-with=tcp-reset -s %s
%s -t filter -A INPUT -j DROP -s %s
I haven't done any further analysis on the executable, both from lack of time and knowledge.
4 - Wrapup / Lessons Learned
The machine is back on-net, having been formatted and reinstalled. Grad student admin has been counselled on proper passwords.
We should consider blocking inbound ssh to machines in this server's class from off-campus, unless otherwise requested.
We should consider implementing a password checking policy on all machines, not just those we directly administer, with the co-operation of the researchers.
We need to know more about reverse-assembling Linux binaries.
I need to whitelist known abuse contacts, I never know what I might be sent.