Worm outbreak case study: Incident response in the business with Volatility
Category : Threat Analysis
A few days ago, our team had to respond to a worm outbreak in one of our customer sites. The threat name was Sality which is a well known worm that started propagating back in 2003. The Sality worm spreads by infecting legitimate executable files and copying itself to different location on the local system and on network shares. In this blog, we’re going to cover the steps required to do essential and effective incident response with the Volatility framework.
It’s not uncommon to witness malware that employs polymorphic code to bypass antivirus solutions in the current threat landscape , in this specific case we’re about to cover, the network had a fully updated business grade antivirus solution from, what is considered to be, a leading vendor.
One of the challenges when a mass infection occurs on any network is: “what to do now?” When a mass infection affects a business network, it automatically becomes the incident with the highest priority and comes accompanied a with sense of extreme urgency which affects every single member of the local security team . Triage is mostly immediate with such infections as they’re quite impactful on business assets and become apparent quite quickly (in good cases) – they spread like fire and therefore assume the highest priority.
Back to the original question, we have a worm outbreak on the network so “what to do now?” The local security response team probably started treating and cleaning suspected infections by attempting to contain and eradicate the threat, while other stakeholders may engage in calls with the antivirus vendor and ask for an urgent release of an updated antivirus definition file.
What else can the security team do? The team can make a huge difference by doing some in-house quick and effective forensic analysis in order to understand the threat and assume more control of the situation. Adopting this approach would be in favour to just relying on security products alone or chasing up infections that probably keep popping up without explanation – after all, there’s an urgent need to assume control of the situation as current security controls are failing and can’t be trusted at that point in time. Specifically, acquiring the next set of information about the threat would help the team assume more control of the situation:
- Threat prototyping: define the threat type or find the threat name. The goal of this step is to understand and articulate the risk to the business, contextualise it with existing threat information from online or open-source resources and sharing that information with the business security vendors for faster response.
- Threat impact and indicators of compromise: define what the threat does. The goal here is to obtain tangible threat symptoms and how it impacts a system in the business environment once it infects it.
- Threat confirmation and identification: confirm if a system is clean or infected. The goal is to come up with an independent method to any security control that can allow to confirm if a system is currently infected or clean.
Knowledge is power and it can help the security team gain more confidence and assurance in fighting the infection quicker and successfully. As time is short, It’s ok to expect that at this point a full forensic rundown is not feasible or prove to be very valuable, but forensics findings do need to be sufficient, accurate and with enough quality so they can make a positive impact.
Assume control step by step with Volatility
A great tool that can help achieve quick forensic results and acquire the forensic information illustrated above is the Volatility framework – a tool dedicated to forensic memory analysis.
Let’s take it step by step and show how in practice Volatility can be used for incident response with regards to the Sality worm outbreak on the business network we touch earlier. The outline of the process would be as follows:
- Acquire an infected memory image from an infected machine
- Analyse the infected memory image with Volatility
- Come up with artefacts that cover the information we’d like to achieve
Prerequisite: Installing Volatility and other tools
Volatility is easy to install, in fact it doesn’t require installation for our purpose, you can download a precompiled binary for Windows, Mac and Linux platforms that encompasses all of Volatility’s core functionality (latest version at the time of writing this blog is: 2.5) . In this blog we’ll use the precompiled version for Windows. In addition to Volatility, you’ll need to download and use three additional tools:
- Memory dump utility: DumpIt by MoonSols
- String extraction utility: Strings by Microsoft Sysinternals
- Text editor: any (we like Sublime)
Download all the tools to one designated directory on your analysis machine. However, you will need the MoonSols DumpIt tool available on a portable storage device so you can acquire a memory dump from an infected machine. We highly recommend using a dedicated analysis machine that is detached from the corporate network to analyse the memory image on – although it’s quite safe to analyse raw memory images, artifacts that are extracted from the memory images may be malicious. In addition, if you use MoonSols DumpIt on a portable device we suggest you take under account that the device could get infected.
Step1: Acquire an infected memory image from an infected machine
Using DumpIt is arcsight01.performanta.local, in this case we run the tool on of the machines that are suspected to be infected. All it takes is to double click on dumpit.exe (no need to run from the command line) and then a window opens that indicates where the memory dump is going to get saved, click ‘y’ (yes) and wait until the tool confirms completing dumping memory to a file with ‘Success’. You can now copy the generated memory dump file to your portable storage device.
Step2: Analyse the infected memory image with Volatility
Now it’s time to do some analysis on the memory image with Volatility. First, Volatility needs to know the operating system type the memory image was part of, for that purpose we can use the imageinfo plugin and we’re going to use the -f switch to point it to our memory dump file:
volatility-2.5.standalone.exe imageinfo -f .\WIN-1K1Q65FRITO-20151208-112008.raw
Volatility Foundation Volatility Framework 2.5 Determining profile based on KDBG search... Suggested Profile(s) : Win7SP0x86, Win7SP1x86 AS Layer1 : IA32PagedMemoryPae (Kernel AS) AS Layer2 : FileAddressSpace (C:\Tools\WIN-1K1Q65FRITO-20151208-185807.raw) PAE type : PAE DTB : 0x185000L KDBG : 0x82b80c28L Number of Processors : 1 Image Type (Service Pack) : 1 KPCR for CPU 0 : 0x82b81c00L KUSER_SHARED_DATA : 0xffdf0000L Image date and time : 2015-12-08 18:58:15 UTC+0000 Image local date and time : 2015-12-08 18:58:15 +0000
Next, we can start looking what running processes were in memory. The plugins to use for this purpose can be pstree, pslist and psscan. We can use pstree to see the list of processes in a tree form. Now that we know the profile name we need to use, we can incorporate it too:
volatility-2.5.standalone.exe pstree --profile=Win7SP1x86 -f .\WIN-1K1Q65FRITO-20151208-112008.raw
Volatility Foundation Volatility Framework 2.5 Name Pid PPid Thds Hnds Time -------------------------------------------------- ------ ------ ------ ------ ---- 0x867a1030:wininit.exe 396 316 3 81 2015-12-08 18:52:38 UTC+0000 . 0x86bf0c78:lsm.exe 508 396 10 149 2015-12-08 18:52:38 UTC+0000 . 0x86be3118:services.exe 492 396 7 207 2015-12-08 18:52:38 UTC+0000 .. 0x86cb0740:svchost.exe 896 492 37 1075 2015-12-08 18:52:39 UTC+0000 .. 0x86868498:svchost.exe 3768 492 10 140 2015-12-08 18:54:42 UTC+0000 .. 0x86e9f848:dllhost.exe 272 492 16 201 2015-12-08 18:52:43 UTC+0000 .. 0x86ef5030:TPAutoConnSvc. 1812 492 10 141 2015-12-08 18:52:42 UTC+0000 ... 0x870013e0:TPAutoConnect. 2208 1812 8 125 2015-12-08 18:52:47 UTC+0000 .. 0x86ce3198:svchost.exe 1052 492 10 529 2015-12-08 18:52:39 UTC+0000 .. 0x86fe7030:taskhost.exe 2132 492 11 193 2015-12-08 18:52:47 UTC+0000 .. 0x86c8e788:svchost.exe 808 492 19 456 2015-12-08 18:52:39 UTC+0000 ... 0x86cbe4f8:audiodg.exe 964 808 5 129 2015-12-08 18:52:39 UTC+0000 .. 0x86d94030:spoolsv.exe 1436 492 13 342 2015-12-08 18:52:41 UTC+0000 .. 0x86de6170:svchost.exe 1464 492 18 322 2015-12-08 18:52:41 UTC+0000 .. 0x86f73030:msdtc.exe 956 492 14 152 2015-12-08 18:52:45 UTC+0000 .. 0x86ca28a0:svchost.exe 848 492 18 383 2015-12-08 18:52:39 UTC+0000 ... 0x86fd8428:dwm.exe 2100 848 8 118 2015-12-08 18:52:47 UTC+0000 .. 0x86da3530:svchost.exe 1336 492 15 376 2015-12-08 18:52:41 UTC+0000 .. 0x86c0f298:svchost.exe 724 492 7 282 2015-12-08 18:52:39 UTC+0000 .. 0x85ba9030:sppsvc.exe 3940 492 4 160 2015-12-08 18:54:43 UTC+0000 .. 0x86fbad40:SearchIndexer. 2748 492 13 654 2015-12-08 18:52:53 UTC+0000 .. 0x86cfc5d0:AEEngine.exe 1124 492 9 165 2015-12-08 18:52:39 UTC+0000 .. 0x86c41560:svchost.exe 616 492 12 357 2015-12-08 18:52:39 UTC+0000 ... 0x852eaa58:WmiPrvSE.exe 2080 616 7 130 2015-12-08 18:57:58 UTC+0000 ... 0x86fd7030:WmiPrvSE.exe 3348 616 8 122 2015-12-08 18:53:40 UTC+0000 .. 0x86c5dd40:vmacthlp.exe 680 492 3 53 2015-12-08 18:52:39 UTC+0000 .. 0x86ef7c88:svchost.exe 1784 492 6 90 2015-12-08 18:52:42 UTC+0000 .. 0x86eac638:vmtoolsd.exe 1620 492 9 293 2015-12-08 18:52:42 UTC+0000 . 0x86bee9b0:lsass.exe 500 396 7 573 2015-12-08 18:52:38 UTC+0000 0x861d4270:csrss.exe 336 316 9 434 2015-12-08 18:52:38 UTC+0000 0x86fdad40:explorer.exe 2176 2076 42 939 2015-12-08 18:52:47 UTC+0000 . 0x852b03b8:cmd.exe 2708 2176 1 20 2015-12-08 18:57:20 UTC+0000 .. 0x852c3c08:DumpIt.exe 1796 2708 2 37 2015-12-08 18:58:07 UTC+0000 . 0x86cfd378:coanuuh.exeoan 1868 2176 20 258 2015-12-08 18:56:40 UTC+0000 .. 0x8692c030:duineew.exe 972 1868 5 72 2015-12-08 18:56:42 UTC+0000 . 0x8706f2b8:vmtoolsd.exe 2564 2176 8 256 2015-12-08 18:52:48 UTC+0000 . 0x87015030:Antiexecutable 2556 2176 5 163 2015-12-08 18:52:48 UTC+0000 0x861b0358:csrss.exe 388 380 10 282 2015-12-08 18:52:38 UTC+0000 . 0x853c2a58:conhost.exe 2508 388 2 53 2015-12-08 18:57:20 UTC+0000 . 0x87017460:conhost.exe 2240 388 4 37 2015-12-08 18:52:47 UTC+0000 0x86bb8d40:winlogon.exe 432 380 5 115 2015-12-08 18:52:38 UTC+0000 0x84fad020:System 4 0 85 525 2015-12-08 18:52:37 UTC+0000 . 0x85b1bc68:smss.exe 252 4 2 29 2015-12-08 18:52:37 UTC+0000
There’s a couple of processes that stand out, one is called coanuuh.exe (process id: 1868) and the other duineew (process id: 972). This could be what we’re looking for.
Volatility has more than a few malware plugins that can help investigating memory images that are suspected or infected with malware. One excellent plugin for that purpose is malfind which is designed to look for hidden or injected code/DLLs in user mode memory, based on characteristics such as VAD tag and page permissions.
Let’s give malfind a shot, this time we might want to redirect output to a file as it can be quite long:
volatility-2.5.standalone.exe malfind --profile=Win7SP1x86 -f .\WIN-1K1Q65FRITO-20151208-112008.raw > malfind_out.txt
Process: coanuuh.exeoan Pid: 1868 Address: 0x1d40000 Vad Tag: VadS Protection: PAGE_EXECUTE_READWRITE Flags: CommitCharge: 4238, MemCommit: 1, PrivateMemory: 1, Protection: 6 0x01d40000 4d 5a 90 00 03 00 00 00 04 00 00 00 ff ff 00 00 MZ.............. 0x01d40010 b8 00 00 00 00 00 00 00 40 00 00 00 00 00 00 00 ........@....... 0x01d40020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 0x01d40030 00 00 00 00 00 00 00 00 00 00 00 00 d0 00 00 00 ................ 0x01d40000 4d DEC EBP 0x01d40001 5a POP EDX 0x01d40002 90 NOP 0x01d40003 0003 ADD [EBX], AL 0x01d40005 0000 ADD [EAX], AL 0x01d40007 000400 ADD [EAX+EAX], AL 0x01d4000a 0000 ADD [EAX], AL 0x01d4000c ff DB 0xff 0x01d4000d ff00 INC DWORD [EAX] 0x01d4000f 00b800000000 ADD [EAX+0x0], BH 0x01d40015 0000 ADD [EAX], AL 0x01d40017 004000 ADD [EAX+0x0], AL 0x01d4001a 0000 ADD [EAX], AL 0x01d4001c 0000 ADD [EAX], AL 0x01d4001e 0000 ADD [EAX], AL 0x01d40020 0000 ADD [EAX], AL 0x01d40022 0000 ADD [EAX], AL 0x01d40024 0000 ADD [EAX], AL 0x01d40026 0000 ADD [EAX], AL 0x01d40028 0000 ADD [EAX], AL 0x01d4002a 0000 ADD [EAX], AL 0x01d4002c 0000 ADD [EAX], AL 0x01d4002e 0000 ADD [EAX], AL 0x01d40030 0000 ADD [EAX], AL 0x01d40032 0000 ADD [EAX], AL 0x01d40034 0000 ADD [EAX], AL 0x01d40036 0000 ADD [EAX], AL 0x01d40038 0000 ADD [EAX], AL 0x01d4003a 0000 ADD [EAX], AL 0x01d4003c d000 ROL BYTE [EAX], 0x1 0x01d4003e 0000 ADD [EAX], AL
The results include one of the processes we suspected earlier coanuuh.exe (process id: 1868), the start of the region starts with MZ which is an executable header, in addition the memory region has the PAGE_EXECUTE_READWRITE flag set, which means that this region had probably been rewritten and then executed – one of the signs of polymorphic or decrypted code.
Next we can dump this memory region to disk and have a closer look. The base address of this memory region is 0x1d40000 and it’s possible to use it and the process id we found it was part of (1868) to dump the executable from memory to disk. The command to achieve that is dlldump along with -D switch to redirect Volatility’s output to a desired directory:
volatility-2.5.standalone.exe --profile=Win7SP1x86 -f .\WIN-1K1Q65FRITO-20151208-112008.raw dlldump -p 1868 --base=0x1d40000 -D .\
Volatility Foundation Volatility Framework 2.5 Process(V) Name Module Base Module Name Result ---------- -------------------- ----------- -------------------- ------ 0x86cfd378 coanuuh.exeoan 0x001d40000 UNKNOWN OK: module.1868.3e0fd378.1d40000.dll
There’s one file that was dumped. Uploading the file to Virustotal reveals that it’s most likely a Sality worm variant. At this point, we can also investigate the strings in the file. Remember, as time is critical, we’re interested in fast forensics, we managed to prototype the threat as very probable to be the Sality worm, we’d also like to understand its impact and capabilities . Let’s run strings on the dumped executable and output to a file:
strings .\module.1868.3e0fd378.1d40000.dll > malfind_strings_output.txt
There are many interesting strings in that memory region (see this link for a copy), for e.g. there are references to a lot of security software vendors in what appears to be services names and processes names – it’s likely that the threat attempts to disable security software:
avast! iAVS4 Control Service
avast! Mail Scanner
avast! Web Scanner
avast! Asynchronous Virus Monitor
avast! Self Protection
AVG E-mail Scanner
Avira AntiVir Premium Guard
Avira AntiVir Premium WebGuard
Avira AntiVir Premium MailGuard
COMODO Firewall Pro Sandbox Driver
Eset HTTP Server
Eset Personal Firewall
F-Prot Antivirus Update Monitor
F-Secure Gatekeeper Handler Starter
Outpost Firewall main module
ProtoPort Firewall service
Symantec Core LC
Symantec Password Validation
Furthermore, we can find strings that suggest impact on Microsoft Security Center handles notifications and Windows task manager:
And lastly we can find references to these URLs and also a user-agent:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:22.214.171.124) Gecko/20100401 Firefox/3.6.1 (.NET CLR 3.5.30731)
Step3: target a string in the infected memory images and verify infections
So far, we managed to prototype the threat and understand from looking at strings what the impact of the threat is. We can now use this information to quickly identify if a machine is infected with the threat.
It’s mostly true for production environments where it’s preferred that critical services or servers to not be wiped clean and restored in case of an infection. There’s a need to verify with high confidence infections or suspected infections in order to start containment, eradication and recovery – especially if security software fails to detect threats that impacted the environment.
One way to achieve that verification rather quickly is to take a memory dump from a suspected or infected machine and search if it hold certain strings that we found the threat uses. In the case of Sality, we could choose a string we believe would be unique and check if it exists in the memory dump we’d like to scrutinise, for e.g., let’s look for part of this URL that we found referenced earlier in Sality’s memory: info/home.gif . For that purpose we can use Moonsols DumpIt, Strings and the built in FindStr command in Windows to look for it with a regular expression (“/R” switch to look for a regular expression) .
strings .\WIN-1K1Q65FRITO-20151208-112008.raw | FindStr /R \.info/home\.gif
.info/home.gifI888 .info/home.gifI888 http://kukutrustnet777.info/home.gif http://kukutrustnet888.info/home.gif http://kukutrustnet987.info/home.gif
In this case we could verify that the memory image is infected.
In this blog we showed how analysing memory with Volatility can help prototype and identify a threat, glean indicators of compromise from it and verify if a machine is infected . Volatility can help obtain some crucial information that can help any organisation to assume control and conduct forensic analysis in order to respond to security incident, whether urgent or not, effectively.