Using Pintools to Detect Bugs

Recently I spent some time to get into Pin and to explore how feasible to write Pin-based tools to detect programming errors. Here is the summary of my experiment. I think it could be useful for someone who might want to write Pintools.

I had been thinking of dynamic ways to catch programming error without making the detection logic complex. My decision was to write a tool that can detect division by zero errors when the division is performed by a function argument. The tool works as follows.

The tool inspects division instructions that are reading stack via EBP.

   if (INS_Opcode(ins) == XED_ICLASS_IDIV || INS_Opcode(ins) == XED_ICLASS_DIV)
   {
[...]
      if (INS_IsStackRead(ins) && INS_RegRContain(ins, REG_EBP) && INS_IsMemoryRead(ins))
      {
         INS_InsertCall(ins, IPOINT_BEFORE, (AFUNPTR)TrackDivisionByArg, IARG_INST_PTR, IARG_MEMORYREAD_EA, IARG_REG_VALUE, REG_EBP, IARG_END);
      }

Since the function arguments are stored at EBP+XX it's possible to detect when a division is performed by a function argument.

VOID TrackDivisionByArg(ADDRINT inst_ptr, ADDRINT memoryread_ea, ADDRINT reg_ebp)
{
   // Do we read argument of the function (EBP+XX)?
   if (memoryread_ea >= reg_ebp)
   {
[...]

However, this doesn't mean there is a division by zero error because there might be a test for zero. To filter out obvious false positives the tool inspects if the parameter is accessed before the division.

   else if ((INS_Opcode(ins) == XED_ICLASS_CMP || INS_Opcode(ins) == XED_ICLASS_MOV) && INS_IsStackRead(ins) && INS_RegRContain(ins, REG_EBP) && INS_OperandIsImmediate(ins, 1))
   {
      INS_InsertCall(ins, IPOINT_BEFORE, (AFUNPTR)TrackAccessToArgs, IARG_INST_PTR, IARG_MEMORYREAD_EA, IARG_REG_VALUE, REG_EBP, IARG_REG_VALUE, REG_ESP, IARG_END);
   }

The tool checks the functions on isolation so if the function parameter is checked for zero by the caller the program may report division by zero error. This is the consequence of the design.

The tool uses std::map data structure to maintain the state of the analysis. It also contains other research implementation that is not mentioned in this blog post. The source code is available to download here.

If you use std::map functions you have to use mutex and write lock to protect the data structure from potential corruption unless the target uses no threads.

To get better performance it's better to do analysis once the application exists rather than on-the-fly. However, this increases the memory footprint as the data can be accumulating during the execution.

Sometimes it might be a good idea to use Window Beep function to make noise if a bug is detected.

While it's possible to write efficient Pintools to detect bugs, sometimes if the tool is made to depend on the target application it can perform better.

Sample Pintools for Visual Studio

Creating a Visual Studio Project for Pintools (32-bit)

• Start Visual Studio 2010 and create a new Win32 Project.
• In the Win32 Application Wizard, select DLL for Application type, and tick Empty project in the Additional options.
• Create directory Pin in the Solution folder and copy Pin framework here.
• Copy Pin\source\tools\ManualExamples\inscount0.cpp to the Project folder. This is a sample Pintools file. Add the copied inscount0.cpp to the project.
• Set Active Configuration to Release.
• Add the following to the Additional Include Directories:

$(SolutionDir)Pin\source\include\pin;$(SolutionDir)Pin\source\include\pin\gen;$(SolutionDir)Pin\extras\xed2-ia32\include;$(SolutionDir)Pin\extras\components\include;%(AdditionalIncludeDirectories)

• Add the following to the Additional Library Directories:

$(SolutionDir)Pin\ia32\lib;$(SolutionDir)Pin\ia32\lib-ext;$(SolutionDir)Pin\extras\xed2-ia32\lib;%(AdditionalLibraryDirectories)

• Add the following to the Preprocessor:

TARGET_WINDOWS
TARGET_IA32
HOST_IA32

• Set Enable C++ Exceptions to No, and set Runtime Library to Multi-threaded (/MT).
• Add the following to the Additional Dependencies in the Linker settings:

ntdll-32.lib
libxed.lib
pin.lib
pinvm.lib
libcmt.lib
libcpmt.lib

• Set Ignore All Default Libraries to Yes (/NODEFAULTLIB).
• Set Entry Point to Ptrace_DllMainCRTStartup@12.
• Add the following to the Additional Options of the Command Line linker option:

/EXPORT:main

Downloading&Building Sample Pintools for Visual Studio

• To download Sample Pintools Project created with the above settings click here [SkyDrive].
• Open the solution file in Visual Studio and set the active configuration to Release.
• Create directory Pin in the Solution folder and copy Pin framework here.
• Copy Pin\source\tools\ManualExamples\inscount0.cpp to the Project folder. This is a sample Pintools file.
• Build the project.

Tracing Thread ID of ModLoad and ModFree Events

This log was created by an experimental Windbg extension that is to trace the thread ID of ModLoad/ModFree events. When a module is being unloaded the extension queries the ID of the current thread and compares to the thread ID that loaded the module. If two different threads used to load and to unload the module the extension issues a notification as seen in yellow below.

ModLoad 59440000 00000454          PRNFLDR
ModFree 59440000 00000454 00000454 PRNFLDR
ModLoad 59440000 00000454          prnfldr
ModLoad 74b90000 00000454          WINSPOOL
ModLoad 5aa30000 00000454          prncache
ModLoad 74950000 00000454          RpcRtRemote
ModLoad 673a0000 00000454          actxprxy
ModLoad 6b180000 00000454          ieproxy
ModLoad 5a9f0000 00000454          thumbcache
ModLoad 5fcc0000 00000454          ieframe
ModLoad 75800000 00000454          api-ms-win-downlevel-ole32-l1-1-0
ModLoad 74490000 00000454          api-ms-win-downlevel-shlwapi-l2-1-0
ModLoad 74470000 00000454          api-ms-win-downlevel-advapi32-l2-1-0
ModLoad 6b250000 00000454          api-ms-win-downlevel-shell32-l1-1-0
ModFree 5aa30000 0000055c 00000454 prncache
-->prncache (5aa30000) is allocated by thread id 454 but freed by 55c
ModLoad 55250000 00000454          NPSWF32_11_7_700_169
ModLoad 771d0000 00000454          urlmon
ModLoad 6c310000 00000454          DSOUND
ModLoad 744a0000 00000454          POWRPROF
ModLoad 735a0000 00000454          mlang

I observed that normally the same thread is responsible to load and to unload the module. However, that's not always the case. If you can force a context switch on the code path of dereference, and to get the unloading thread to trigger ModFree, you could end up to dereference freed memory.

Browsers Could Enable to Plant Malware

Today, Opera released a patch which attempts to address one of the issue I reported them in February, 2013.

The most likely attack is that a remote attacker may trick a user to visit a specially crafted web page; and to perform undisclosed operation (social engineering). As a result, a malicious executable file may be planted on the user's computer.

Either social engineering or a second bug is required to plant malware, so it's reasonable to assign moderate severity to the issue.

Another, probably less likely attack vector includes that an attacker has physical access to the machine. When the file downloads are blocked by security policy it may be easily possible for the attacker to bypass it and plant malware. This is a risk, specially in a corporate environment.

Firefox (Bug 845880), Chrome (Issue 177980), and Safari (Follow-up: 260250675) are affected.

Internet Explorer 10 is not affected. I concluded they recognized this issue internally and implemented defense.

I haven't verified Opera's patch yet.

Technical description scheduled to be posted later this year.

Integer Overflow and Memory Failures in WavPack

The common consequences of the integer overflow are denial-of-service and out-of-bounds memory access. What I'm describing here is neither of them but rather a rare consequence I discovered recently.

WavPack is an open source project providing audio compression and decompression. The project consists of the library and utilities exercising the library. The library compresses and decompresses Wav and WavPack streams. It's used in several software (e.g. WinZip) and hardware. One of the utilities called WVUNPACK decompresses the WavPack file.

I'm discussing a bug in WVUNPACK that is not shipped with the 3rd party products using WavPack library, and so the bug doesn't affect 3rd party products but WVUNPACK only.

WVUNPACK has an undocumented command line switch k that allows the user to control the size of the buffer to operate with. This switch expects a number to calculate the size of the buffer with.

case 'K': case 'k':
    outbuf_k = strtol (++*argv, argv, 10);

outbuf_k is int, and its value is controlled by the user. Assume the user calls WVUNPACK specifying -k4194303 in the command line. The integer after k is copied to outbuf_k. Note, 4194303 is 0x3fffff.

The program calculates the size of the buffer to operate with. However, there is an integer overflow when calculating output_buffer_size and it becomes 0xfffffc00 after the execution of the code snippet below.

if (outbuf_k)
    output_buffer_size = outbuf_k * 1024;

The program attempts to allocate memory with 0xfffffc00. The allocation fails, and the returning pointer that is NULL is not sanity-checked.

output_pointer = output_buffer = malloc (output_buffer_size);

Without detecting the memory allocation failure, the execution continues and the decompression is starting by creating an output file and writing a header in it.

if (!DoWriteFile (outfile, WavpackGetWrapperData (wpc), WavpackGetWrapperBytes (wpc), &bcount) ||

The program unpacks the content of the file to a temporary buffer.

samples_unpacked = WavpackUnpackSamples (wpc, temp_buffer, samples_to_unpack);

However, the block within the if statement is not reached because output_buffer is NULL, and so the decompressed data is not written to the output.

if (output_buffer) {
[...]
        if (!DoWriteFile (outfile, output_buffer, (uint32_t)(output_pointer - output_buffer), &bcount) ||

To recap the issue, the integer overflow causes that the unpacked data is not written to output and there is no error displayed believing the unpacking is successfully completed.

The screenshot below demonstrates there is no error displayed but when manually checking the file size there is a mismatch.

In addition to that, WVUNPACK supports to calculate and to display MD5 signature to verify the output. This can be enabled by m command line switch. This check is performed on the temporary buffer that is never written to output, therefore the error remains undetected. In fact, the program could display the correct MD5 while the output has different checksum.

Here is the code snippet demonstrates MD5 calculation is performed on temporary buffer.

if (calc_md5 && samples_unpacked) {
[...]
    MD5Update (&md5_context, (unsigned char *) temp_buffer, bps * samples_unpacked * num_channels);

And here is the screenshot demonstrates the bug in checksum verification.

The bug described above is found in WVUNPACK, however, I'd like to provide information about the library, too, for those use it in 3rd party products. According to WavPack website the library is used in several hardware including jukeboxes, multimedia and network players, and in many software including VLC Media Player.

WavPack library, probably for performance reasons, doesn't check the return value of memory allocation functions. This looks safe when investigating the library on isolation as those seem to work with small buffers and so it's difficult to make the allocation to fail, and to possibly enter in vulnerable paths. However, the library is widespread and used in different systems, and in different software environment, it could even possibly run in browser process. Earlier this year, I proved how to make allocation fails with fixed or small size.

Few examples could access near NULL:

orig_data = malloc (sizeof (f32) * ((flags & MONO_DATA) ? sample_count : sample_count * 2));
memcpy (orig_data, buffer, sizeof (f32) * ((flags & MONO_DATA) ? sample_count : sample_count * 2)); 

[...]

wps->blockbuff = malloc (wps->wphdr.ckSize + 8);
memcpy (wps->blockbuff, &wps->wphdr, sizeof (WavpackHeader));

[...]

riffhdr = wrapper_buff = malloc (wrapper_size);
memcpy (wrapper_buff, WavpackGetWrapperLocation (first_block, NULL), wrapper_size);

Flash Player Unloading Vulnerability

Opera unloads Flash Player after two minutes of not being used. Flash Player can create dialog that can run even when the player itself is unloaded. When it's unloaded, the running dialog makes a call back to the unloaded module, in a time window.

If an attacker can exploit the time window between the unload and the dereference, it's possible to redirect the execution flow.

While experimenting, I used heap spray to inject 0xCC bytes to the deleted memory. I succeeded once of many attempts and this suggests it's practically possible to create an exploit that could inject malicious code within the time window.

When investigated this vulnerability, I wanted evidence to the question: what module unloads the Flash Player? I used Windbg to place breakpoint at unload events. I let the application to trigger the unload events, and when the debugger hit the breakpoint, printed the call stack.

0:014> sxe ud
0:014> g
Unload module C:\Windows\SysWOW64\Macromed\Flash\NPSWF32_11_4_402_287.dll at 54760000
eax=00000000 ebx=035f5584 ecx=00000000 edx=00000000 esi=035f5bc8 edi=00000000
eip=77b1fc72 esp=0037dab0 ebp=0037db30 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00200246
ntdll!ZwUnmapViewOfSection+0x12:
77b1fc72 83c404          add     esp,4
0:000> kc
ntdll!ZwUnmapViewOfSection
ntdll!LdrpUnloadDll
ntdll!LdrUnloadDll
KERNELBASE!FreeLibrary
WARNING: Stack unwind information not available. Following frames may be wrong.
Opera_534b0000!OpSetSpawnPath
Opera_534b0000!OpWaitFileWasPresent

I saw the Flash Player was freed by Opera indeed.

I needed the answer for the question also: what module created the thread that calls back to the unloaded module? Here are the series of Windbg commands used to get the answer.

sxn et
sxn ud
bp CreateRemoteThreadEx "kc; gu; !handle eax 8; gc"
g

KERNELBASE!CreateRemoteThreadEx
kernel32!CreateThreadStub
WARNING: Stack unwind information not available. Following frames may be wrong.
NPSWF32_11_4_402_287!native_ShockwaveFlash_TCallLabel
Opera_534b0000!OpGetNextUninstallFile
Opera_534b0000!OpGetNextUninstallFile
NPSWF32_11_4_402_287!NP_Shutdown
NPSWF32_11_4_402_287
NPSWF32_11_4_402_287!DllUnregisterServer
Opera_534b0000!OpWaitFileWasPresent
ntdll!TppPoolReserveTaskPost
ntdll!TppTimerpSet
Opera_534b0000!OpGetNextUninstallFile
Opera_534b0000!OpSetLaunchMan
Opera_534b0000!OpSetLaunchMan
Handle a20
  Object Specific Information
    Thread Id   88ac.1700
    Priority    10
    Base Priority 0
    Start Address 5497083d NPSWF32_11_4_402_287!native_ShockwaveFlash_TCallLabel
ModLoad: 6bda0000 6bdd0000   C:\Windows\SysWOW64\wdmaud.drv
ModLoad: 72cb0000 72cb4000   C:\Windows\SysWOW64\ksuser.dll
ModLoad: 6d9e0000 6d9e7000   C:\Windows\SysWOW64\AVRT.dll
ModLoad: 6cad0000 6cad8000   C:\Windows\SysWOW64\msacm32.drv
ModLoad: 6bd80000 6bd94000   C:\Windows\SysWOW64\MSACM32.dll
ModLoad: 6bd70000 6bd77000   C:\Windows\SysWOW64\midimap.dll
Exit thread 6:3f90, code 0
Exit thread 7:4074, code 0
Unload module C:\Windows\SysWOW64\Macromed\Flash\NPSWF32_11_4_402_287.dll at 54760000
Unload module C:\Windows\system32\WINSPOOL.DRV at 736f0000
(88ac.1700): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000102 ebx=00000000 ecx=75a914d0 edx=00000000 esi=11c8d000 edi=11c8d470
eip=549706c0 esp=0aacfd38 ebp=0aacfd80 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246
<Unloaded_NPSWF32_11_4_402_287.dll>+0x2106c0:
549706c0 ??              ???

I saw that thread 88ac.1700 is created by the Flash Player. When the player is unloaded the thread is still running. Finally, the thread calls back to the unloaded module. Therefore, I knew Flash Player created the thread that calls back to its own module that has been unloaded.

And here is some interesting observation.

When started to look into triggering a dialog in Flash Player I created the following ActionScript that executes an infinite loop.

class InfiniteLoop {
static function main() {
           for (var i = 0; i < 1;) {}
}
}

I built the Flash file with the command below.

mtasc.exe -swf InfiniteLoop.swf -main -header 125:125:20 InfiniteLoop.as

The Flash file triggers a slow script warning dialog. I was able to unload and to crash the Flash Player while the dialog was still running. I achieved this with version 11.4.402.278. When reported the problem to Opera they couldn't reproduce it. That time there was a Flash Player update, and I found it out that they used the latest version of Flash Player that was 11.4.402.287. With the latest version, I couldn't reproduce the problem either.

It got me thinking if Adobe fixed security problem either silently or "accidentally". When I checked the release notes of the latest version I didn't see indication of this kind of fix. I still don't know what had happened between the two releases but thought to investigate this issue further. I tried to trigger the bug via different code paths. I looked at my Flash file collection to dig out a file that triggers a dialog. And here it is, with the one I found, I could crash the player with the same call stack as before.

Initially, I thought Opera could fix this issue by leaving the module permanent in the address space, so I reported this to Opera. However, they notified Adobe after confirming it's not the problem in Opera (though they considered it as a stability issue). Adobe received the report at the end of November, 2012. The bug is now fixed in the security update that has a number of APSB13-09 at 12 March, 2013.

The preliminary version of this blog post was reviewed by Adobe. The preliminary version is same to the published version apart from some changes in the wording.