Instrumenting Flash Player to Inspect JITted Pages for Integer Errors

In this blog post I'm writing about the method I experiment with to discover potential areas, that may or may not be prone to integer errors, in Flash Player.

I have 26k flash files that are used as a corpus to generate the test samples for Flash Player. The test samples have an element of 0x41424241 in the integer pool. I have a total of 344k files generated to test Flash Player with.

During the test, I use a pintool to instrument the JITted code. The pintool is based on this implementation. Since the elements in the integer pool being dereferenced by action script it makes sense to restrict the instrumentation to JITted code.

I use instruction-level instrumentation that allows to check the register values at every single instruction being executed. If any of the general registers have the value of 0x41424241 and the instruction is referencing to that register, the instruction information along with general registers are logged.

The pintool pre-allocates the address of 0x41424241 so Flash Player won't use that memory address and so reducing the irrelevant lines in the log.

There is no need to wait for the test to finish to get partial results. A log file is generated for each test file rendered by Flash Player.

The size of log files are vary. Some are close to size 0 specially if the value above makes the code to fail early. There are many log files with size 4k. When the execution keeps going long the size is about 16k. If the value has to do something with a loop the log can reach 100k but that's rare.

Logs can be grouped and many can be thrown away as they don't contain instructions associated with vulnerabilities.

What I look for is like signed shift, addition, subtraction, or multiplication instructions. If the value is used in displacement with lea instruction that counts suspicious too.

Once an address in the log is chosen for closer inspection, I reproduce the log on isolation with an option to dump the JITted pages so I can manually analyze the surrounding area of that address in disassembler. Knowing the state information it's also possible to debug the code.

If results positive, certain level of automation can be added.