When testing software, it's reasonable to think that error conditions like above are unlikely to be reached as often as other error conditions. Some of them may never get reached because it's not obvious to make the allocation with fixed-size to fail.
#define MAX_SIZE 0x08000000 /* 128M */
ptr = malloc(MAX_SIZE)
if (ptr == NULL)
OOM error condition could be a potential to enter into vulnerable code path. One example I've heard that the memory allocation had failed and the error condition led to call the cleanup functionality which caused to free of uninitialized pointer.
When testing, one possible way to make memory allocation to fail is to hook the memory allocation function and change the functionality to fail, that is to return with
NULLpointer. However, I wanted to avoid this approach of testing because even if I find bug I'd still need a lot of work to reproduce it in real word scenario (if possible at all), when the hook is not applied.
Therefore, what I thought is that the other way to force fixed-size memory allocation to fail is to consume large amount of memory of the virtual address space, and as a consequence the program will be operating under low-memory conditions. This means if memory allocation occurs with reasonable small size it could fail because there is not much memory available. My theory is that any vulnerability found that we reach via OOM error conditions can be reproduced in a real word scenario because we always expect to find something that triggers to consume large amount memory in order to make other allocation to fail. This is pretty much possible in browsers where basically you can easily fill arbitrary amount of the heap memory.
Experimentally, I involved this approach in my fuzzing methodology. I attach the target process to Windbg and execute
.dvalloccommand to allocate arbitrary amount of memory in the virtual address space. The amount of memory available in the address space can be queried with
!address -summarycommand, or can be seen in Process Explorer, and so you can allocate more for your test if needed.
When testing Firefox with this methodology I notice increased amount of
int 3crashes in
mozalloc_abort()that is due to out of memory condition - those crashes are not exploitable though.