Integer Overflow and Memory Failures in WavPack

The common consequences of the integer overflow are denial-of-service and out-of-bounds memory access. What I'm describing here is neither of them but rather a rare consequence I discovered recently.

WavPack is an open source project providing audio compression and decompression. The project consists of the library and utilities exercising the library. The library compresses and decompresses Wav and WavPack streams. It's used in several software (e.g. WinZip) and hardware. One of the utilities called WVUNPACK decompresses the WavPack file.

I'm discussing a bug in WVUNPACK that is not shipped with the 3rd party products using WavPack library, and so the bug doesn't affect 3rd party products but WVUNPACK only.

WVUNPACK has an undocumented command line switch k that allows the user to control the size of the buffer to operate with. This switch expects a number to calculate the size of the buffer with.

case 'K': case 'k':
    outbuf_k = strtol (++*argv, argv, 10);

outbuf_k is int, and its value is controlled by the user. Assume the user calls WVUNPACK specifying -k4194303 in the command line. The integer after k is copied to outbuf_k. Note, 4194303 is 0x3fffff.

The program calculates the size of the buffer to operate with. However, there is an integer overflow when calculating output_buffer_size and it becomes 0xfffffc00 after the execution of the code snippet below.

if (outbuf_k)
    output_buffer_size = outbuf_k * 1024;

The program attempts to allocate memory with 0xfffffc00. The allocation fails, and the returning pointer that is NULL is not sanity-checked.

output_pointer = output_buffer = malloc (output_buffer_size);

Without detecting the memory allocation failure, the execution continues and the decompression is starting by creating an output file and writing a header in it.

if (!DoWriteFile (outfile, WavpackGetWrapperData (wpc), WavpackGetWrapperBytes (wpc), &bcount) ||

The program unpacks the content of the file to a temporary buffer.

samples_unpacked = WavpackUnpackSamples (wpc, temp_buffer, samples_to_unpack);

However, the block within the if statement is not reached because output_buffer is NULL, and so the decompressed data is not written to the output.

if (output_buffer) {
[...]
        if (!DoWriteFile (outfile, output_buffer, (uint32_t)(output_pointer - output_buffer), &bcount) ||

To recap the issue, the integer overflow causes that the unpacked data is not written to output and there is no error displayed believing the unpacking is successfully completed.

The screenshot below demonstrates there is no error displayed but when manually checking the file size there is a mismatch.

In addition to that, WVUNPACK supports to calculate and to display MD5 signature to verify the output. This can be enabled by m command line switch. This check is performed on the temporary buffer that is never written to output, therefore the error remains undetected. In fact, the program could display the correct MD5 while the output has different checksum.

Here is the code snippet demonstrates MD5 calculation is performed on temporary buffer.

if (calc_md5 && samples_unpacked) {
[...]
    MD5Update (&md5_context, (unsigned char *) temp_buffer, bps * samples_unpacked * num_channels);

And here is the screenshot demonstrates the bug in checksum verification.

The bug described above is found in WVUNPACK, however, I'd like to provide information about the library, too, for those use it in 3rd party products. According to WavPack website the library is used in several hardware including jukeboxes, multimedia and network players, and in many software including VLC Media Player.

WavPack library, probably for performance reasons, doesn't check the return value of memory allocation functions. This looks safe when investigating the library on isolation as those seem to work with small buffers and so it's difficult to make the allocation to fail, and to possibly enter in vulnerable paths. However, the library is widespread and used in different systems, and in different software environment, it could even possibly run in browser process. Earlier this year, I proved how to make allocation fails with fixed or small size.

Few examples could access near NULL:

orig_data = malloc (sizeof (f32) * ((flags & MONO_DATA) ? sample_count : sample_count * 2));
memcpy (orig_data, buffer, sizeof (f32) * ((flags & MONO_DATA) ? sample_count : sample_count * 2)); 

[...]

wps->blockbuff = malloc (wps->wphdr.ckSize + 8);
memcpy (wps->blockbuff, &wps->wphdr, sizeof (WavpackHeader));

[...]

riffhdr = wrapper_buff = malloc (wrapper_size);
memcpy (wrapper_buff, WavpackGetWrapperLocation (first_block, NULL), wrapper_size);