LZSSE writes outside buffer for very short data

[nemequ]

I've started fuzzing the decompressor, and it found this issue pretty much instantly.

Here is the program I'm using. It writes the uncompressed size at the beginning of the files as a size_t; for the test archives to work size_t must be 8 (though changing a few instances of size_ts to int64_t should make them work elsewhere).

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
#include <stdint.h>
#include <string.h>
#include <stdbool.h>

#include "lzsse2.h"

static void
print_help(const char* program_name) {
  printf("Usage: %s c|d\n", program_name);
  printf("Compress or decompress from stdin to stdout\n");
}

#define IO_BLOCK_SIZE ((size_t) (1024 * 1024))

int
main(int argc, char** argv) {
  bool compress;

  if (argc != 2) {
    print_help(argv[1]);
    return EXIT_SUCCESS;
  }

  compress = *(argv[1]) == 'c';

  uint8_t* input = (uint8_t*) malloc(IO_BLOCK_SIZE);
  size_t input_length = 0;

  while (!feof(stdin)) {
    const size_t bytes_read = fread(input + input_length, 1, IO_BLOCK_SIZE, stdin);
    input_length += bytes_read;
    input = (uint8_t*) realloc(input, input_length + IO_BLOCK_SIZE);
    assert(input != NULL);
  }

  uint8_t* output = NULL;
  size_t output_length = 0;

  if (compress) {
    unsigned int level = 7;
    if (argv[1][1] != '\0') {
      level = atoi(argv[1] + 1);
    }

    LZSSE2_OptimalParseState* state = LZSSE2_MakeOptimalParseState(input_length);
    assert(state != NULL);

    output = (uint8_t*) malloc(input_length);

    output_length = LZSSE2_CompressOptimalParse(state, input, input_length, output, input_length, level);
    assert(output_length != 0);
    assert(output_length <= input_length);
    LZSSE2_FreeOptimalParseState(state);

    fwrite(&input_length, sizeof(size_t), 1, stdout);
  } else {
    if (input_length <= sizeof(size_t))
      goto cleanup;

    const size_t decompressed_length = *((size_t*) input);
    output = (uint8_t*) malloc(decompressed_length);
    if (output == NULL)
      goto cleanup;

    output_length = LZSSE2_Decompress (input + sizeof(size_t), input_length - sizeof(size_t), output, decompressed_length);
    if (output_length == 0)
      goto cleanup;
  }

  fwrite(output, 1, output_length, stdout);

 cleanup:
  free(output);
  free(input);

  return EXIT_SUCCESS;
}

Here are some archives which cause a crash when compiled with AddressSanitizer: http://code.coeusgroup.com/afl-results/52558fa6-cf92-4446-a84b-636a02faecff.tar.xz

I'm posting this publicly in spite of https://github.com/nemequ/compfuzz/#disclosure-policy because I don't see this being an issue in real life (you would need a malloc implementation which puts tiny buffers immediately before a page boundary, which only tools designed to find issues like this do), and I doubt anyone is using LZSSE in production yet. If I find more realistic issues I'll disclose them privately.

[ConorStokes]

Sorry, I thought I had replied to this issue. Will try and look into it this weekend, I've got an idea why it might occur (I think it might be related to the minimum compressible length handling).