Getting java.io.IOException: Compressed with incompatible lzo version: 0x2090 (expected 0x2050)

[ronakvorabr]

I'm trying to decompress files that were decompressed with a native lzo library. I'm wondering if this library only supports a specific version of lzo?

The version the other files were compressed with is:

lzop --version
lzop 1.03
LZO library 2.06

Is this a quick fix? Or should I try to find another way to go about decompressing these files?

Thanks in advance!

Ronak

[ronakvorabr]

UPDATE:

I have updated the source code in lzoconf.h such that #define LZO_VERSION 0x2050 is now #define LZO_VERSION 0x2090. The file still decompresses correctly.

I've attached a sample of the lzo file that didn't work with the current state of the code.

[zman0900]

Any chance of fixing this, or adding some option to try ignore it? I'm getting a similar error when trying to decompress a file created with Twitter's hadoop-lzo 0.4.20.

java.io.IOException: Compressed with incompatible lzo version: 0x2060 (expected 0x2050)

I'm am using it like so:

org.apache.hadoop.fs.Path file = ...
try (InputStream stream = new LzopInputStream(file.getFileSystem(hadoopConf).open(file))) {
    // Do stuff with stream
}

EDIT:
I realized the problem here is that LZO is not forwards-compatible. This library appears to have been originally created from the liblzo2 2.05 source, while Twitter's hadoop-lzo uses whatever liblzo2 is installed, which is 2.0.6 in my hadoop cluster.

There is a check in this library's code to error out if the file to decompress has been compressed with a newer version. If I use a debugger and force that check to pass, decompression works anyway, but I may have just gotten lucky with my specific file.

Seems like this library will need to eventually be updated to match the changes in newer versions of liblzo2, which I assume will not be an easy task.