Add zstd compression/decompression IO Stream

[mdedetrich]

zstd decompression/compression should be added a standard pekko IO stream as its far superior to gzip/lzma in every way. The API design should be the same as it is for gzip.

Doing this requires using https://github.com/luben/zstd-jni, which uses the standard reference C library via JNI (unfortunately there is no such implementation in native Java and even if there would be I am not sure it would be as fast as the C one).

[He-Pin]

@mdedetrich We are using this too, at Taobao's api gateway

[He-Pin]

We pre-trained the dictionary and delivered it to the client. If you want to leverage it.

[mdedetrich]

We pre-trained the dictionary and delivered it to the client. If you want to leverage it.

I will add an API for compressing/decompressing via dictionaries as well, ill see if adding an API to create dictionaries is warranted (as you can just use zstd-jni directly for this)

[jrudolph]

Doing this requires using luben/zstd-jni, which uses the standard reference C library via JNI (unfortunately there is no such implementation in native Java and even if there would be I am not sure it would be as fast as the C one).

There is at https://github.com/airlift/aircompressor

[jrudolph]

(I also wrote a very high-level, pure implementation in Scala using scodec but it's not optimized and ~ 100-1000x slower than native at https://github.com/jrudolph/decoders/blob/main/src/main/scala/net/virtualvoid/codecs/zstd/Zstd.scala)

[mdedetrich]

Doing this requires using luben/zstd-jni, which uses the standard reference C library via JNI (unfortunately there is no such implementation in native Java and even if there would be I am not sure it would be as fast as the C one).

There is at https://github.com/airlift/aircompressor

According to their readme

This library requires a Java 22+ virtual machine containing the sun.misc.Unsafe interface running on a little endian platform.

So I don't think it can be used as Pekko 2.0.x supports JVM 1.7 or older. Another alternative would be to only release an interface for zstd compression/decompression and there is a separate pekko artifact for the implementation which can be zstd-jni or airlift (or anything else) but that this to me is a bit of overengineering but I can be convinced otherwise.

[He-Pin]

FYI, Netty 5 will require JDK 25:)

[mdedetrich]

FYI, Netty 5 will require JDK 25:)

Yup, but would be a Pekko 3.0.0 before we can use it, and at that time we can also consider swapping out the underlying zstd implementation for a better one if it exists (which we can do since a bump in major version allows breaking changes)