Setting decompression WindowSize?

[siara-cc]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.20.1 darwin/amd64

What operating system and processor architecture are you using (go env)?

MacOS Monterey Intel i5 64 bit

What did you do?

I am trying to use NewReader() for decompressing huge zstd archives from Reddit in stream mode, but I am getting
failed to decompress: Frame requires too much memory for decoding
I was using https://github.com/klauspost/compress, which has zstd.WithWindowSize(2<<30) so it was working fine, but it uses twice the memory so I end up occupying 4gb for decompression instead of 2gb window size as mentioned here:

klauspost/compress#675 (reply in thread)

What did you expect to see?

I was hoping your lib can expose this window size option so I can see if memory usage decreases?

What did you see instead?

I am getting error failed to decompress: Frame requires too much memory for decoding

[Viq111]

Hi @siara-cc,

It looks like the zstd C API allows you to set the max window size: https://github.com/DataDog/zstd/blob/1.x/zstd.h#L2075-L2082

Currently to keep things simple, the Go library doesn't expose that C object (creates and keep it internally): https://github.com/DataDog/zstd/blob/1.x/zstd_stream.go#L397

but I've seen the need to set different more advanced options from different use cases so I will mark this one as a feature request. If you are willing to contribute to add a new interface (maybe NewReaderWithOptions(ctx *C.ZSTD_DCtx, r io.Reader, dict []byte)), I'd be happy to review the PR!

If this is just for testing, you can probably just call ZSTD_DCtx_setMaxWindowSize here: https://github.com/DataDog/zstd/blob/1.x/zstd_stream.go#L398 by modifying your local DataDog/zstd library

[cce]

Hi, I had similar concerns about managing window sizes, especially when handling multiple concurrent streams or untrusted senders, so I followed the advice of @Viq111 and opened PR #150 providing Reader and Writer support for setting and limiting window sizes. Hope this is useful!