fix: stop dropping the bytes between the chunk boundary and the read size

[arpitjain099]

Proposed Changes

ReadChunk reads up to size + maxPeekSize bytes out of the reader, and generateChunk then walks forward from size looking for a safe \n\n split. When it finds one it breaks and returns the prefix. The bytes that the read already consumed after that split are dropped on the floor. The next ReadChunk call starts from where the reader is now, which is past them, so that content never reaches the detectors.

Both loops (engine.go:337 for filesystem, plugins/confluence.go:370 for Confluence pages) call ReadChunk in a loop until io.EOF, so the gap is silent. Nothing errors, the scan just does not cover part of the file.

With a chunk size of 10 and a peek size of 5, scanning "0123456789\n\nSECRET\n" gives:

expected: "0123456789\n\nSECRET\n"
actual  : "0123456789\n\nRET\n"

"SEC" is gone. At the shipped defaults the window is 100KiB/25KiB, so any file over 100KiB can lose up to 25KiB per chunk, and a secret sitting in that region is not scanned.

The fix is to Peek rather than Read, and Discard exactly the number of bytes the returned chunk covers. That is what the "look-ahead" naming already implies. Both call sites build the reader with bufio.NewReaderSize(..., GetSize()+GetMaxPeekSize()), so the peek fits; bufio.ErrBufferFull is tolerated anyway, which means a caller with a smaller reader degrades to a shorter look-ahead instead of erroring.

Error behaviour is unchanged: empty input still returns io.EOF, and the unsupported-file-type check still runs on the first read with the same window of bytes.

Side note: GetPeekedBuf/PutPeekedBuf and peekedBufPool are no longer used by ReadChunk. I left them in place since they are exported and covered by tests. Happy to remove them if you would rather not carry them.

Checklist

• I covered my changes with tests.
• I Updated the documentation that is affected by my changes:
  • Change in the CLI arguments
  • Change in the configuration file

TestReadChunkKeepsEveryByte reads a source to EOF and compares the concatenation of every chunk against the input. It fails on master for two of its three cases and passes with this change. go test ./engine/... ./plugins/... is green, and gofmt is clean.

I submit this contribution under the Apache-2.0 license.