Repository navigation
fix(tools): stop crush_logs dropping entries at 8KB chunk boundaries - #3988
Open
devYRPauli wants to merge 5 commits into
Open
devYRPauli wants to merge 5 commits into
devYRPauli wants to merge 5 commits into
Conversation
readLastLines reads the log backwards in 8KB chunks. It treated the line at the end of each chunk as the incomplete one. When reading backwards, the incomplete line is the first one in the chunk, because its start is in the earlier chunk. Entries that crossed a chunk boundary were dropped or replaced by older entries, and a log of lines over 8KB returned no entries. Carry the first line of each chunk back and join it with the earlier chunk. When the carried part passes maxLogLineSize, scan back to the previous newline instead of buffering the whole line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
crush_logsagent tool skips log entries and returns older entries in their place. For a log of 20KB entries, it returnsLog file is empty.readLastLinesininternal/agent/tools/crush_logs.goreads the file backwards in 8KB chunks. It treats the line at the end of each chunk as the incomplete one. When reading backwards, the incomplete line is the first one in the chunk, because its start is in the earlier chunk. A line that crosses a chunk boundary is never joined back together. The tool drops it and reads further back to fill the count.The
crush logscommand reads the same file withnxadm/tailand is not affected.Effect
The log has 200 JSON entries of about 360 bytes each, and entry 150 is padded to 20KB. This is the file
TestCrushLogs_EntriesAcrossChunkBoundarieswrites.crush_logswithlines: 100:mainA log of 5 entries of 20KB each returns
Log file is emptyonmainand all 5 entries on this branch.Change
Each chunk carries its first line back, and the next, earlier chunk joins it to its own last line. The block after the loop that parsed a leftover remainder goes away, because the chunk at offset 0 has no incomplete line.
When the carried part grows past
maxLogLineSize, the loop drops it and scans back to the previous newline. The parse loop already skipped such lines. Without the scan-back, the carried part is copied again for every 8KB chunk. I timed a log of 10 entries, one 20MB line and 10 more entries. Withlines: 20, it takes about 20s onmainand with only the carry fix. It takes about 65ms on this branch. All three versions return the same 20 entries.Verification
New test
TestCrushLogs_EntriesAcrossChunkBoundarieswrites the file above and requires entries 100 to 199 in order. It fails before the change. This excerpt of the output cuts the long pad value and theError TraceandTestlines, and shows tabs as spaces:It passes after it.
New test
TestCrushLogs_ScanBackPastOversizedLineexercises the scan-back. Its big entry is longer thanmaxLogLineSizeby more than one 8KB chunk. Of the 3 padded entries before it, 2 cross chunk boundaries, and 3 short entries share the chunk where the big entry starts. The test requires the 6 entries before the big entry and the entry after it, in order. It fails onmain, and it fails if the scan-back skips the entries before the big entry.New test
TestCrushLogs_LinesAlignedToChunksguards the carry logic when every chunk starts at a line start, and it also passes onmain.go build ./...passes.go test -race ./internal/agent/tools/...passes, with 260 passing tests and subtests ininternal/agent/tools, up from 257, and 4 skips on both.go test -race ./internal/app/... ./internal/cmd/...passes. Both depend oninternal/agent/tools.golangci-lint runat the CI-pinned v2.13.1 reports 0 issues on./internal/agent/tools/..., andgofumpt -llists no changed file.