feat: local evidence intake (EML/MBOX parser, panel command, directory harvest) - #151
Closed
titanwings wants to merge 5 commits into
Closed
titanwings wants to merge 5 commits into
titanwings wants to merge 5 commits into
Conversation
Two credential-free local parsers give the material intake path the largest source users already hold: distilly-eml (message/rfc822) and distilly-mbox (application/mbox), with .eml/.mbox/.mbx mapped in the runtime. Structural scanning works on the raw bytes and decodes each part with the charset that part declares, so an 8-bit legacy body is not decoded twice. Multipart recursion prefers a non-empty text alternative, attachments are counted and reported, quoted-printable and base64 decode through the declared charset, HTML degrades to text with tags and entities handled, and both an over-limit result and a binary payload are refused rather than truncated or published as evidence. Three independent audit rounds shaped this: the second found that the first round of fixes had introduced an mbox regression that merged messages and a quadratic markup scan, and the third found the scan was still quadratic and that a raw-byte control check rejected ISO-2022-JP and UTF-16. Every counter-example from all three rounds is now a regression test.
Reviewing a suspended candidate previously required the host to be in the loop, because only the MCP presenter started the Panel. The new command starts the same loopback-only Panel for the installed DISTILLY_ROOT, prints the URL a human opens, waits for SIGINT, SIGTERM, or a closed stdin, and then closes the runtime. PanelLauncher gains a start() that returns the validated root URL; present() keeps appending the exact review route to that same root.
An independent fourth verification round confirmed the linear markup scan and the
charset handling, then found four residual defects, each now fixed and covered by
a regression test:
- a date-less separator with trailing whitespace ("From x@y ") has an empty
remainder, which the rule described but the code did not accept, so a whole
mailbox merged into one message;
- any digit later in a remainder still split prose such as "From the desk of
Bob, 2nd floor"; the remainder must now start with a date shape;
- a commented-out opener ("<!-- <script> -->") was honoured as real and deleted
visible text up to the next closer, so comments are removed before the scan;
- element names were matched by prefix, so <scripty> and <stylesheet> contents
were removed even though their text is visible.
Pointing Distilly at one export folder is the cheapest intake path a user has, and nothing could do it: ingestFiles takes explicit files, and the model-facing surface must stay at five tools. The new harvest command walks one user-selected directory deterministically (entries ordered by the UTF-8 bytes of the root-relative path), never follows a symlink, never leaves the root, and accounts for every discovered path under one of eight skip reasons so a run reports what it left out instead of treating it as evidence. Selection is capped, and files are ingested in batches sized by the protocol's per-call material limit with visible progress. Distilly.ingestFiles is added to the browser-safe facade so a direct-user surface can batch a large selection without widening any model-facing tool.
A fifth verification round confirmed the round-four fixes and found three more,
each now fixed and covered by a regression test:
- the separator rule required a literal space, so a tab-terminated date-less
separator ("From x@y\t") merged a whole mailbox; it now accepts spaces or tabs;
- the date shape was too narrow and rejected shapes real writers emit (padded
sender, full weekday, hyphenated or slashed dates, leading time, compact date,
short epoch, angle-bracketed epoch) while still rejecting prose that merely
contains a digit;
- comment removal ran before the raw-text scan, so an unclosed "<!--" inside
script or style deleted every later visible character. Comments and raw-text
elements are now resolved in one scan, where a commented-out opener is not
markup and a comment opener inside script text is not a comment.
Owner
Author
|
Superseded: this stacked series is consolidated into the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
feat: local evidence intake (EML/MBOX parser, panel command, directory harvest)
Commits
Stack
Stacked on
pr/01-dsh-binding; the whole series ends atdistilly-work(0bd924e).Verification
Every feature carries an author report and an independent audit in the delivery evidence folder (
host-verification/), including screenshots and the audit verdict that drove the fixes. Gates on the top of the stack: format, lint, docs, release check, and 1163 tests.Real hosts
Codex, Claude Code 2.1.268, OpenClaw 2026.9.2, Hermes v0.19.0, and DSH 0.1.5-rc.1 were each exercised end to end (install, host-visible confirmation, exactly five MCP tools, person Skill install/remove, uninstall). Capacity fixtures are recorded for Codex 0.146.0, OpenClaw 2026.3.24, and Hermes v0.9.0; newer host versions run on the labelled conservative floor.
Evidence
Author reports and screenshots (delivery evidence folder
host-verification/):Independent audit reports: