Skip to content

Fix AI post-processing modes and add Smart Edit + Tags - #1253

Open
lacksfish wants to merge 5 commits into
blinkospace:mainfrom
lacksfish:fix/ai-post-processing-mode-routing
Open

lacksfish wants to merge 5 commits into
blinkospace:mainfrom
lacksfish:fix/ai-post-processing-mode-routing

Conversation

@lacksfish

Copy link
Copy Markdown

This PR fixes AI post-processing that could leave the original note unchanged while adding two AI comments. It makes each processing mode perform its selected action and adds an explicit Smart Edit + Tags option.

This PR builds on the changes in #1245. It is a follow-up to that work and should be reviewed and merged after it. Until #1245 is merged, a comparison against main also includes those earlier changes.

The intended workflow is straightforward: record a voice note, transcribe it, replace the note text with the cleaned-up version, keep the audio, and add real tags; without additional AI comments.

Why “Both” produced two comments

The Git history explains how this behavior developed:

As a result, a model could return a perfectly usable rewritten note, but Blinko would leave the transcript in the main note and display the rewrite as another comment. Even a successful editing-tool call could become a suggestion because of the transcript-preservation rule.

The historical commits establish the original meaning of Both. They do not explain why Smart Edit was later added to it. This PR restores the original Comment + Tags combination and gives editing plus tagging its own clearly named option.

Changes to the processing options

Option Behavior after this change
Comment Continues to create one AI comment without replacing the note. Empty responses leave the step available for retry.
Automatic Tags Saves real tag relationships without appending a hashtag line to the note text. Existing spelling is preferred. Custom prompts are no longer cut off at five tags; the default remains five.
Smart Edit Replaces the current note text. An editing-tool result or an ordinary text response can supply the replacement. Neither becomes a fallback comment.
Both Renamed Comment + Tags in the UI. Creates one comment and applies tags. It no longer runs the additional Smart Edit step. The stored both setting remains compatible.
Custom Adds an explicit result-action choice: replace the note, create a comment, or use tools. Existing configurations default to tools. In that setting, a response without an action leaves the step available for retry instead of silently creating a comment.
Smart Edit + Tags — new Replaces the note text, then applies real tags, with no AI comments.

For automatic editing and tagging, select the new Smart Edit + Tags option and put the editing instructions in the Smart Edit prompt field. The existing Comment prompt is not silently reused for editing.

Preserving the note and its attachments

Smart Edit now uses editing instructions rather than the Comment Assistant's emoji and length guidelines. It offers only the current-note update tool and also accepts a normal text response as the replacement, so success does not depend on the model choosing a tool call.

Audio attachments, metadata and existing tags remain attached to the same note. The previous text is saved in revision history; the source transcript does not have to be copied into the visible note. Independent tags remain intact after later text edits and are displayed separately from the note body.

AI updates do not start another round of post-processing. Completed steps are not repeated on retry, and a concurrent manual edit prevents an outdated AI result from overwriting it. This follow-up requires no additional database migration beyond the changes in #1245.

Validation

Added pipeline regression tests and OpenAI-compatible response tests, and extended the existing tag tests. The focused suite passes 36 tests, including existing hashtag-parsing coverage:

  • Smart Edit with an actual update-tool call or a plain-text response.
  • One note update and no comment, including multiple tool responses.
  • Comment-only, tags-only, Comment + Tags, and Smart Edit + Tags behavior.
  • Audio retention, original-text history, and preservation of existing tags.
  • Transcription before editing, completed-step retries, expired workers and concurrent manual edits.
  • Custom editing and comment actions, plus responses without a requested tool action.
  • Real Mastra/OpenAI SDK handling of both ordinary content and tool-call responses using controlled responses.

Run the focused suite from server with bun run test:note-processing. Existing LiteLLM provider, audio-recorder and file-upload checks also passed. Changed TypeScript files passed syntax/transpilation checks, and git diff --check passed.

I have also manually tested the updated behavior in my self-hosted Blinko instance.

Persist audio-processing state in PostgreSQL and run it through pg-boss so transcriptions are saved before embeddings or AI post-processing
    begin.

- append completed transcripts without overwriting concurrent note edits
  - retain partial successes and provide retry/skip controls with visible status
  - prevent duplicate processing and fence stale workers after restart or timeout
  - use the selected voice provider's explicit OpenAI-compatible endpoint
  - distinguish recorded voice notes from ordinary audio uploads
  - add regression coverage for delays, failures, retries, conflicts, and provider routing
  - require Node 22 because the supported pg-boss version requires Node >= 22.12
Fix voice recording downloads and make AI-generated tag processing accept the textual formats that tagging models commonly return.

- keep the voice download action visible inside the audio player
  - prevent download and delete controls from toggling playback or opening the note
  - add a translated tooltip, accessible label, and keyboard focus styling
  - support comma-, whitespace-, and newline-separated hashtags
  - support Unicode and hierarchical tags such as #Parent/Child
  - discard explanatory prose, code blocks, URLs, and malformed tag paths
  - deduplicate generated tags and retain the five-tag limit
  - preserve existing note tags when appending generated tags
…removal

Prevent dialog rerenders from restarting voice recordings and make voice attachments removable with mouse,
    keyboard, and touch.

- show the existing removal X on hover or focus for mouse users and keep it visible for touch-primary devices
  - retain captured audio across visibility changes and unexpected recorder stops
  - resume browser-paused sessions without overriding manual pauses
  - request a screen wake lock while recording and release it on stop
  - use one elapsed-time counter and preserve the duration after stopping
  - add tests for interruptions, delayed permissions, wake locks, and attachment removal
Save Smart Edit tool calls and plain-text responses as replacement note bodies,
without comment fallbacks or mandatory visible copies of source transcripts.
Preserve audio, metadata, revision history and independent tag relations.

Restore Both's original comment-plus-tags behavior and expose Smart Edit plus
Tags explicitly. Git history confirms that c76381e introduced Both before
Smart Edit existed; 1d12409 later added a third stage and a second comment.
Document this provenance without inferring undocumented developer intent.

Add explicit Custom result actions, canonical relational tagging, and tag
preservation/display when note bodies contain no hashtags.

Validation: 36 focused regression tests passed, including real Mastra/OpenAI
SDK response handling. Existing LiteLLM, audio and upload checks also passed.
Full typechecks remain incomplete due to memory growth; builds are blocked
by Windows directory access errors. No live Docker/model validation.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant