Problem
When crystallizeDraft() calls the LLM to promote an L2 policy into a Skill, the LLM sometimes returns an empty response (or malformed JSON). The error is logged as skill.crystallize.failed and the policy is skipped via continue.
On the next trigger event (reward.updated, l2.policy.induced, etc.), the same policy is re-evaluated and crystallization is attempted from scratch — the same prompt, the same LLM call, with no memory of the previous failure.
If the underlying issue (e.g. empty response for a particular prompt shape) persists, the policy enters an infinite retry loop with no progress.
Suggestion
Implement a retry-with-context mechanism for failed crystallization attempts:
- On failure: store the LLM's raw output (even if empty/malformed) as a temporary artifact linked to the policy.
- On subsequent retry: inject the previous attempt as additional context in the LLM prompt — e.g. "The previous attempt produced the following output: ... Please correct this and generate a valid skill definition."
- On success: clear the temporary artifact.
This mirrors the pattern used elsewhere in MemOS (e.g. LlmClient.completeJson already retries internally with maxMalformedRetries), but applies it at the orchestrator level so the context of why the previous attempt failed is visible to the model.
Why it matters
- Without this, a policy that consistently triggers empty LLM responses will be retried infinitely on every pipeline cycle with zero signal for the model to adjust.
- With context, even a single retry can succeed if the model sees its own blank output and corrects course.
- The change is scoped to
crystallizeDraft() and/or runSkill() — no upstream API changes needed.
Implementation sketch
- Add a
failureArtifact KV store keyed by policyId in the skill repos (or reuse kv table).
- In
crystallizeDraft() catch block, save { raw, error, timestamp } to the artifact.
- In
packPrompt(), check for existing artifact and inject a "correction" block to the LLM prompt.
- In the success path, clear the artifact.
Would be happy to discuss further or submit a PR if the direction is agreed on.
Problem
When
crystallizeDraft()calls the LLM to promote an L2 policy into a Skill, the LLM sometimes returns an empty response (or malformed JSON). The error is logged asskill.crystallize.failedand the policy is skipped viacontinue.On the next trigger event (
reward.updated,l2.policy.induced, etc.), the same policy is re-evaluated and crystallization is attempted from scratch — the same prompt, the same LLM call, with no memory of the previous failure.If the underlying issue (e.g. empty response for a particular prompt shape) persists, the policy enters an infinite retry loop with no progress.
Suggestion
Implement a retry-with-context mechanism for failed crystallization attempts:
This mirrors the pattern used elsewhere in MemOS (e.g.
LlmClient.completeJsonalready retries internally withmaxMalformedRetries), but applies it at the orchestrator level so the context of why the previous attempt failed is visible to the model.Why it matters
crystallizeDraft()and/orrunSkill()— no upstream API changes needed.Implementation sketch
failureArtifactKV store keyed bypolicyIdin the skill repos (or reusekvtable).crystallizeDraft()catch block, save{ raw, error, timestamp }to the artifact.packPrompt(), check for existing artifact and inject a "correction" block to the LLM prompt.Would be happy to discuss further or submit a PR if the direction is agreed on.