diff --git a/README.md b/README.md index 709ff19..e9ff306 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Claude Code + Obsidian + Graphify: The Definitive Guide to Token Savings & Persistent Memory -> **71.5x fewer tokens per session** with Graphify + **permanent memory across sessions** with Obsidian Zettelkasten. +> **Persistent memory across sessions** with Obsidian Zettelkasten and selective code context with Graphify. Token savings depend on the task; the historical 71.5x/session and 499x/query claims are author-reported and not reproducible from published data. A complete setup to turn Claude Code into an agent with long-term memory and full codebase awareness — without wasting tokens re-reading files. @@ -28,7 +28,7 @@ When working with Claude Code, two problems silently eat your tokens: **Problem 1 — Amnesia between sessions.** Every time you open a new session, you have to re-explain your project: stack, past decisions, current bugs, what's left to do. Claude Code remembers nothing from the previous session. -**Problem 2 — Codebase re-reading.** Claude Code re-reads all your project files every session to understand the structure. A project with ~40 files burns ~20,000 tokens just for Claude to orient itself — before you even ask a question. If you run 10 sessions a day, that's **200,000 wasted tokens**. +**Problem 2 — Codebase re-reading.** Repeated source exploration can consume context tokens, but how much depends on the task and navigation tools. The historical illustration compares ~20,000 tokens of source context with ~280 tokens of graph query output: approximately 71.4x less retrieval context, not a measured reduction in complete session usage. It assumes the smaller output answers the same question; the original payloads and tokenizer were not published. --- @@ -43,7 +43,7 @@ Two complementary systems, each solving a different problem: | Conversation history | Import pipeline | Lost chat insights | Free | | Continuity | `/resume` and `/save` commands | Picking up where you left off | Free | -Obsidian handles **what was decided** (declarative memory). Graphify handles **how the code is structured** (structural map). Together, Claude Code starts every session knowing everything — without re-reading anything. +Obsidian handles **what was decided** (declarative memory). Graphify handles **how the code is structured** (structural map). Together, they can help Claude Code recover relevant context. Source reads remain necessary when the graph or notes are incomplete or stale. --- @@ -527,7 +527,7 @@ Open Claude Code session | Layer | Without it | With it | |-------|-----------|---------| | `/resume` | Re-explain project every session | Claude already knows the context | -| Graphify | Re-read ~40 files (~20k tokens) | Query 1 graph (~280 tokens) | +| Graphify | Historical illustration: ~40 files (~20k tokens) | Selected graph query output (~280 tokens); unverified payload estimate, not session usage | | Chat pipeline | Insights lost in chat history | Everything indexed and searchable | | `/save` + logs | Forget what was done | Complete history with wikilinks | @@ -638,7 +638,7 @@ Open Claude Code session ## Real Results -Tested on a React + Supabase project with 126 TypeScript files: +Author-reported example from a React + Supabase project with 126 TypeScript files. The source snapshot, graph, prompts, tokenizer and usage logs were not published, so these inventory counts and the token ratio cannot currently be reproduced: | Metric | Value | |--------|-------| @@ -647,12 +647,20 @@ Tested on a React + Supabase project with 126 TypeScript files: | Communities detected | 124 | | graph.json size | 172 KB | | Obsidian notes generated | 456 | -| Token reduction per query | **499x** | -| LLM cost for generation | **0 tokens** (AST mode) | +| Historical token reduction per query | **499x, author-reported; baseline and query counts unavailable** | +| Reported LLM usage for graph generation | **0 tokens** (AST mode); excludes agent reads, queries and session usage | | Imported chats in vault | 137 | | Accumulated permanent notes | 65+ | | Total vault notes | 780+ | +### Evidence and benchmark + +The historical **71.5x per session** headline is not supported by a published session ledger. The ~20,000 / ~280 illustration gives **~71.4x for retrieval context** (about 98.6% less), while **499x per query** is a separate undocumented comparison. Neither establishes end-to-end session savings or dollar savings. + +Hlido reports **82/100** for its repository review; the accessible public evidence does not validate these token ratios. See [issue #12](https://github.com/lucasrosati/claude-code-memory-setup/issues/12) and the [evidence audit and benchmark protocol](./benchmarks/README.md) for provenance, date discrepancies, assumptions and limitations. + +The protocol defines fixed inputs, paired source-navigation and memory arms, correctness criteria, cache accounting, graph construction costs, and separate query/session measurements. An [offline usage summarizer](./benchmarks/summarize.py) is included. Its synthetic fixture checks arithmetic only; no new performance results or paid runs are claimed. + --- ## Architecture diff --git a/README.pt-BR.md b/README.pt-BR.md index 74e721b..1295060 100644 --- a/README.pt-BR.md +++ b/README.pt-BR.md @@ -1,6 +1,6 @@ # Claude Code + Obsidian + Graphify: O Guia Definitivo para Economia de Tokens e Memória Persistente -> **71.5x menos tokens por sessão** com Graphify + **memória permanente entre sessões** com Obsidian Zettelkasten. +> **Memória persistente entre sessões** com Obsidian Zettelkasten e contexto seletivo de código com Graphify. A economia depende da tarefa; as alegações históricas de 71,5x/sessão e 499x/consulta são autorrelatadas e não são reproduzíveis com os dados publicados. 🇺🇸 [Read in English](./README.md) @@ -28,7 +28,7 @@ Quando você trabalha com o Claude Code, dois problemas consomem seus tokens sil **Problema 1 — Amnésia entre sessões.** Toda vez que você abre uma sessão nova, precisa re-explicar o projeto: stack, decisões tomadas, bugs em andamento, o que falta fazer. O Claude Code não lembra de nada da sessão anterior. -**Problema 2 — Releitura do codebase.** O Claude Code relê todos os seus arquivos de código a cada sessão para entender a estrutura. Um projeto com ~40 arquivos consome ~20.000 tokens só para o Claude se orientar — antes de você fazer a primeira pergunta. Se você faz 10 sessões por dia, são **200.000 tokens desperdiçados**. +**Problema 2 — Releitura do codebase.** Explorar o código repetidamente pode consumir tokens de contexto, mas o consumo depende da tarefa e das ferramentas de navegação. A ilustração histórica compara ~20.000 tokens de contexto de código com ~280 tokens de saída de consulta ao grafo: cerca de 71,4x menos contexto de recuperação, não uma redução medida no consumo da sessão completa. Ela pressupõe que a saída menor responde à mesma pergunta; os payloads originais e o tokenizer não foram publicados. --- @@ -43,7 +43,7 @@ Dois sistemas complementares, cada um resolvendo um problema: | Histórico de conversas | Pipeline de importação | Chats perdidos | Gratuito | | Continuidade | Comandos `/retomar` e `/salvar` | Retomar de onde parou | Gratuito | -O Obsidian cuida de **o que foi decidido** (memória declarativa). O Graphify cuida de **como o código está organizado** (mapa estrutural). Juntos, o Claude Code começa cada sessão sabendo tudo — sem reler nada. +O Obsidian cuida de **o que foi decidido** (memória declarativa). O Graphify cuida de **como o código está organizado** (mapa estrutural). Juntos, eles podem ajudar o Claude Code a recuperar contexto relevante. A leitura do código continua necessária quando o grafo ou as notas estão incompletos ou desatualizados. --- @@ -527,7 +527,7 @@ Abrir sessão no Claude Code | Camada | Sem ela | Com ela | |--------|---------|---------| | `/retomar` | Re-explicar projeto a cada sessão | Claude já sabe o contexto | -| Graphify | Reler ~40 arquivos (~20k tokens) | Consultar 1 grafo (~280 tokens) | +| Graphify | Ilustração histórica: ~40 arquivos (~20k tokens) | Saída seletiva de consulta (~280 tokens); estimativa de payload não verificada, não consumo da sessão | | Pipeline de chats | Insights perdidos no histórico | Tudo indexado e buscável | | `/salvar` + logs | Esquecer o que foi feito | Histórico com wikilinks | @@ -638,7 +638,7 @@ Abrir sessão no Claude Code ## Resultados Reais -Testado em um projeto React + Supabase com 126 arquivos TypeScript: +Exemplo autorrelatado de um projeto React + Supabase com 126 arquivos TypeScript. O snapshot do código, grafo, prompts, tokenizer e logs de consumo não foram publicados; portanto, as contagens de inventário e a razão de tokens não podem ser reproduzidas atualmente: | Métrica | Valor | |---------|-------| @@ -647,12 +647,20 @@ Testado em um projeto React + Supabase com 126 arquivos TypeScript: | Comunidades detectadas | 124 | | Tamanho do graph.json | 172 KB | | Notas Obsidian geradas | 456 | -| Redução de tokens por query | **499x** | -| Custo LLM da geração | **0 tokens** (modo AST) | +| Redução histórica de tokens por consulta | **499x, autorrelatada; contagens da baseline e da consulta indisponíveis** | +| Consumo LLM relatado na geração do grafo | **0 tokens** (modo AST); exclui leituras do agente, consultas e consumo da sessão | | Chats importados no vault | 137 | | Notas permanentes acumuladas | 65+ | | Total de notas no vault | 780+ | +### Evidências e benchmark + +A chamada histórica de **71,5x por sessão** não tem um registro publicado de consumo de sessões. A ilustração ~20.000 / ~280 resulta em **~71,4x para contexto de recuperação** (cerca de 98,6% menos), enquanto **499x por consulta** é outra comparação sem protocolo documentado. Nenhuma delas demonstra economia da sessão completa ou economia financeira. + +A Hlido informa **82/100** na avaliação do repositório; as evidências públicas acessíveis não validam essas razões de tokens. Veja a [issue #12](https://github.com/lucasrosati/claude-code-memory-setup/issues/12) e a [auditoria de evidências e o protocolo de benchmark, em inglês](./benchmarks/README.md) para origem, divergências de datas, pressupostos e limitações. + +O protocolo define entradas fixas, execuções pareadas com navegação de código e memória, critérios de correção, contabilização de cache, custos de construção do grafo e medições separadas por consulta e sessão. Inclui um [sumarizador offline de consumo](./benchmarks/summarize.py). O exemplo sintético verifica apenas a aritmética; não são apresentados novos resultados de desempenho nem execuções pagas. + --- ## Arquitetura Final diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000..a9d738d --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,54 @@ +# Token metrics: evidence audit and reproduction protocol + +Audit date: 2026-10-07. Public `main` inspected at +[`a89c275e139d25deb01529e6c7f1954844164ff7`](https://github.com/lucasrosati/claude-code-memory-setup/tree/a89c275e139d25deb01529e6c7f1954844164ff7). +Scope: tracked files, full available Git history, issue #12 and its comments. +No private project, vault, provider usage records, or paid model execution was available. + +## Where the claims came from + +| Claim | Earliest public evidence found | What can be reproduced now | +| --- | --- | --- | +| 71.5x fewer tokens **per session** | Initial README, [`ad8e4ef`](https://github.com/lucasrosati/claude-code-memory-setup/commit/ad8e4ef182cd5556579f9e45d8b7cedaa1d81b59), 2026-04-12 | Presence of the claim in history; no session usage ledger | +| ~20,000 tokens reading ~40 files versus ~280 querying a graph | Full PT-BR guide, [`884ee58`](https://github.com/lucasrosati/claude-code-memory-setup/commit/884ee58889d7e15745330e4cdca22242710b1e5b), 2026-04-12 | Arithmetic only: 20,000 / 280 = 71.428571…; approximately 98.6% less context under those assumptions | +| 499x per query; React + Supabase, 126 TypeScript files | Same full guide, `884ee58` | Claim and accompanying inventory only; no baseline/graph token counts, exact question or answer, tokenizer, model, or raw records | + +English translation arrived in `09ddc0f` on the same day; `a444d64` made it the main README. Later README edits inspected do not add a measurement protocol. The proximity of 71.5x to 20,000/280 suggests a connection, but the history does not prove the derivation. Rounded inputs cannot justify the extra precision. The 40-file illustration and 126-file report are different scenarios and must not be combined. + +The project source snapshot, graph, retrieval output and transcripts behind the inventory are not published here. All inventory counts remain author-reported. This is an absence-of-public-evidence finding, not evidence that the claims are false. + +## Independent review is not independent token validation + +[Issue #12](https://github.com/lucasrosati/claude-code-memory-setup/issues/12), opened 2026-07-20, reports 82/100 and offers a right of reply; it had no comments at audit time. Its body uses both hands-on language and an explicit repo-surface/not-hands-on description. The [Hlido passport](https://hlido.eu/passport/lucasrosati-claude-code-memory-setup/) retrieved on 2026-10-07 dates the review to 2026-07-16 and lists repository checks, not token measurements. That date differs from the September 24 date supplied with this audit request. The [review URL](https://hlido.eu/reviews/lucasrosati-claude-code-memory-setup/) could not be retrieved during this audit, so its exact current wording/publication date was not independently verified. Neither accessible source supplies reproduction data for 71.5x or 499x. Do not present the score as validation of either ratio. + +## Measurement boundaries + +**Retrieval payload:** count the exact text returned to the agent (including file names, wrappers and truncation) using a named, versioned tokenizer or provider count endpoint. A small graph query output is not the size of the entire graph. Bytes, characters and file counts are not tokens. This measurement excludes model reasoning, tool schemas, prompts and repeated context; label it a payload comparison, never session savings. + +**Query:** include every model request and tool step needed to answer one fixed question, including retries, graph retrieval and fallback source reads. Report input and output separately. Compare correct answers at equivalent completeness; a shorter but incorrect answer is a failed task, not a saving. + +**Session:** sum usage over all requests from a clean start through the identical ordered task list and final save. Include instructions, vault reads, conversation history resent to the model, graph queries, source edits, retries and save operations. Count each actual model request once. Never infer this total from a retrieval payload ratio. + +Report initial graph/vault construction and incremental updates separately, including any LLM calls, CPU time and elapsed time. AST-only extraction can avoid LLM generation calls; reading its output still consumes model context. Semantic extraction can cost tokens. For N sessions, report both steady-state usage and memory-arm usage plus build/update tokens amortized over an explicitly stated N. Physical token reduction is not dollar reduction: cached inputs, model rates, output tokens and subscription billing differ. + +## Reproducible paired benchmark (requires real data) + +1. Choose a public fixture repository and pin its full commit SHA. Publish its file inclusion/exclusion list, graph/vault snapshots and SHA256 hashes. Pin Python, Graphify, Claude Code, model identifier, tokenizer, OS and every extraction/query command with flags. Save the graph before querying. Capture actual commands rather than assuming a CLI version supports the guide's commands. +2. Freeze `tasks.json` before running either arm. Each entry must contain an ID, verbatim prompt, expected facts or patch/test acceptance criteria, permitted tools, maximum steps and timeout. Use at least structural, behavioral and editing tasks; include a task needing source fallback. Example fixed prompt: “Which functions call X, in which files?” Replace X with a real symbol and publish the source-grounded expected list. Have a reviewer score correctness against that list, blind to the arm where possible. +3. Baseline arm: source navigation with normal search/read tools, no generated graph or vault. Memory arm: same source and tools plus the pinned graph and vault. Keep model, prompts, limits and unrelated instructions identical. Publish the differing navigation instructions. Do not force baseline to read every file unless explicitly studying that artificial baseline; separately report whole-codebase reading as a stress case. +4. Run each task in a fresh isolated session for the query experiment. Run the full ordered list in fresh isolated sessions for the separate session experiment. Reset history and workspace changes between paired runs, preserve identical starting source, and alternate arm order over at least five paired repetitions. State random seed/model sampling settings and cache state; run cold and warm cache separately. Do not quietly exclude failures/timeouts; report success rate and failed-run usage separately. +5. Preserve request/response logs, tool outputs and provider usage objects with request IDs. Record build/update usage in a separate ledger. Normalize each actual model request to the schema below, without summing duplicate streaming events or snapshots. Document the provider/version-specific mapping. If `input_tokens` already includes cache reads/writes, subtract them before filling `input_uncached`; if it excludes them, retain it. Unknown or missing counts are **missing data**, never zeros. Zero means measured zero. Keep raw usage for auditing. +6. Summarize each ledger with the command below. Publish per-pair results, median and range across repetitions, sample size, correctness scores, payload counts and build/update totals. Compare matched successful pairs but also publish all failures and their usage. Do not pool query and session scopes or average ratios across unrelated tasks. A session ledger may include the same requests as a query ledger for a separate analysis, but those analyses must never be added together. + +Normalized JSONL row (one model request): `scope` (`query`/`session`), `pair` (task/repetition or session/repetition ID), `arm` (`baseline`/`memory`), `request_id`, and `usage` containing exactly four **disjoint** nonnegative integer fields: `input_uncached`, `input_cache_read`, `input_cache_write`, `output`. Logs remain responsible for proving completeness, correctness and provenance; the summarizer only validates pairing, basic counts and duplicate IDs within an arm/scope. + +```bash +# Offline plumbing check: invented counts, NOT project benchmark results +python3 benchmarks/summarize.py benchmarks/example.synthetic.jsonl +# Real captured data, once available +python3 benchmarks/summarize.py /path/to/normalized-usage.jsonl > /path/to/results.json +``` + +The synthetic example yields 140 versus 70 total tokens (2x, 50% reduction) solely to verify arithmetic. It does not invoke Graphify or a model and provides no evidence for this project's savings. The script reports `baseline / memory` and `100 * (1 - memory / baseline)`; undefined ratios are null. It retains cache categories so cost can be calculated separately with dated provider prices. + +**Current result:** no historical token-saving ratio is reproducible from the published artifacts. The arithmetic illustration is reproducible; empirical query and session savings require the inputs and captures above. No paid runs have been performed and no replacement performance numbers are claimed. diff --git a/benchmarks/example.synthetic.jsonl b/benchmarks/example.synthetic.jsonl new file mode 100644 index 0000000..34d41a0 --- /dev/null +++ b/benchmarks/example.synthetic.jsonl @@ -0,0 +1,2 @@ +{"scope":"query","pair":"synthetic-only","arm":"baseline","request_id":"b1","usage":{"input_uncached":100,"input_cache_read":20,"input_cache_write":10,"output":10}} +{"scope":"query","pair":"synthetic-only","arm":"memory","request_id":"m1","usage":{"input_uncached":40,"input_cache_read":10,"input_cache_write":10,"output":10}} diff --git a/benchmarks/summarize.py b/benchmarks/summarize.py new file mode 100644 index 0000000..c348829 --- /dev/null +++ b/benchmarks/summarize.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""Summarize paired, normalized usage ledgers; never invoke a model.""" +import argparse +import json +from pathlib import Path + +FIELDS = ("input_uncached", "input_cache_read", "input_cache_write", "output") + + +def summarize(records): + totals = {} + seen = set() + for row in records: + key = (row["scope"], row["pair"], row["arm"]) + if key[0] not in ("query", "session") or key[2] not in ("baseline", "memory"): + raise ValueError("scope must be query/session; arm must be baseline/memory") + identity = (*key, row["request_id"]) + if identity in seen: + raise ValueError("duplicate request within an arm and scope") + seen.add(identity) + usage = row["usage"] + if set(usage) != set(FIELDS): + raise ValueError("usage must contain exactly the four disjoint token fields") + if any(type(usage[f]) is not int or usage[f] < 0 for f in FIELDS): + raise ValueError("token counts must be non-negative integers") + bucket = totals.setdefault(key, dict.fromkeys(FIELDS, 0)) + for field in FIELDS: + bucket[field] += usage[field] + results = [] + for scope, pair in sorted({key[:2] for key in totals}): + arms = {} + for arm in ("baseline", "memory"): + if (scope, pair, arm) not in totals: + raise ValueError(f"unpaired measurement: {scope}/{pair}/{arm}") + arms[arm] = totals[(scope, pair, arm)] + baseline = sum(arms["baseline"].values()) + memory = sum(arms["memory"].values()) + results.append({"scope": scope, "pair": pair, "usage": arms, + "total_tokens": {"baseline": baseline, "memory": memory}, + "baseline_over_memory": baseline / memory if memory else None, + "reduction_percent": 100 * (1 - memory / baseline) if baseline else None}) + if not results: + raise ValueError("empty ledger") + return results + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("ledger", type=Path) + args = parser.parse_args() + try: + records = [json.loads(line) for line in args.ledger.read_text().splitlines() if line.strip()] + print(json.dumps(summarize(records), indent=2, allow_nan=False)) + except (ValueError, KeyError, TypeError, OSError) as error: + parser.exit(2, f"Invalid ledger: {error}\n") + + +if __name__ == "__main__": + main()