data-sources.md

Data sources

There are three tiers of Claude usage data. The grouping is by whether they perturb the quota, not by mechanism: any source that piggybacks on the user's normal activity is "passive" and shares Tier 1, regardless of whether it reaches the trayapp via file tailing or via a Stop-hook POST. Tier 2 actively reads the dashboard but only when the user is already there. Tier 3 actively scrapes on a schedule and is reserved for cases where the cheaper tiers fall short.

Tier 1 — Passive observation

Status: primary. Two paths, same tier. Both ride on activity the user is already performing, so neither perturbs the session (5-hour) window or the weekly quota.

1a. Session JSONL tailing (host)

Claude Code writes per-session JSONL files to ~/.claude/projects/<encoded-project-path>/. Each line is a JSON object representing one event in the conversation. Assistant messages include a usage block with token counts (input, output, cache creation, cache read), and Claude Code records dollar-equivalent cost when available.

The tailer:

• Uses fsnotify (or polling fallback on filesystems where fsnotify is flaky) to detect appends.
• Reads new bytes from the last persisted offset for each file. The backup poll walks every transcript under the projects directory, so it compares each file's size with its offset first and opens only the ones that grew.
• Parses each line; ignores lines without a usage block.
• Inserts a row into usage_events keyed by (session_id, message_id) so replays are idempotent. The line is stored in raw_json without its conversation text (see "raw_json trimming" in docs/data-model.md).
• Persists offsets so a trayapp restart resumes from the right place without rereading history.

1b. Stop hook + transcript POST (containers without shared ~/.claude)

When a container does not bind-mount ~/.claude from the host, the host tailer cannot see its activity. Instead, a Stop hook installed in the container runs clusage-cli log --from-hook after each turn, reading Claude Code's hook payload from stdin, opening the transcript file referenced therein, and POSTing each new assistant message's usage block to the host server. See docs/container-cli.md for the wiring.

This path is also passive in the sense that matters: it observes work the user chose to do, never makes a request to provoke a measurement, and never starts a new 5-hour window. It just delivers the same data over a different transport.

The host-side tailer (1a) and the container-side Stop hook (1b) deduplicate against the same (session_id, message_id) key, so both can coexist on a host that runs some sessions natively and some in unshared containers.

Pros

• Zero perturbation. No requests issued, no wakeups triggered.
• Authoritative per-message. The same numbers Claude Code itself reports.

Container coverage — pick one path per container

For each container, decide between two equally valid options:

• Bind-mount ~/.claude from the host (e.g. -v ~/.claude:/root/.claude). The host tailer (1a) sees the container's sessions for free. Zero in-container setup.
• Install the CLI and Stop hook in the container image. The container reports its own usage via 1b. No mount needed.

Both paths land in the same DB and dedup against the same key. A user can use one for some containers and the other for others. The recommended choice for a container-heavy workflow is whichever requires less change to existing images.

Cons / caveats

• Assumes file format stability. The JSONL schema is not a public contract. The tailer must be defensive and log parse failures loudly.
• Quota baseline is not in the JSONL. The files give us dollar consumption per message, not the plan-limit % used number Anthropic shows in the UI. We need a snapshot from somewhere else to anchor the baseline.
• Container path coverage. See above; the unshared-container case is common enough that it shapes the install flow, not just an edge case.

Second root: Cowork ("local agent mode") sessions

The desktop app's Cowork feature runs Claude Code sessions in a sandboxed VM/background mode. Each Cowork session gets its own private, nested .claude home several directories under %APPDATA%\Claude\local-agent-mode-sessions\<workspace>\<id>\local_<uuid>\ — its own .claude/projects/<encoded-cwd>/<session>.jsonl, distinct from the user's real ~/.claude/projects. The schema is identical (same usage block), but the primary tailer above never reaches these files because they aren't under its root at all.

A second ingest.Tailer instance is started against claude.cowork_sessions_dir (default %APPDATA%\Claude\local-agent-mode-sessions) alongside the primary one. Because the tailer's poll pass does a full recursive filepath.Walk, this second root picks up newly-created Cowork sessions automatically — no per-session registration needed. Both tailers share the same DB and dedup key, so there's no risk of double-counting. This is still Tier 1a: no hook, no perturbation, just a second recursion root.

Regular desktop-app Claude Code (the embedded pane, as opposed to Cowork) writes straight into the normal ~/.claude/projects, indistinguishable from a terminal session — it needs no special handling.

Recovering from a silent extraction failure

The tailer's per-file byte offset only records how far a file was read, not whether the lines in that range actually produced a usage_event. A parser bug — silently returning "no usage block" for a schema it misreads — can leave an offset sitting at EOF for content that was never successfully recorded, and a normal poll has nothing new to read past that offset, so it can't self-heal. There's also no per-line "was this ingested" marker, so a selective repair isn't possible without already knowing which files/sessions are affected.

clusage-cli reimport (see docs/container-cli.md) is the recovery tool for this: it wipes every persisted tailer offset and re-walks every root from byte 0, leaning entirely on the UNIQUE(session_id, message_id) dedup to discard whatever's already correctly recorded. Slow and wasteful by design — it re-reads everything — but that's the acceptable cost of guaranteed recovery without naming files in advance. Not routine maintenance; use it once after fixing a bug like this, not on a schedule.

Schema fragility mitigation

The parser should:

• Treat unknown fields as opaque.
• Require only usage.input_tokens and usage.output_tokens. Everything else is bonus.
• Surface parse-error rate in the dashboard as a health metric.

Tier 2 — Userscript snapshots

Status: secondary, used for baseline anchoring.

A Tampermonkey/Violentmonkey userscript on claude.ai/* reads the dashboard's displayed quota numbers and POSTs them to the local server. See docs/userscript.md for the implementation plan.

The userscript performs freshness-driven dedup on the client side: a POST is emitted only when at least one meaningful-change signal has fired since the last successful send (percent moved, "Resets …" text changed, limbo appeared/disappeared, five minutes elapsed while a session window is active, or — in limbo — the "Last updated" age decreased). Identical-value re-observations are intentionally suppressed; a 60-second backstop catches missed observer events but is gated by the same dedup, so stable plateaus produce no extra rows. Every POST also carries a continuous_with_prev boolean (see docs/userscript.md) the server uses for write-time plateau compaction.

Pros

• Authoritative for baselines. This is what Anthropic's UI shows, by definition.
• Trivial to install. A single .user.js file, install via userscript manager.
• Free recalibration. Fires whenever the user naturally visits the dashboard.

Cons / caveats

• Only fires when the page is open. Cannot be relied on for a fixed cadence.
• DOM-dependent. Anthropic can change the page layout at any time. The script must fail soft — logging a parse error rather than corrupting the DB.
• No background acquisition. If the user goes a week without opening the dashboard, baselines drift and we depend entirely on Tier 1 plus the previous baseline.

When the userscript is the answer

• First-time setup (need any baseline at all).
• Recovering from a multi-day gap in passive data.
• Cross-checking when the slack indicator looks suspicious.

Limbo signal: session_active / weekly_active

When Anthropic's UI shows "Starts when a message is sent" instead of a "Resets …" hint on the Current session row, no 5-hour window is currently open. The same UI treatment applies independently to the weekly "All models" / "This week" row. The userscript detects each cue and reports it on the snapshot POST as session_active: false and/or weekly_active: false. For each field:

• Field absent ⇒ unknown. The script could not positively decide, so it does not include the key. The server records NULL.
• <field>: false ⇒ positively-detected limbo. The script matched the "Starts when a message is sent" hint on that row.
• <field>: true is never sent by the userscript. Absence already covers "the script saw a normal Resets hint"; the server must not infer truth from omission.

This three-valued convention lets downstream consumers distinguish "we have no information" from "we know there is no active window." Both fields are independent. See docs/no-active-session.md for what the trayapp does with each signal.

Tier 2b — OAuth usage endpoint

Status: implemented, opt-in (oauth_usage.enabled, default false).

GET https://api.anthropic.com/api/oauth/usage, authenticated with the access token Claude Code already stores in ~/.claude/.credentials.json. Returns the same account-scoped quota Anthropic renders on the usage page, as structured JSON. See internal/oauthusage.

It shares Tier 2's job — anchoring baselines — but not its dependency on a browser tab being open, which is why it sits beside rather than above it.

Why this is not Tier 0

The rejection of clusage polling turns on perturbation: invoking Claude Code to read the figures starts a 5-hour window if none is active. This endpoint reports usage rather than consuming it. Six consecutive reads left session utilization unchanged at 3% with no window opened. Nothing is inferred, no tokens are spent, and no window is created — so unlike Tier 0 it is admissible on a schedule.

What it gives that the DOM cannot

• limits[] is self-describing. Each entry carries its own kind (session, weekly_all, weekly_scoped) and, for scoped rows, scope.model.display_name. This replaces the userscript's entire heading-prefix / row-label / positional-fallback apparatus. A newly-scoped model arrives as another weekly_scoped entry instead of breaking the parse.
• Exact reset boundaries. resets_at is RFC3339, so none of userscript/lib/resets.js is needed — no weekday-name parsing, no nearest-occurrence heuristic, no relative-form arithmetic.
• No staleness. Figures are current at request time, so there is no "Last updated: N minutes ago" delta to back-date by.
• Usage credits in dollars. extra_usage / spend report real spend. The per-window limit_dollars / used_dollars are null on the observed plan, so the dollar-per-percent exchange rate remains unpublished and the reconciliation problem below is unchanged.

Caveats

• The token expires in about 8 hours and is refreshed only when Claude Code itself runs. A host idle overnight holds an expired one, and this source goes quiet until the next session. The poller detects that locally from expiresAt and skips the request rather than sending one that will 401.

  It deliberately does not refresh the token itself. If refresh tokens rotate single-use, losing a race against Claude Code's own refresh would log the user out of their primary tool — a poor trade for filling a mostly-idle gap. claude auth status does not help either, as it reads local state and never refreshes.

  What it can do, opt-in (oauth_usage.refresh_with_claude), is have Claude Code do the refresh: running claude -p /usage refreshes the token as a side effect, so Claude Code's own refresh logic stays the only writer. It is attempted once per stale episode and re-armed only by a successful poll, so a run that does not help is never repeated in a loop (see docs/configuration.md). Early observation was that this run opened a 5-hour window when none was active; later observation suggests it no longer does. That is Claude Code's behavior, not this app's, so it may change again. claude setup-token remains the escalation if uninterrupted coverage is ever needed.

• A 401 is not a broken source. Both observed conditions ("token has expired", "token is invalid") return 401 with error.type == "authentication_error". The poller reports these as temporarily unavailable and never routes them to the parse-error health signal, which must keep meaning "the scraper broke, go fix it". Classification keys off status and error.type, never the prose message.

• resets_at carries jittery sub-second precision. The server recomputes it per request: across two reads of the same window the whole seconds are stable but the microseconds differ, and they differ between entries within a single response. Values are truncated to the second on ingest; comparing raw values would report a changed window on every poll.

• is_active does not mean "this window is open." It read false on the session row while session utilization climbed 3% → 7%, and only the highest-utilization row was true. Its meaning is unconfirmed, so it is not mapped to session_active / weekly_active. Those stay absent — "unknown" under the tri-state convention — and the userscript remains the only authority on limbo. Running both sources side by side is what will settle this.

• Undocumented endpoint. The top-level response carries a couple of dozen keys under internal codenames (tangelo, iguana_necktie, nimbus_quill, …) that churn as features ship. Only limits[] is decoded; unknown kinds are skipped rather than guessed at. Rate limits are unpublished, hence the deliberately unhurried 3-minute default and the 30-second floor.

• Only Authorization is required. A User-Agent and anthropic-beta were both verified unnecessary — five header variants sent back to back with the same token, including one with no User-Agent at all, returned identical payloads. The client sends its own honest User-Agent and does not impersonate Claude Code.

Relationship to Tier 2

Enabling this does not disable the userscript. Both may run; their rows carry different source values (oauth vs userscript) and reach the database through the same Server.RecordSnapshot path, so validation, raw-JSON retention, metrics and window derivation cannot drift between them. They are never averaged — see "Cross-source reconciliation" below.

Checking it works

GET /api/oauth/status reports the poller's own health, separately from the shared snapshot counters, which cannot tell the sources apart:

{"enabled": true, "available": true, "credential_stale": false,
 "last_attempt": "2026-09-24T14:00:00Z", "last_success": "2026-09-24T14:00:00Z",
 "last_error": "",
 "refresh_enabled": true, "refresh_armed": true,
 "last_refresh_attempt": null, "last_refresh_error": ""}

• enabled: false means oauth_usage.enabled is off. That is a normal configuration, so the endpoint still answers 200.
• available reflects the most recent attempt only.
• credential_stale: true is the routine, self-healing case above: the token is waiting for Claude Code to refresh it. available: false without it is the case that needs attention, and last_error says why.
• last_attempt and last_success are null until first set. last_success is kept through later failures, so the gap between the two is how long the source has been down.
• refresh_* describe oauth_usage.refresh_with_claude. refresh_armed: false while refresh_enabled is true means an attempt did not help and no other will be made until a poll succeeds; last_refresh_error says why.

The poller polls once at startup, so a restart reports a result within seconds.

TestLiveEndpoint in internal/oauthusage is the schema-drift canary; it is skipped unless OAUTH_USAGE_LIVE=1.

Tier 3 — Headless browser scrape (deferred)

Status: not built. Escalation only.

Use Playwright (or similar) from the trayapp to drive a headless Chrome against claude.ai, log in via a stored profile, and read the same DOM the userscript reads.

Why deferred

• Profile locking. Chrome holds an exclusive lock on the profile directory while it's running, so the user cannot have their normal browser open at the same time. Workaround is a copy of the profile, which then needs periodic refresh as cookies rotate.
• Login fragility. Sessions expire. Re-authenticating headlessly is hard if MFA is on.
• Anti-bot risk. Anthropic could plausibly fingerprint headless traffic.
• Maintenance cost. Selectors break.

When to escalate to Tier 3

Only if Tier 1 + Tier 2 in practice produce baselines stale enough that the slack indicator becomes untrustworthy. This will probably not happen for a user who naturally checks the dashboard a few times a week.

Tier 0 (rejected) — clusage polling

clusage works by invoking Claude Code itself to read the current usage figures. This starts a 5-hour window if one is not already active, perturbing the very quantity being measured. It is therefore unsuitable as an automated data source.

clusage remains useful for ad-hoc manual reads where the user wants a definitive number right now and is willing to start a window to get it.

Cross-source reconciliation

Tier 1 measures dollar consumption per message; Tier 2 measures the plan-limit percentage Anthropic chose to display. They are in different units (dollars vs. percent) and Anthropic does not publish the exchange rate (the per-window dollar quota that backs the % bar). Reconciling the two is therefore an open problem: we can detect direction (more dollars in Tier 1 → expect higher % in Tier 2) but not absolute magnitude.

For now the system stores both and surfaces them side-by-side. Closing the loop — inferring the dollar-per-percent rate from observed pairs over time, then alerting on divergence — is follow-up work.

The system never silently averages or "splits the difference." Disagreement is data.