You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 41fa1e8
Browse filesBrowse the repository at this point in the historyBrowse files
fix(tracing): record standard streaming TTFT telemetry (#1194)
* fix(tracing): record standard streaming TTFT telemetry
Add the actual SDK streaming flag and seconds-valued TTFT to per-request
LLM spans, and record the existing TTFT histogram once at first output.
Preserve legacy timestamps, omit absent observations, and keep metrics
independent of usage and trace sampling.
Add regression coverage and synchronize bilingual tracing documentation.
Validation after rebase: 156 affected tests and pre-commit pass. Default
regression: 8044 pass; one pre-existing Markdown MIME test still fails.
Change-Id: If7113949389b8c5707feb20b8c67479700f0f19b
* fix(tracing): replace custom timing fields with standard telemetry
Represent HTTP, Runner and SDK boundaries with native span timestamps.
Remove custom timing attributes and first-token events without aliases.
Use standard error status and bounded error types, and retain once-per-
request TTFT attributes and histogram recording.
Update bilingual documentation and regression tests. After synchronization,
294 affected tests and pre-commit pass. Full regression has 8053 passing
cases and the previously reproduced unrelated Markdown MIME failure.
Change-Id: I3b4f2eacb84bb1c378915cb244e68497190738cd
* chore(docs): remove root design specs and opt out of regeneration
Remove prd-spec and specs; keep plans and verification summaries in conversation or PR descriptions. Require an explicit user request before creating standalone design/spec documents.
Change-Id: Ibabda377bee6fc64b844dfe4164febb4fa1d6cd5
Validation: all pre-commit hooks passed; YAML secret scanner unit tests passed (4 tests). Documentation and workflow guidance only; no runtime code or public API changes.
Copy file name to clipboardExpand all lines: AGENTS.md
+10-8Lines changed: 10 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,21 +5,23 @@ VeADK includes a Python SDK, CLI/runtime integrations, and AgentKit Studio with
5
5
## Language and document ownership
6
6
7
7
- Keep this repository-wide development standard in English. Use English for new code identifiers, comments, and docstrings; user-facing text follows the owning area's localization rules.
8
-
- Keep durable change intent, scope, tasks, evidence, and acceptance in the PR/MR description, linked issue, external design document, or affected user documentation when the change needs long-lived context. Do not create repository-local design or component-spec files by default.
8
+
- Do not create standalone PRD or component-spec files, or recreate the root `prd-spec/` and `specs/` directories, as part of routine development. Create such documents only when the user explicitly requests them; generic skills or workflow templates do not make them mandatory.
9
+
- Discuss change-specific intent, scope, design decisions, and acceptance criteria in the conversation or PR description. Update existing maintained documentation when an affected interface or behavior needs clarification.
10
+
- When updating existing bilingual documentation, keep its language counterparts semantically equivalent in the same change, including identifiers, commands, examples, limits, and acceptance criteria.
9
11
- Before modifying frontend code, read `frontend/SPEC.md`; also apply it to its supporting backend interfaces as specified there. It remains the Studio development standard and is not replaced or translated by this document.
10
12
- Preserve existing user-documentation conventions: the documentation site uses Chinese `.mdx` and English `.en.mdx`; bilingual module/example READMEs use English `README.md` and Chinese `README.zh.md`.
11
13
12
14
## Required development workflow
13
15
14
16
1.**Inspect and agree on scope.** Read the affected implementation, tests, and contracts. When the user describes a requirement, propose an implementation plan and wait for the user to be satisfied before making edits. Distinguish verified facts from assumptions.
15
-
2.**Plan before implementation.**For features, refactors, and bugfixes, size the change against the affected contracts and risk. Record background/evidence, goals, non-goals, affected files, tasks, tests, risks, and acceptance criteria in the PR/MR description, linked issue, external design document, or user documentation when the change needs durable context.
16
-
3.**Assess contracts.**Identify changes to ownership, APIs, configuration, state/data, events, permissions, security, compatibility, runtime behavior, or observability. Record the contract impact or no-impact rationale in the PR/MR description, linked issue, external design document, or user documentation as appropriate. Do not backfill unrelated contracts.
17
-
4.**Clear scope review.**Review plan consistency, feasibility, boundaries, errors, security, compatibility, and testability. Resolve blockers and record user approval before modifying production code or tests.
18
-
5.**Implement against the approved design.**Map changes to requirements/tasks, write a failing regression or contract test first, then implement and refactor within scope. Contract or scope changes return to design review before implementation proceeds.
17
+
2.**Plan before implementation.**Explain the implementation approach, affected areas, risks, and verification plan in the conversation. Scale the detail to the change; no separate design document is required.
18
+
3.**Assess component contracts.**Check affected ownership, APIs, configuration, state/data, events, permissions, security, compatibility, runtime behavior, and observability. Update relevant existing documentation when necessary; do not generate separate component specs or no-impact reports by default.
19
+
4.**Review the approach.**Check feasibility, boundaries, errors, security, compatibility, and testability before editing. Resolve blockers and obtain user agreement for material scope changes. No standalone design review artifact or PRD/spec approval gate is required.
20
+
5.**Implement within the agreed scope.**Write a failing regression or contract test first for bugfixes, then implement and refactor within scope. Reassess the plan when contracts or scope change.
19
21
6.**Verify and review.** Run affected tests first, then required repository and component gates. Review both implementation correctness and edge cases, resource handling, security, test quality, and contract alignment. Fix blocking findings and rerun affected checks.
20
-
7.**Reconcile and deliver.**Synchronize affected user docs/examplesand required generated artifacts. Record actual verification commands, results, unverified areas, and remaining risks. All agreed tasks and acceptance criteria must be satisfied before declaring completion; deferred scope requires user agreement.
22
+
7.**Reconcile and deliver.**Update affected existing user docs/examples, their applicable language counterparts, and required generated artifacts. Summarize actual verification commands, results, unverified areas, and remaining risks in the conversation or PR description. All agreed tasks and acceptance criteria must be satisfied before declaring completion; deferred scope requires user agreement.
21
23
22
-
Pure documentation, translation, or formatting changes that preserve runtime behavior and public contracts do not need a separate change design. They still require an agreed approach, applicable bilingual updates where the owning document convention requires them, and proportionate checks. This exception includes maintaining these development guidelines; it does not exempt code or behavioral changes.
24
+
Documentation, translation, formatting, and development-guideline changes require an agreed approach, applicable updates to existing language counterparts, and proportionate checks. They do not require a separate design document.
23
25
24
26
## Implementation boundaries
25
27
@@ -57,7 +59,7 @@ Use the repository-local Python environment and current `pyproject.toml`, `pytes
57
59
- Before committing, run `uv run --extra dev pre-commit run --all-files` and unit tests after synchronizing the branch as described below. Pre-commit includes Ruff and secret scanning; it does not replace browser, integration, or runtime verification.
58
60
- Prefer targeted checks during iteration. Expand to affected integration/E2E and regression gates for shared or high-risk changes and final delivery; explain any omitted required check.
59
61
- Record check outcomes as `pass`, `fail`, `blocked`, `not_run`, or `not_applicable`, with reasons for the last three. Include the tested revision/diff scope and execution date. Skipped tests do not prove the relevant behavior.
60
-
- Keep change-specific review and verification summaries in the PR/MR description, linked issue, external design document, or affected user documentation as appropriate. Separate evidence artifacts are optional when useful; OpenSpec and a multi-report hierarchy are not prerequisites.
62
+
- Keep change-specific review and verification summaries in the conversation or PR/MR description by default, with links to existing issues or external documentation when useful. Do not generate PRD/spec files, OpenSpec records, or a multi-report hierarchy unless the user requests them.
Copy file name to clipboardExpand all lines: docs/content/docs/framework/tracing.en.mdx
+15-12Lines changed: 15 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -189,20 +189,23 @@ In a multi-agent scenario, spans are linked through `parent_span_id`, showing th
189
189
</Callout>
190
190
191
191
192
-
## Request, model, and first-token timestamps
192
+
## Request spans and streaming TTFT
193
193
194
-
VeADK's default LiteLLM and Ark Responses adapters add the following extension attributes to model spans when OpenTelemetry tracing is enabled. UTC Unix nanoseconds are used throughout.
194
+
VeADK uses native OpenTelemetry span start/end times to describe execution boundaries. VeADK AgentKit/Harness HTTP applications create an `http_request` server span at ASGI ingress, before body parsing, and end it when the application finishes. Direct VeADK Runner execution creates a `runner_request` span; a Runner invoked inside the HTTP request reuses that request scope. These boundaries exclude client/network/gateway delays before the application receives the request.
195
195
196
-
| Attribute | Meaning |
196
+
Each LiteLLM Chat or Ark Responses SDK request creates a `model_request` client span under the owning model span. Its native start time is the same local SDK-request timestamp used to calculate TTFT; it is not provider inference start. VeADK-issued retries/fallbacks create separate spans. Hidden provider/LiteLLM retries remain within the observed SDK call.
197
+
198
+
| Attribute | Type / meaning |
197
199
| --- | --- |
198
-
| `veadk.request.start_time_unix_ns` | HTTP ASGI entry in VeADK AgentKit/Harness apps, or direct VeADK Runner execution entry. |
199
-
| `veadk.request.start_source` | `http` or `runner`; an inherited HTTP timestamp takes precedence. |
200
-
| `veadk.model.start_time_unix_ns` | Start of one adapter SDK request, not provider inference start. |
201
-
| `veadk.model.streaming` | Actual model SDK request mode, independent of HTTP SSE transport. |
202
-
| `veadk.model.first_token_time_unix_ns` | Receipt of the first nonempty generated content delta. |
203
-
| `veadk.model.first_token_type` | `text`, `reasoning`, or `tool_call`. |
204
-
| `veadk.model.first_token_status` | `observed`, `non_streaming`, `no_output`, or `unsupported`. |
200
+
| `gen_ai.span.kind` | `llm` on an SDK request span. |
201
+
| `gen_ai.request.model` | Actual requested model. |
202
+
| `gen_ai.request.stream` | Boolean actual SDK streaming mode, independent of HTTP SSE or Runner configuration. |
203
+
| `gen_ai.chat_completions.streaming_time_to_first_token` | Floating-point seconds from SDK request start to its first generated delta. |
204
+
205
+
The first nonempty text, reasoning, or tool-call delta records TTFT once. Metadata, empty deltas, usage, heartbeats, and completion-only aggregates do not count. Non-streaming, no-output, or uninstrumented paths omit TTFT; unknown SDK mode is not inferred from Runner configuration. One chunk can contain multiple tokens, so this is an SDK-observed duration, not a provider-side per-token timestamp.
206
+
207
+
The same value is recorded once in the `gen_ai.chat_completions.streaming_time_to_first_token` Histogram, unit `s`, using the existing global MeterProvider. Dimensions are `gen_ai.request.model` and `gen_ai.operation.name=chat`. Metrics do not depend on token usage or trace sampling; export requires a configured metrics pipeline. Negative wall-clock intervals are omitted with a diagnostic; genuine measured zero is valid.
205
208
206
-
First-token capture excludes role/id/signature metadata, empty deltas, usage, heartbeats, and completion-only aggregates. It does not require token usage or content tracing. One chunk may contain several tokens, so this is an SDK-observed first-output timestamp. Non-streaming calls have no first-token timestamp; completion time is never substituted.
209
+
Failed model requests set native Span Status to `ERROR` and `error.type` to `llm_error`, `timeout`, or `cancelled`. Request spans use `request_error`, `timeout`, or `cancelled`; HTTP 5xx responses use `http_error`. Explicit generator closure is not an error. Already observed TTFT survives interruption, and exception text is not added by this instrumentation.
207
210
208
-
Each SDK request creates a `model_request` child span; the owning model span also receives the latest attempt's fields. VeADK-issued retries and fallbacks create separate attempts. Internal LiteLLM/provider retries remain within the observed SDK request (`veadk.model.request_scope=sdk_request`). Unsupported external/custom engines are explicitly marked; invocation span start is not used as model start. Applications outside the VeADK HTTP factories must install the request timing middleware or use VeADK Runner to establish a request timestamp. Client-render timing and TTFT/TPOT histogram reporting are separate capabilities.
211
+
The instrumentation does not export custom `veadk.request.*` or `veadk.model.*` attributes or a custom first-token event. Start and first-output values remain internal calculation state. This replaces the previous custom timing fields without compatibility aliases. Client-render timing and TPOT histogram reporting remain outside this feature.
0 commit comments