feat(supervisor): accept OTLP/HTTP JSON in the agent trace relay #5319

User Story

I use the supervisor OTLP agent relay on OpenShift (Kubernetes driver, 0.1.7-pre.4) to send Claude Code traces from a sandbox to MLflow and Tempo. I need the relay to accept OTLP/HTTP JSON, so that agents whose exporters only send JSON get their traces relayed and tagged with openshell.sandbox.id like everyone else.

Problem Statement

The relay accepts POST /v1/traces only with an application/x-protobuf body and answers 415 to JSON. OpenShell injects OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, but that only helps when the agent's exporter honors the variable.

Some don't. Claude Code's detailed tracing mode (ENABLE_BETA_TRACING_DETAILED=1 with BETA_TRACING_ENDPOINT), which adds the model's response text to the spans, always exports to <BETA_TRACING_ENDPOINT>/v1/traces with the OTLP/HTTP JSON exporter and ignores OTEL_EXPORTER_OTLP_PROTOCOL. The same is true of the OpenTelemetry JS exporter-trace-otlp-http package in general. Pointed at the relay, these traces are refused and never reach the collector.

At the default log level the failure is silent. The 415 path logs only at debug and is not counted in the relay's rejected counter, so the periodic OTLP relay summary shows nothing wrong.

Impact / Why This Matters

• Agents with JSON-only exporters can't use the relay at all. The workaround is exporting directly to the collector, which needs a network policy rule for the collector, exposes its address inside the sandbox, and loses relay attribution: the agent would have to set openshell.sandbox.id itself and could set any value.
• For Claude Code, the richer trace content (model output per LLM request) is only available in the mode that exports JSON. With the relay, operators get tool calls but not the model's text between them.
• The failure is hard to diagnose: the agent run succeeds and the collector simply receives nothing.

Proposed Design

The relay accepts POST /v1/traces with Content-Type: application/json (OTLP/HTTP JSON encoding) in addition to application/x-protobuf. JSON requests get the same treatment as protobuf requests: the same size and resource limits, the same openshell.sandbox.id and openshell.telemetry.source=agent enrichment, and forwarding to the collector over the configured protocol as today. Responses use the request's content type, as the OTLP/HTTP spec requires. Compression stays unsupported for both encodings.

No configuration change. The injected OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf default stays as it is.

Suggested UX (if applicable)

Nothing new to configure. With the agent relay enabled, a JSON exporter pointed at the injected endpoint just works:

openshell sandbox exec --name demo -- bash -c \
  'ENABLE_BETA_TRACING_DETAILED=1 BETA_TRACING_ENDPOINT=$OTEL_EXPORTER_OTLP_ENDPOINT claude -p "..."'
# The trace arrives at the collector with openshell.sandbox.id=<id>,
# and the llm_request spans carry the model output.

Acceptance Criteria

• A POST /v1/traces with an uncompressed OTLP/HTTP JSON body is accepted, enriched with openshell.sandbox.id and openshell.telemetry.source=agent, and reaches the collector with all spans and attributes intact.
• Trace and span IDs in JSON (hex-encoded per the OTLP JSON mapping) arrive at the collector with the same values the agent sent.
• JSON requests are subject to the same limits and status codes as protobuf requests (413, 408, 503 with Retry-After), with error bodies in the request's content type.
• Protobuf requests behave exactly as today.
• Claude Code with detailed beta tracing, pointed at the relay endpoint, produces a trace in the collector.
• The relay section of docs/how-it-works/gateways/configuration.mdx lists JSON as accepted.

Alternatives Considered

• Configure the agent for protobuf: not possible when the exporter hard-codes JSON, as Claude Code's detailed tracing does. Asking each agent vendor to change their exporter doesn't scale.
• Export directly to the collector: works with a network policy rule, but the traces carry no relay attribution, the agent can forge openshell.sandbox.id, and the collector address becomes visible inside the sandbox. That defeats what the relay is for.
• Convert in the collector: not possible, since the relay refuses the request before anything is forwarded.
• Supervisor middleware or gateway interceptors: the relay address is served by the supervisor itself inside the sandbox boundary and not proxied upstream, so as far as I can tell egress middleware never sees these requests. Gateway interceptors act on gateway gRPC traffic, not on the agent's OTLP exports.

Agent Investigation

• The receiver rejects anything but application/x-protobuf with 415 and an unsupported content type reason (crates/openshell-supervisor/src/otlp_relay/receiver.rs, around line 238). rejected() logs at debug and does not touch the relay counters. RelayCounters::record_rejection only counts buffer-full 503s.
• Enrichment works on the protobuf wire format (otlp_relay/enrichment.rs), so JSON would need to be transcoded to protobuf before enrichment (or enriched as JSON). opentelemetry-proto (already a workspace dependency, 0.33) has a with-serde feature implementing the OTLP JSON mapping, including hex trace and span IDs. This is a possible approach, not a requirement.
• Claude Code 2.1.316: with ENABLE_BETA_TRACING_DETAILED=1, the bundled code constructs the trace exporter as new OTLPTraceExporter({url: ${BETA_TRACING_ENDPOINT}/v1/traces}) from the http/json exporter module, and logs the same way to /v1/logs (found by inspecting the binary in the sandbox image).
• Observed on a ROSA cluster with 0.1.7-pre.4 and the sandbox at log level debug: a Claude Code run with detailed tracing produced one OTLP relay discarded an agent export ... signal=logs line, OTLP relay rejected request debug lines, and no new trace in the collector. The same run without detailed tracing (protobuf) produced a trace as expected.