gap: align client TTFT, admission, and first-response timing boundaries

[simone-chen]

This was generated by AI during triage.

Problem and affected scope

• Client TTFT includes request ingress, tokenization/dispatch, admission and executor queueing, prefill execution, and first-response delivery. Replay includes modeled queueing but does not establish coverage of every real serving boundary.
• Separate these intervals before attributing a client-versus-replay difference to GPU kernels or adding a fixed latency offset.
• Observed cases: GPT-OSS-120B / TensorRT-LLM PyTorch 1.3.0rc14 / B200 TP2 C4, and DeepSeek-V4-Pro / SGLang 0.5.14 / B200 TP8 C1/2/8/16.
• Triage state: needs maintainer review. Boundary contributions are confirmed in the measured cases; the detailed causes of later admission tails and the remaining forward residual are open.

Existing evidence

• GPT-OSS uninstrumented mean TTFT is 48.8565 ms versus 16.5326 ms predicted. Separate native-metrics runs average 50.5588 ms, 3.48% higher than the uninstrumented baseline.
• Native averages: server-to-executor 2.6915 ms, executor queue 5.6543 ms, first schedule to first token 34.9355 ms, and first token to server response 2.0356 ms. Client mean minus server mean is 5.2418 ms, an aggregate difference rather than ID-paired network time.
• Thus a frontend constant cannot explain the whole GPT-OSS gap. Forward execution belongs to gap: TRT-LLM GPT-OSS TTFT omits eager host dispatch and rank waiting #277 and the execution-timing contract follow-up; do not count that same delay twice.
• A later SGLang C16 request has 1340.33 ms TTFT, including 615.83 ms before scheduler admission and 400.71 ms in the scheduler queue. Tokenization takes 3.08 ms. No matching new JIT artifacts were found during that burst, but the exact host/startup cause remains unresolved.
• The historical SGLang config880 mean/median split alone did not locate the delay. New 0.5.14 results do not prove causality in the unavailable original nightly. Stream interval 30 emits token 1 immediately in the inspected path; an unconditional 29-token TTFT penalty is not justified.

Next steps and acceptance

• Preserve request IDs across client, server, executor, admission, forward and first-response timestamps, with explicit clock domains and join coverage.
• Reproduce later warm C16 bursts with lightweight stage timing and matched uninstrumented controls; distinguish host preparation, admission policy and scheduler queueing.
• Compare matched arrival policies and batch composition against replay, separately from the forward timing correction.
• Report mean, median and tails over identical request populations. Keep incomplete joins, failed runs and observer effects visible.
• Document which boundaries AISim predicts and which remain outside its qualified domain. Validate any new serving model independently without fitting an E2E constant.

Evidence and slides

• TTFT-06: client and admission boundaries.
• GPT-OSS native-stage report.
• GPT-OSS measured stages.
• SGLang baseline and later burst, internal.
• Cause catalog: GAP-014. Related gap: TRT-LLM GPT-OSS TTFT omits eager host dispatch and rank waiting #277, PR fix: correct TRT-LLM MLA labels and GPT-OSS MXFP4 profiles #264, and the separate warmup/JIT follow-up.