[Bug] BoTorch adaptive search aborts with a LogNormalPrior error when the DSP kernel lengthscale becomes zero

[shm197]

Description

During a max-throughput-ttft-sla adaptive search, AIPerf aborted after completing 10 search points. The failure occurred while fitting the Gaussian process to propose the next candidate, rather than during an inference request.

The traceback ends in LogNormalPrior.log_prob() because the lengthscale passed to the prior is exactly zero:

ValueError: Expected value argument (Tensor of shape (1, 1))
to be within the support (GreaterThan(lower_bound=0.0))
of the distribution LogNormalPrior(), but found invalid values:
tensor([[0.]], dtype=torch.float64, grad_fn=<SoftplusBackward0>)

An isolated check of AIPerf's DSP kernel reproduces the same exception: its default Positive() constraint uses a Softplus transformation, which can underflow to exactly zero for sufficiently negative raw lengthscale values.

The isolated reproduction below demonstrates the numerical failure mechanism. It does not yet reproduce the original optimizer trajectory.

Environment

Observed benchmark failure

• AIPerf: 0.12.0
• Python: 3.12
• Search recipe: max-throughput-ttft-sla
• Candidate-generation path: Optuna BoTorchSampler → AIPerf qlognei_candidates_func
• Search dimension: concurrency, from 1 to 1000
• Maximum search iterations: 30
• Per-point profiling limits: 1000 requests and 300 seconds
• No explicit --search-random-seed was supplied

Isolated numerical reproduction

The failure mechanism was also reproduced using the DSP kernel factory from upstream commit:

7db2ba37a62aa80c882bc90eaf61cc8073e2387b

Environment used for that check:

Component	Version
Platform	Linux ARM64, CPU
Python	3.12
PyTorch	2.14.0+cpu
BoTorch	0.16.1
GPyTorch	1.15.2
Tensor dtype	torch.float64

Both an unbatched kernel and a kernel with batch_shape=torch.Size([2]) exhibited the same zero-value prior-validation failure.

Observed failure path

Relevant call sequence from the benchmark traceback:

planner.ask
  → BoTorchSampler.sample_relative
  → qlognei_candidates_func
  → fit_gpytorch_mll
  → fit_gpytorch_mll_scipy
  → scipy.optimize.minimize / L-BFGS-B
  → prior.log_prob
  → ValueError: LogNormalPrior received a zero value

The benchmark terminated before producing the next search point.

Minimal mechanism reproduction

Run the following from the referenced AIPerf checkout, with its BoTorch dependencies already installed:

PYTHONPATH=src uv run --no-sync python - <<'PY'
from importlib.metadata import version

import torch

from aiperf.orchestrator.search_planner._botorch_kernel import make_dsp_kernel

print({
    name: version(name)
    for name in ("torch", "botorch", "gpytorch")
})

for batch_shape in (torch.Size([]), torch.Size([2])):
    kernel = make_dsp_kernel(
        d=1,
        batch_shape=batch_shape,
    ).double()
    base_kernel = kernel.base_kernel

    print("\nbatch_shape:", batch_shape)
    print("constraint:", base_kernel.raw_lengthscale_constraint)

    # Deliberately exercise numerical underflow.
    # This value was not captured from the original benchmark.
    with torch.no_grad():
        base_kernel.raw_lengthscale.fill_(-1000.0)

    print("lengthscale:", base_kernel.lengthscale)

    try:
        value = base_kernel.lengthscale_prior.log_prob(
            base_kernel.lengthscale
        )
        print("prior log probability:", value)
    except ValueError as exc:
        print(type(exc).__name__ + ":", exc)
PY

The unbatched case produces:

batch_shape: torch.Size([])
constraint: Positive()
lengthscale: tensor([[0.]], dtype=torch.float64, grad_fn=<SoftplusBackward0>)

ValueError: Expected value argument (Tensor of shape (1, 1))
to be within the support (GreaterThan(lower_bound=0.0))
of the distribution LogNormalPrior(), but found invalid values:
tensor([[0.]], dtype=torch.float64, grad_fn=<SoftplusBackward0>)

The batched case produces the same exception with a tensor shape of (2, 1, 1).

This reproduction requires neither a model nor an inference endpoint.

Expected behavior

During GP fitting, the lengthscale should remain within the strictly positive numerical domain required by its LogNormalPrior.

A raw parameter value reached during numerical optimization should not cause the lengthscale transformation to produce an invalid zero value and terminate the adaptive search.

Investigation

make_dsp_kernel() constructs a MaternKernel with a LogNormalPrior but does not specify an explicit numerical lower bound for the lengthscale.

In the tested dependency combination:

1. The kernel uses the default Positive() constraint.
2. Its Softplus transformation is mathematically positive, but sufficiently negative inputs underflow to exactly zero in floating-point arithmetic.
3. LogNormalPrior requires values strictly greater than zero.
4. Evaluating the prior at that zero value raises the observed exception.

This is consistent with the benchmark traceback. However:

• The original raw lengthscale value was not captured.
• The original full-precision GP training tensors were not captured.
• -1000 is a deliberately chosen diagnostic input, not an observed optimizer value.
• A deterministic, end-to-end GP-fitting reproducer is still needed to confirm the complete optimization trajectory.

Possible fix direction

Consider an explicit, numerically safe positive lower bound for the DSP kernel lengthscale, with compatible initialization.

The exact bound and its effect on GP fitting should be validated rather than selected solely to suppress the exception. Disabling prior validation would leave the invalid numerical state unresolved.

Suggested regression coverage:

• Extreme negative raw lengthscales retain a strictly positive transformed value.
• Prior evaluation remains finite.
• Unbatched and batched multi-output kernels remain supported.
• A deterministic GP-fitting regression exercises the failure.
• Multi-step candidate generation continues successfully after the fix.

Related work

PR #1321 fixed the DSP kernel's batch dimensions for constrained multi-output GPs.

This appears to be a separate numerical issue: the zero-lengthscale mechanism also reproduces using the kernel factory after that change.

I would be happy to follow up with a focused fix and regression tests.