Skip to content

openenv collect --llm-endpoint rejects any URL that includes a port or path #1188

Closed

Description

@surajsharan

Summary

--llm-endpoint is documented as an "OpenAI-compatible endpoint URL", but the value is handed to OpenAIClient(endpoint, port), which builds its base URL as f"{endpoint}:{port}/v1". Any URL that already names a port is mangled, and any path prefix gets /v1 appended after it.

$ openenv collect reasoning_gym:chain_sum --base-url http://localhost:8001 \
    --output-dir ./rollouts --llm-endpoint http://localhost:8000 --model Qwen/Qwen3.5-4B -n 2
Error: Invalid port: '8000:8000'

The only working form is the undocumented split --llm-endpoint http://localhost --llm-port 8000. It is shown in examples/ttt_collect_with_llm.py but not in --help or anywhere under docs/.

Where it lives

  • src/openenv/core/llm_client.py — LLMClient.base_url is f"{self.endpoint}:{self.port}"; OpenAIClient.__init__ passes f"{self.base_url}/v1" to AsyncOpenAI.
  • src/openenv/cli/commands/collect.py — --llm-endpoint help text says "endpoint URL"; --llm-port defaults to 8000.

The same constructor backs LLMJudge (core/rubrics/llm_judge.py) and build_model_step (core/harness/collect.py), so the limitation applies to every self-hosted judge or teacher, not only the CLI. Servers behind a path prefix (a LiteLLM proxy, vLLM behind a reverse proxy, Azure's /openai/v1) cannot be reached at all.

Repro without a server

from openenv.core.llm_client import OpenAIClient
OpenAIClient("http://localhost:8000", 8000, model="x")
# httpx.InvalidURL: Invalid port: '8000:8000'
OpenAIClient("http://proxy:4000/litellm", 4000, model="x")._client.base_url
# http://proxy:4000/litellm:4000/v1/

Proposed fix

Treat endpoint as a real URL in LLMClient: append /v1 only when the URL has no path and use a URL with a path as-is (the OpenAI SDK convention, so gateway prefixes such as Gemini's /v1beta/openai keep working), append port only when the URL names none, reject an explicit port that contradicts the URL, and raise a clear ValueError for malformed endpoints. Existing ("http://localhost", 8000) callers are unchanged; --llm-endpoint http://localhost:8000 and http://localhost:8000/v1 become equivalent. --llm-port should default to unset so a full URL without a port is left alone. Add a collect section to the CLI reference while at it, since the command is not documented there.

I have this working end to end against vLLM on a local GPU and will open a PR.

Environment: main @ 0521700, Python 3.12, openai 3.14.1.

Activity

  1. added a commit that references this issue on Sep 23, 2026
    1eda4b4
  2. surajsharan commented on Oct 20, 2026

    @surajsharan
    ContributorAuthor

    Fixed on main in 81379bd, which landed through #1237 an on-repo fold of #1189 for the 0.6.0 cut.

    --llm-endpoint now parses the value as a URL instead of treating it as a bare host: a pathless endpoint gets /v1 appended, an existing path is kept for gateways and proxies, --llm-port applies only when the URL has no port of its own, and bad schemes, credentials, query strings, fragments and out-of-range ports are rejected. OpenAIClient takes the same path, so a full base URL works there too.

    One deliberate behaviour change that came with it: a bare host no longer gets an implicit port 8000. Use http://localhost:8000, or pass --llm-port 8000.

    Closing as fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions