diff --git a/desktop/docs/architecture.md b/desktop/docs/architecture.md index 94928ec35..c4307ffaa 100644 --- a/desktop/docs/architecture.md +++ b/desktop/docs/architecture.md @@ -373,6 +373,13 @@ Personal AI Router and the Go services share one current per-user root: - `Nvidia Corporation/Personal AI Router`. +Managed engine payloads normally live under that root. vLLM is the deliberate +exception: its load-time GPU build tools cannot parse spaces in source/include +paths, so PAIR installs its private environment at +`~/.nvpair/engine-bin/vllm`. The append-only wipe inventory and every platform +uninstaller include that exact PAIR-owned path; model weights remain in the +separate Hugging Face cache. + On first launch after the rename, Electron data under `NVIDIA Corporation/PAIR` is merged into the shared directory without overwriting backend files already present there. diff --git a/desktop/docs/services-parity.md b/desktop/docs/services-parity.md index b4dc07a17..449604143 100644 --- a/desktop/docs/services-parity.md +++ b/desktop/docs/services-parity.md @@ -187,8 +187,11 @@ rather than driver-aware or offline-repacked. The bundled `vllm` manifest covers Linux (NVIDIA GPU) and Apple Silicon macOS (vLLM-Metal). It installs a pinned vLLM release into a private virtualenv -through a checksum-pinned `uv` bootstrap, or adopts a `vllm` already installed -at a detected path, then runs `vllm serve` on `8001` behind the proxy facade on +at the space-free managed path `~/.nvpair/engine-bin/vllm` through a +checksum-pinned `uv` bootstrap, or adopts a `vllm` already installed at a +detected path. The space-free physical path lets FlashInfer and DeepGEMM compile +load-time GPU kernels; reset and uninstall include the path while preserving the +model cache. It runs `vllm serve` on `8001` behind the proxy facade on `8000`. vLLM serves one model per process and has no load or unload API, so Eject (`unload_model`, a `stop_engine` action) stops the engine to free the GPU and Load on a stopped engine starts it; its model list diff --git a/desktop/scripts/build/installer.nsh b/desktop/scripts/build/installer.nsh index bb2df03de..1c3fb92ca 100644 --- a/desktop/scripts/build/installer.nsh +++ b/desktop/scripts/build/installer.nsh @@ -47,7 +47,8 @@ ; model store as models_dir, and a test in services/nvpair-engine-manager fails ; if one of them resolves inside the app data root. ; -; UNINSTALLER SAFETY: only ever delete the three per-user AppData roots below. +; UNINSTALLER SAFETY: only ever delete the three per-user AppData roots and the +; exact PAIR-managed vLLM payload below. ; Never touch $INSTDIR (C:\Program Files\...). The uninstaller itself lives at ; $INSTDIR\Uninstall ${PRODUCT_FILENAME}.exe and is removed by the template's ; own RMDir /r $INSTDIR + DeleteRegKey; deleting it here would orphan the @@ -66,6 +67,7 @@ ; - Shared app data: %LOCALAPPDATA%\Nvidia Corporation\Personal AI Router ; - Previous app data: %LOCALAPPDATA%\NVIDIA Corporation\PAIR ; - electron-updater: %LOCALAPPDATA%\nvpair-updater +; - Managed vLLM: %USERPROFILE%\.nvpair\engine-bin\vllm ; On Windows the "NVIDIA Corporation" / "Nvidia Corporation" casing resolves to ; the same case-insensitive folder. ; Per-user data roots. Keep in sync with src/shared/constants/app.ts and the @@ -74,10 +76,14 @@ DetailPrint "Removing NVIDIA PAIR user data..." ClearErrors ReadEnvStr $0 LOCALAPPDATA + ReadEnvStr $1 USERPROFILE RMDir /r "$0\Nvidia Corporation\Personal AI Router" RMDir /r "$0\NVIDIA Corporation\PAIR" RMDir "$0\NVIDIA Corporation" RMDir /r "$0\nvpair-updater" + RMDir /r "$1\.nvpair\engine-bin\vllm" + RMDir "$1\.nvpair\engine-bin" + RMDir "$1\.nvpair" ClearErrors !macroend @@ -109,6 +115,7 @@ ; Best-effort: when the user opts to remove data, stop any process whose ; executable lives UNDER one of the data roots (e.g. an engine like Ollama ; running from %LOCALAPPDATA%\Nvidia Corporation\Personal AI Router\engine-bin\) +; or the PAIR-managed vLLM directory ; so RMDir /r can then delete it. Scoped by ExecutablePath, so an external ; Ollama installed elsewhere is never touched. nsExec::ExecToLog never aborts the (un)installer, ; PowerShell swallows its own errors (-ErrorAction SilentlyContinue), and a @@ -117,7 +124,7 @@ ; containing spaces need no extra quoting. !macro pairKillProcessesInDataDirs DetailPrint "Stopping any processes running from NVIDIA PAIR data folders..." - nsExec::ExecToLog 'powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command "Get-CimInstance Win32_Process | Where-Object { $$_.ExecutablePath -and ($$_.ExecutablePath -like \"$$env:LOCALAPPDATA\NVIDIA Corporation\PAIR\*\" -or $$_.ExecutablePath -like \"$$env:LOCALAPPDATA\Nvidia Corporation\Personal AI Router\*\") } | ForEach-Object { Stop-Process -Id $$_.ProcessId -Force -ErrorAction SilentlyContinue }"' + nsExec::ExecToLog 'powershell -NoProfile -NonInteractive -ExecutionPolicy Bypass -Command "Get-CimInstance Win32_Process | Where-Object { $$_.ExecutablePath -and ($$_.ExecutablePath -like \"$$env:LOCALAPPDATA\NVIDIA Corporation\PAIR\*\" -or $$_.ExecutablePath -like \"$$env:LOCALAPPDATA\Nvidia Corporation\Personal AI Router\*\" -or $$_.ExecutablePath -like \"$$env:USERPROFILE\.nvpair\engine-bin\vllm\*\") } | ForEach-Object { Stop-Process -Id $$_.ProcessId -Force -ErrorAction SilentlyContinue }"' ; Give the OS a moment to release the file handles before RMDir /r. Sleep 1000 !macroend @@ -128,6 +135,7 @@ ; continue — the uninstall must never abort over locked user data. !macro pairWarnIfDataRemains ReadEnvStr $0 LOCALAPPDATA + ReadEnvStr $1 USERPROFILE StrCpy $9 "" IfFileExists "$0\Nvidia Corporation\Personal AI Router\*.*" 0 +2 StrCpy $9 "$9$\n$0\Nvidia Corporation\Personal AI Router" @@ -135,6 +143,8 @@ StrCpy $9 "$9$\n$0\NVIDIA Corporation\PAIR" IfFileExists "$0\nvpair-updater\*.*" 0 +2 StrCpy $9 "$9$\n$0\nvpair-updater" + IfFileExists "$1\.nvpair\engine-bin\vllm\*.*" 0 +2 + StrCpy $9 "$9$\n$1\.nvpair\engine-bin\vllm" StrCmp $9 "" pairNoLeftover MessageBox MB_OK|MB_ICONEXCLAMATION "Some NVIDIA PAIR data could not be removed because files were still in use (for example, a running engine such as Ollama).$\n$\nClose those programs, then delete these folders manually:$9" pairNoLeftover: diff --git a/desktop/scripts/build/linux/after-remove.sh b/desktop/scripts/build/linux/after-remove.sh index 5257a8e71..0db0bc354 100755 --- a/desktop/scripts/build/linux/after-remove.sh +++ b/desktop/scripts/build/linux/after-remove.sh @@ -44,7 +44,8 @@ fi # 4. Remove per-user data only on `purge`, never on a plain `remove`. Per-user # data here means settings, logs, cluster identity and certificates, and the -# engines NVIDIA PAIR installed under those roots (Ollama and llama.cpp); +# engines NVIDIA PAIR installed under those roots (Ollama and llama.cpp) or in +# its managed space-free vLLM directory; # downloaded model weights live outside them (~/.ollama, ~/.llamacpp, # ~/.lmstudio/models) and are never touched. # @@ -81,10 +82,13 @@ case "${1:-}" in # Every delete is best-effort so a locked or missing path never aborts removal. rm -rf "$user_home/.config/Nvidia Corporation/Personal AI Router" 2>/dev/null || true rm -rf "$user_home/.config/NVIDIA Corporation/PAIR" 2>/dev/null || true + rm -rf "$user_home/.nvpair/engine-bin/vllm" 2>/dev/null || true # Remove the current and previous parents only when empty so other NVIDIA # applications survive. rmdir "$user_home/.config/Nvidia Corporation" 2>/dev/null || true rmdir "$user_home/.config/NVIDIA Corporation" 2>/dev/null || true + rmdir "$user_home/.nvpair/engine-bin" 2>/dev/null || true + rmdir "$user_home/.nvpair" 2>/dev/null || true fi ;; esac diff --git a/desktop/scripts/build/macos/uninstall.sh b/desktop/scripts/build/macos/uninstall.sh index 15f195a06..2d876840a 100755 --- a/desktop/scripts/build/macos/uninstall.sh +++ b/desktop/scripts/build/macos/uninstall.sh @@ -156,10 +156,13 @@ echo "Removing user data..." # scripts/wipe-app-data.sh — do not silently diverge. rm -rf "$APP_SUPPORT/Nvidia Corporation/Personal AI Router" 2>/dev/null || true rm -rf "$APP_SUPPORT/NVIDIA Corporation/PAIR" 2>/dev/null || true +rm -rf "$target_home/.nvpair/engine-bin/vllm" 2>/dev/null || true # Remove the current and previous parents only when empty so other NVIDIA # applications survive. rmdir "$APP_SUPPORT/Nvidia Corporation" 2>/dev/null || true rmdir "$APP_SUPPORT/NVIDIA Corporation" 2>/dev/null || true +rmdir "$target_home/.nvpair/engine-bin" 2>/dev/null || true +rmdir "$target_home/.nvpair" 2>/dev/null || true # Best-effort removal of macOS framework state keyed on the bundle id. rm -rf "$target_home/Library/Preferences/$PACKAGE_ID.plist" 2>/dev/null || true diff --git a/desktop/tests/modular/app-data-wipe.test.ts b/desktop/tests/modular/app-data-wipe.test.ts index 12336080a..6f1ec6f9e 100644 --- a/desktop/tests/modular/app-data-wipe.test.ts +++ b/desktop/tests/modular/app-data-wipe.test.ts @@ -40,6 +40,19 @@ describe('repo-root wipe scripts', () => { } }) + it('lists the managed space-free vLLM directory in both inventories', () => { + const shText = fs.readFileSync(sh, 'utf8') + const psText = fs.readFileSync(ps1, 'utf8') + expect(shText).toContain('VLLM_INSTALL_ROOT="$HOME_DIR/.nvpair/engine-bin/vllm"') + expect(shText).toContain('TARGETS+=("$VLLM_INSTALL_ROOT|PAIR-managed vLLM payload")') + expect(psText).toContain( + "$VLLMInstallRoot = Join-Path $HomeDir (Join-Path '.nvpair' (Join-Path 'engine-bin' 'vllm'))" + ) + expect(psText).toContain( + "@{ Path = $VLLMInstallRoot; Reason = 'PAIR-managed vLLM payload' }" + ) + }) + it('accepts the app-invoked wait-pid and relaunch options in both twins', () => { const shText = fs.readFileSync(sh, 'utf8') const psText = fs.readFileSync(ps1, 'utf8') diff --git a/docs/building.mdx b/docs/building.mdx index ee93969dc..31da17423 100644 --- a/docs/building.mdx +++ b/docs/building.mdx @@ -255,8 +255,9 @@ application running: On Windows use `scripts\wipe-app-data.cmd` with the same flags. Neither script needs Node, and both exclude engine model libraries such as `~/.ollama`, -`~/.lmstudio/models`, and `~/.llamacpp` by design, so a reset does not cost you -re-downloading models. +`~/.lmstudio/models`, `~/.llamacpp`, and `~/.cache/huggingface` by design, so a +reset does not cost you re-downloading models. They do remove PAIR's managed +space-free vLLM environment at `~/.nvpair/engine-bin/vllm`. Start with `--dry-run`. It prints the exact paths and deletes nothing. If the machine belongs to a cluster, leave the cluster as well, or the other diff --git a/docs/engine-lifecycle.mdx b/docs/engine-lifecycle.mdx index b705c7aa6..58276e485 100644 --- a/docs/engine-lifecycle.mdx +++ b/docs/engine-lifecycle.mdx @@ -104,7 +104,11 @@ When you install an engine, consider the following: and Qwen3.6, so on a Mac they are left out of **Add model**, and loading or downloading one is refused with a message to choose the BF16 version. Linux, DGX Spark included, runs them. - Nothing is compiled. vLLM serves a single model per process. Until you choose + PAIR installs its private environment at `~/.nvpair/engine-bin/vllm` so spaces + in the app-data path do not break load-time GPU builds. Reset and uninstall + remove that managed environment while preserving downloaded models. + Installation uses prebuilt wheels; loading a model may compile FlashInfer or + DeepGEMM kernels. vLLM serves a single model per process. Until you choose one it serves Qwen3.5-4B, downloaded on the first start (about 9.3 GB, and a GPU with at least 16 GB of memory); on a smaller GPU, add and load a smaller model first. Its model list diff --git a/scripts/wipe-app-data.ps1 b/scripts/wipe-app-data.ps1 index fed5459f8..be4b42312 100644 --- a/scripts/wipe-app-data.ps1 +++ b/scripts/wipe-app-data.ps1 @@ -24,9 +24,10 @@ # - scripts/wipe-app-data.sh (Unix twin — update both in the same change) # # Explicit exclusions (never add): %USERPROFILE%\.ollama, %USERPROFILE%\.llamacpp, -# %USERPROFILE%\.lmstudio, external engine installs, and the application install -# tree (Program Files\PAIR). Those three are the engines' model stores; each -# engine declares its own as models_dir, and nothing PAIR removes may delete one. +# %USERPROFILE%\.lmstudio, %USERPROFILE%\.cache\huggingface, engine installs PAIR +# did not create, and the application install tree (Program Files\PAIR). Those +# home directories are model stores; each engine declares its own as models_dir, +# and nothing PAIR removes may delete one. PAIR's managed vLLM payload is included. # --------------------------------------------------------------------------- [CmdletBinding()] @@ -51,9 +52,10 @@ Usage: scripts\wipe-app-data.cmd [options] scripts\wipe-app-data.ps1 [options] Delete all NVIDIA PAIR-owned application data (settings, logs, cluster -identity, chat history, PAIR-managed engines under the app data root). +identity, chat history, and PAIR-managed engines). -Does NOT delete model libraries (%USERPROFILE%\.ollama, .llamacpp, .lmstudio\models). +Does NOT delete model libraries (%USERPROFILE%\.ollama, .llamacpp, +.lmstudio\models, .cache\huggingface). Does NOT uninstall the application binary. Options: @@ -106,9 +108,14 @@ if (-not $LocalAppData) { $HomeDir = $env:USERPROFILE if (-not $HomeDir) { $HomeDir = $HOME } +if (-not $HomeDir) { + Write-Error 'USERPROFILE is unset' + exit 1 +} $CurrentRoot = Join-Path $LocalAppData (Join-Path $AppOrg $AppDataDirName) $LegacyRoot = Join-Path $LocalAppData (Join-Path $PreviousOrg $PreviousDataDirName) +$VLLMInstallRoot = Join-Path $HomeDir (Join-Path '.nvpair' (Join-Path 'engine-bin' 'vllm')) $Username = $env:USERNAME if (-not $Username) { $Username = 'user' } @@ -128,6 +135,7 @@ $ControlDir = Join-Path $Tmp "nvpair-$Scope" $Targets = @( @{ Path = $CurrentRoot; Reason = 'Current shared Electron + backend app data root' }, @{ Path = $LegacyRoot; Reason = 'Legacy pre-rename app data root (NVIDIA Corporation/PAIR)' }, + @{ Path = $VLLMInstallRoot; Reason = 'PAIR-managed vLLM payload' }, @{ Path = (Join-Path $LocalAppData 'nvpair-updater'); Reason = 'electron-updater download cache (Windows)' }, @{ Path = (Join-Path $CurrentRoot (Join-Path 'bin' 'nvpair.cmd')); Reason = 'Generated Windows nvpair launcher under userData' }, @{ Path = $ControlDir; Reason = 'Removed CLI control token directory' } @@ -204,9 +212,10 @@ if ($running.Count -gt 0 -and $ForceKill) { Write-Host 'Stopping PAIR processes (--force-kill)...' $killScript = @' Get-CimInstance Win32_Process | Where-Object { - $_.ExecutablePath -and ( - $_.ExecutablePath -like "$env:LOCALAPPDATA\NVIDIA Corporation\PAIR\*" -or - $_.ExecutablePath -like "$env:LOCALAPPDATA\Nvidia Corporation\Personal AI Router\*" + $_.ExecutablePath -and ( + $_.ExecutablePath -like "$env:LOCALAPPDATA\NVIDIA Corporation\PAIR\*" -or + $_.ExecutablePath -like "$env:LOCALAPPDATA\Nvidia Corporation\Personal AI Router\*" -or + $_.ExecutablePath -like "$env:USERPROFILE\.nvpair\engine-bin\vllm\*" ) } | ForEach-Object { Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue } '@ @@ -227,7 +236,7 @@ if (-not $Confirmed) { } Write-Host '' Write-Host 'WARNING: This permanently deletes all NVIDIA PAIR app data.' - Write-Host 'Model libraries (%USERPROFILE%\.ollama, .llamacpp, .lmstudio\models) are NOT removed.' + Write-Host 'Model libraries (%USERPROFILE%\.ollama, .llamacpp, .lmstudio\models, .cache\huggingface) are NOT removed.' Write-Host '' Write-Host 'Paths to remove:' foreach ($t in $Targets) { Write-Host (" - {0}" -f $t.Path) } @@ -266,6 +275,18 @@ foreach ($org in @($AppOrg, $PreviousOrg)) { } } +foreach ($parent in @( + (Join-Path $HomeDir (Join-Path '.nvpair' 'engine-bin')) + (Join-Path $HomeDir '.nvpair') +)) { + if (Test-Path -LiteralPath $parent) { + $kids = @(Get-ChildItem -LiteralPath $parent -Force -ErrorAction SilentlyContinue) + if ($kids.Count -eq 0) { + try { Remove-Item -LiteralPath $parent -Force -ErrorAction SilentlyContinue } catch { } + } + } +} + Write-Host ("Removed {0} path(s)." -f $removed) if ($failures -gt 0) { Write-Error ("{0} path(s) could not be removed." -f $failures) diff --git a/scripts/wipe-app-data.sh b/scripts/wipe-app-data.sh index 7a8cfbfc8..83905fa18 100755 --- a/scripts/wipe-app-data.sh +++ b/scripts/wipe-app-data.sh @@ -24,10 +24,11 @@ # - desktop/scripts/build/{installer.nsh,linux/after-remove.sh,macos/uninstall.sh} # - scripts/wipe-app-data.ps1 (Windows twin — update both in the same change) # -# Explicit exclusions (never add): ~/.ollama, ~/.llamacpp, ~/.lmstudio, external -# engine installs, and the application install tree (Program Files / /opt/PAIR / -# the macOS app bundle). Those three are the engines' model stores; each engine -# declares its own as models_dir, and nothing PAIR removes may delete one. +# Explicit exclusions (never add): ~/.ollama, ~/.llamacpp, ~/.lmstudio, +# ~/.cache/huggingface, engine installs PAIR did not create, and the application +# install tree (Program Files / /opt/PAIR / the macOS app bundle). Those home +# directories are model stores; each engine declares its own as models_dir, and +# nothing PAIR removes may delete one. PAIR's managed vLLM payload is included. # --------------------------------------------------------------------------- set -uo pipefail @@ -44,9 +45,10 @@ usage() { Usage: scripts/wipe-app-data.sh [options] Delete all NVIDIA PAIR-owned application data (settings, logs, cluster -identity, chat history, PAIR-managed engines under the app data root). +identity, chat history, and PAIR-managed engines). -Does NOT delete model libraries (~/.ollama, ~/.llamacpp, ~/.lmstudio/models). +Does NOT delete model libraries (~/.ollama, ~/.llamacpp, ~/.lmstudio/models, +~/.cache/huggingface). Does NOT uninstall the application binary. Options: @@ -113,6 +115,7 @@ fi CURRENT_ROOT="$CONFIG_BASE/$APP_ORG/$APP_DATA_DIR_NAME" LEGACY_ROOT="$CONFIG_BASE/$APP_PREVIOUS_ORG/$APP_PREVIOUS_DATA_DIR_NAME" +VLLM_INSTALL_ROOT="$HOME_DIR/.nvpair/engine-bin/vllm" USERNAME="${USER:-user}" if command -v sha256sum >/dev/null 2>&1; then @@ -130,6 +133,7 @@ CONTROL_DIR="$RUNTIME_BASE/nvpair-$SCOPE" TARGETS=() TARGETS+=("$CURRENT_ROOT|Current shared Electron + backend app data root") TARGETS+=("$LEGACY_ROOT|Legacy pre-rename app data root (NVIDIA Corporation/PAIR)") +TARGETS+=("$VLLM_INSTALL_ROOT|PAIR-managed vLLM payload") TARGETS+=("$CACHE_HOME/nvpair-updater|electron-updater download cache") TARGETS+=("$CONTROL_DIR|Removed CLI control socket directory and auth token") TARGETS+=("$HOME_DIR/.local/bin/nvpair|Generated Linux nvpair launcher") @@ -216,7 +220,7 @@ if [[ "$CONFIRM" -eq 0 ]]; then fi echo "" echo "WARNING: This permanently deletes all NVIDIA PAIR app data." - echo "Model libraries (~/.ollama, ~/.llamacpp, ~/.lmstudio/models) are NOT removed." + echo "Model libraries (~/.ollama, ~/.llamacpp, ~/.lmstudio/models, ~/.cache/huggingface) are NOT removed." echo "" echo "Paths to remove:" for entry in "${TARGETS[@]}"; do @@ -251,6 +255,8 @@ done # Remove org parents only when empty (other NVIDIA apps may share them). rmdir "$CONFIG_BASE/$APP_ORG" 2>/dev/null || true rmdir "$CONFIG_BASE/$APP_PREVIOUS_ORG" 2>/dev/null || true +rmdir "$HOME_DIR/.nvpair/engine-bin" 2>/dev/null || true +rmdir "$HOME_DIR/.nvpair" 2>/dev/null || true echo "Removed $removed path(s)." if [[ "$failures" -gt 0 ]]; then diff --git a/services/nvpair-engine-manager/MANIFEST.md b/services/nvpair-engine-manager/MANIFEST.md index eb5c0442b..aedbbc66b 100644 --- a/services/nvpair-engine-manager/MANIFEST.md +++ b/services/nvpair-engine-manager/MANIFEST.md @@ -124,14 +124,15 @@ and recovery. Editing `args`/`start` directly remains trusted manifest authoring | `manifest_version` | int | yes | Must be `1`. A higher value is rejected (asks for behavior this binary lacks). Unknown optional fields within a supported version are ignored, so the schema can grow additively. | | `platforms` | object | yes | Map of `"/"` → platform block (e.g. `"windows/amd64"`, `"darwin/arm64"`, `"linux/amd64"`). At least one entry. The runner selects the block matching the host. | | `actions` | object | no | Map of action name → action (see below). | -| `detect` / `install` / `uninstall` / `models_dir` / `runtime` | — | no | Optional **shared defaults** inherited by every platform (see below). | +| `install_dir` / `detect` / `install` / `uninstall` / `models_dir` / `runtime` | — | no | Optional **shared defaults** inherited by every platform (see below). | -**Shared defaults & per-platform overrides.** The platform-level fields `detect`, `install`, `uninstall`, `models_dir`, and `runtime` may also be given once at the top level as shared defaults; each `platforms` entry is then merged onto them. Nested objects (e.g. `runtime`, `runtime.env`) merge key-by-key with the platform value winning, while arrays and scalars are replaced wholesale. So a runtime that's identical across platforms except `cli` is declared once at the top level, and each platform sets only `"runtime": { "cli": "…" }`. Omitting a key inherits the default; setting it (even to a zero value like `"port": 0`) overrides it. A manifest that fully specifies each platform with no top-level defaults behaves exactly as before. +**Shared defaults & per-platform overrides.** The platform-level fields `install_dir`, `detect`, `install`, `uninstall`, `models_dir`, and `runtime` may also be given once at the top level as shared defaults; each `platforms` entry is then merged onto them. Nested objects (e.g. `runtime`, `runtime.env`) merge key-by-key with the platform value winning, while arrays and scalars are replaced wholesale. So a runtime that's identical across platforms except `cli` is declared once at the top level, and each platform sets only `"runtime": { "cli": "…" }`. Omitting a key inherits the default; setting it (even to a zero value like `"port": 0`) overrides it. A manifest that fully specifies each platform with no top-level defaults behaves exactly as before. ## Platform block | Field | Type | Required | Notes | |---|---|---|---| +| `install_dir` | string | no | Literal managed payload directory, with OS env refs and a leading `~` expanded. Defaults to the app-data root's `engine-bin/`. Use an external location only when the engine requires it, and add that path to every app-data wipe/uninstall inventory. Available to the rest of the platform block as `{install_dir}`. | | `detect` | string[] | no | Paths that, if any exists, mean the engine is already installed. Supports OS env refs (`%VAR%`, `$VAR`), a leading `~`, and the `{install_dir}` placeholder. | | `install` | object | no | How to obtain the engine (see Install). Omit for engines that are only ever detected/launched. | | `uninstall` | object | no | How to remove a user-mode install (see Uninstall). Backs `engine:uninstall`. | @@ -181,10 +182,11 @@ PAIR cannot replace for them. Two separate removals have to leave them alone: data directory. Nothing can exempt a subdirectory of it, so the only defense is for `models_dir` to resolve somewhere else entirely. -That second one is why `models_dir` must be a literal path: `{install_dir}` is -inside the app data root, so any store derived from it is destroyed when a user -uninstalls the app — even though the prompt promises their models are kept, and -even though no engine uninstall ever touched them. +That second one is why `models_dir` must be a literal path. By default, +`{install_dir}` is inside the app data root, so any store derived from it is +destroyed when a user uninstalls the app. A platform may declare an external +`install_dir`, but that PAIR-owned path must also be in the app wipe inventory, +so it is no safer for models. Keep the store separate from both locations. Declare the engine's own default where it has one, so PAIR and a user's own copy of that engine share a library rather than downloading the same weights twice. @@ -496,7 +498,7 @@ validation at load: | `{cli}` | The platform's `runtime.cli` path | runtime start/stop, action `cmd` | | `{download}` | Path of the verified download | `install.run` | | `{download_}` | Path of a verified member of `install.artifacts` | `install.run` | -| `{install_dir}` | Per-engine user-scoped install dir | `detect`, `install`, runtime | +| `{install_dir}` | Per-engine managed payload directory: the platform's expanded `install_dir`, or the app-data root's `engine-bin/` by default | `detect`, `install`, uninstall, runtime | | `{hf_hub_cache}` | The Hugging Face hub cache the engine itself uses, from its launch environment as `huggingface_hub` resolves it: `HF_HUB_CACHE`, else `HUGGINGFACE_HUB_CACHE`, else `HF_HOME/hub`, else `$XDG_CACHE_HOME/huggingface/hub`, else `~/.cache/huggingface/hub` | the `dir` of an `hf_cache` or `hf_pull` action only | | `{models_dir}` | The platform's `models_dir`, expanded | `detect`, runtime args/env, `uninstall.run`, actions. **Not** `install`, which resolves only `{install_dir}` and the download paths — and not `uninstall.remove`, where naming the store is rejected at load. Unset when the platform declares no store, so a template referencing it then fails to resolve rather than silently becoming a path under `/` | @@ -651,7 +653,7 @@ GPU selection, and auth. | **GPT4All** | weak (GUI app) | none (Local API Server toggled in the GUI, `:4891`) | desktop settings / Python SDK — not env | OpenAI `/v1` once enabled in-app | Caveats worth encoding when authoring these: -- **Model store location**: leave each engine's model cache at the engine's own default — `~/.ollama`, `~/.llamacpp`, `~/.lmstudio/models` — and declare that path as `models_dir`. Never point one inside `{install_dir}` or anywhere else under the app data root: `uninstall` skips `models_dir`, but the app-level "remove all data" uninstall deletes the entire app data root and nothing can exempt a subdirectory of it. See Model stores. +- **Model store location**: leave each engine's model cache at the engine's own default — `~/.ollama`, `~/.llamacpp`, `~/.lmstudio/models` — and declare that path as `models_dir`. Never point one inside `{install_dir}` or anywhere else under the app data root: `uninstall` skips `models_dir`, but app reset removes both the data root and PAIR-owned external payload directories. See Model stores. - **llama.cpp** ships prebuilt archives with **published SHA-256s**, so it's an ideal checksum-pinned `fetch`+`run` target (no placeholder SHA needed). - **LM Studio** model dir is settings-controlled (no documented relocation env); surface its config as flags/API, not env. - **vLLM**: native Windows is unsupported (WSL only), and macOS is Apple Silicon only through the vLLM-Metal plugin (`darwin/arm64`); the bind is a flag, not env. The bundled `vllm.json` is the reference: it passes the model as `--model`, since manifest `runtime.args` have no `{model}` token, and installs through a sha256-pinned `uv` bootstrap with a detect-first path. diff --git a/services/nvpair-engine-manager/README.md b/services/nvpair-engine-manager/README.md index 89d25e3b9..f3714e1fb 100644 --- a/services/nvpair-engine-manager/README.md +++ b/services/nvpair-engine-manager/README.md @@ -89,8 +89,8 @@ reads the model's `config.json` from the Hugging Face cache, at the revision vLLM loads (`--revision`, or `main` through `refs/main`, the way `huggingface_hub` resolves it; another cached revision is never read in its place), and the free GPU -memory (`nvidia-smi`; on a unified-memory machine such as DGX Spark, where it -reports `[N/A]`, the total and available system memory from `/proc/meminfo`), +memory (`nvidia-smi`, or the installed runtime when native telemetry is missing; +system RAM is used only after confirming that the selected device shares it), then adds: `--max-model-len` of 64K (PAIR's agent-ready default; a model with a larger window is capped there, and a smaller one is shrunk to what fits), a YaRN `--hf-overrides` when the model's own window is below 64K (for @@ -146,13 +146,28 @@ When no parser reads the model's format (Nemotron Nano v2's ``, or a template with no tool support), nothing is added, and a request that asks for tool calls gets vLLM's own error (HTTP 400). A parser named in the launch arguments is always used as given. -The environment sets `VLLM_USE_FLASHINFER_SAMPLER=0`. vLLM's FlashInfer sampler -compiles GPU kernels on first start, and that build cannot handle the spaces in -PAIR's app-data folder (`Nvidia Corporation/Personal AI Router`), so the -built-in sampler is used instead. As a side effect no CUDA toolkit or C++ -compiler is needed. The same limit means anything that forces a FlashInfer -kernel build (for example choosing a FlashInfer attention backend) fails until -the install can live on a path without spaces. +PAIR installs its managed vLLM payload under `~/.nvpair/engine-bin/vllm`, rather +than under the app-data folder whose `Nvidia Corporation/Personal AI Router` +segments contain spaces. FlashInfer and DeepGEMM write unescaped source and +include paths into load-time GPU builds, so the space-free physical location lets +their attention and FP8/FP4 kernels compile normally. The engine's virtualenv is +put first on `PATH` so those builds find its bundled `ninja`. Engine uninstall, +Reset app data, and the platform uninstallers all remove this managed directory; +downloaded Hugging Face models remain in `~/.cache/huggingface`. + +The environment still sets `VLLM_USE_FLASHINFER_SAMPLER=0`, so an ordinary model +does not need a CUDA toolkit solely for the optional FlashInfer sampler. Other +FlashInfer and DeepGEMM kernels are not disabled, and vLLM chooses their attention +backend normally. + +Before choosing a context window and `--gpu-memory-utilization`, PAIR reads +device memory from `nvidia-smi` when available and otherwise asks the installed +vLLM platform. This covers CUDA/ROCm, Intel XPU, Apple Metal, CPU, and +out-of-tree backends that expose vLLM's memory API. Shared-memory platforms keep +3 GiB for runtime startup in addition to the normal 1 GiB free margin. If neither +provider can measure free memory, PAIR does not invent a limit and leaves memory +sizing to vLLM. + Readiness polls `/health` for up to eight minutes while weights load. The model list is the Hugging Face cache (`~/.cache/huggingface/hub`, or the diff --git a/services/nvpair-engine-manager/devicememory.go b/services/nvpair-engine-manager/devicememory.go new file mode 100644 index 000000000..c2dfe9ac2 --- /dev/null +++ b/services/nvpair-engine-manager/devicememory.go @@ -0,0 +1,74 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "bytes" + "context" + _ "embed" + "encoding/json" + "errors" + "fmt" + "log/slog" + "os" + "os/exec" + "time" +) + +//go:embed devicememory_probe.py +var deviceMemoryProbeScript string + +const deviceMemoryProbeTimeout = time.Minute + +type deviceMemoryProbe struct { + Total int64 `json:"total"` + Free int64 `json:"free"` + Shared bool `json:"shared"` +} + +var nativeDeviceMemory = func() (int64, int64, bool) { return nvidiaSMIMemory(nil) } +var probeVLLMDeviceMemory = runVLLMDeviceMemoryProbe + +func detectDeviceMemory(bin string) (total, free int64, ok bool) { + if total, free, ok := nativeDeviceMemory(); ok { + return total, free, true + } + reading, err := probeVLLMDeviceMemory(bin) + if err != nil { + slog.Warn("could not measure free device memory; leaving memory sizing to vLLM", "err", err) + return 0, 0, false + } + if reading.Shared { + reading.Free = max(reading.Free-unifiedMemoryHeadroom, 0) + } + return reading.Total, reading.Free, true +} + +func runVLLMDeviceMemoryProbe(bin string) (deviceMemoryProbe, error) { + python, err := vllmPython(bin) + if err != nil { + return deviceMemoryProbe{}, err + } + ctx, cancel := context.WithTimeout(context.Background(), deviceMemoryProbeTimeout) + defer cancel() + cmd := exec.CommandContext(ctx, python, "-I", "-c", deviceMemoryProbeScript) + cmd.Dir = os.TempDir() + out, err := cmd.Output() + if err != nil { + return deviceMemoryProbe{}, fmt.Errorf("device memory probe: %w", err) + } + return parseDeviceMemoryProbe(out) +} + +func parseDeviceMemoryProbe(out []byte) (deviceMemoryProbe, error) { + lines := bytes.Split(bytes.TrimSpace(out), []byte("\n")) + var reading deviceMemoryProbe + if len(lines) == 0 || json.Unmarshal(lines[len(lines)-1], &reading) != nil { + return deviceMemoryProbe{}, errors.New("device memory probe printed no result") + } + if reading.Total <= 0 || reading.Free < 0 || reading.Free > reading.Total { + return deviceMemoryProbe{}, errors.New("device memory probe printed an invalid result") + } + return reading, nil +} diff --git a/services/nvpair-engine-manager/devicememory_probe.py b/services/nvpair-engine-manager/devicememory_probe.py new file mode 100644 index 000000000..444eeef16 --- /dev/null +++ b/services/nvpair-engine-manager/devicememory_probe.py @@ -0,0 +1,54 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import json +import platform +import sys + +import psutil +import torch +from vllm.platforms import current_platform + + +def system_reading(total): + memory = psutil.virtual_memory() + return { + "total": total, + "free": min(int(memory.available), total), + "shared": True, + } + + +if torch.cuda.is_available(): + if torch.cuda.device_count() != 1: + raise RuntimeError("multiple CUDA devices require engine-owned memory sizing") + properties = torch.cuda.get_device_properties(0) + if getattr(properties, "is_integrated", False): + reading = system_reading(int(properties.total_memory)) + else: + free, total = torch.cuda.mem_get_info(0) + reading = {"total": int(total), "free": int(free), "shared": False} +elif hasattr(torch, "xpu") and torch.xpu.is_available(): + properties = torch.xpu.get_device_properties(0) + if getattr(properties, "is_integrated_gpu", False): + reading = system_reading(int(properties.total_memory)) + else: + free, total = torch.xpu.mem_get_info(0) + reading = {"total": int(total), "free": int(free), "shared": False} +else: + total = int(current_platform.get_device_total_memory(0)) + apple_silicon = ( + sys.platform == "darwin" + and platform.machine() == "arm64" + and torch.backends.mps.is_available() + ) + available = getattr(current_platform, "get_device_available_memory", None) + if callable(available): + free = int(available(0)) + reading = {"total": total, "free": free, "shared": apple_silicon or current_platform.is_cpu()} + elif current_platform.is_cpu() or apple_silicon: + reading = system_reading(total) + else: + raise RuntimeError("the selected vLLM backend does not report free memory") + +print(json.dumps(reading)) diff --git a/services/nvpair-engine-manager/devicememory_probe_test.go b/services/nvpair-engine-manager/devicememory_probe_test.go new file mode 100644 index 000000000..b168cd14e --- /dev/null +++ b/services/nvpair-engine-manager/devicememory_probe_test.go @@ -0,0 +1,127 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "encoding/json" + "os/exec" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// Run the embedded production probe with device APIs supplied by the fixture. +// Discrete and unknown devices deliberately reject any attempt to read host RAM. +const deviceMemoryProbeHarness = ` +import json +import sys +import types + +sys.excepthook = sys.__excepthook__ +fixture = json.load(sys.stdin) +gib = 1 << 30 +backend = fixture.get("backend", "unknown") +total = fixture.get("total_gib", 16) * gib +free = fixture.get("free_gib", 4) * gib + +def system_memory(): + if not fixture.get("allow_system", False): + raise RuntimeError("system RAM must not be read for this device") + return types.SimpleNamespace(total=fixture.get("system_total_gib", 16) * gib, + available=fixture.get("system_free_gib", 12) * gib) + +def device_memory(device): + if fixture.get("device_memory_error", False): + raise RuntimeError("device memory unavailable") + return free, total + +cuda_properties = types.SimpleNamespace(total_memory=total) +xpu_properties = types.SimpleNamespace(total_memory=total) +if not fixture.get("unknown_topology", False): + cuda_properties.is_integrated = fixture.get("integrated", False) + xpu_properties.is_integrated_gpu = fixture.get("integrated", False) + +torch = types.SimpleNamespace( + cuda=types.SimpleNamespace(is_available=lambda: backend in ("cuda", "rocm"), + device_count=lambda: fixture.get("device_count", 1), + mem_get_info=device_memory, get_device_properties=lambda device: cuda_properties), + xpu=types.SimpleNamespace(is_available=lambda: backend == "xpu", + mem_get_info=device_memory, get_device_properties=lambda device: xpu_properties), + backends=types.SimpleNamespace(mps=types.SimpleNamespace( + is_available=lambda: fixture.get("mps_available", False)))) +platform = types.SimpleNamespace(get_device_total_memory=lambda device: total, + is_cpu=lambda: backend == "cpu") +if fixture.get("has_device_free", False): + platform.get_device_available_memory = lambda device: free +sys.modules["torch"] = torch +sys.modules["psutil"] = types.SimpleNamespace(virtual_memory=system_memory) +sys.modules["platform"] = types.SimpleNamespace(machine=lambda: fixture.get("arch", "x86_64")) +sys.modules["vllm.platforms"] = types.SimpleNamespace(current_platform=platform) +sys.platform = fixture.get("os", "linux") +` + +func TestDeviceMemoryProbe_UsesSystemRAMOnlyForVerifiedSharedDevices(t *testing.T) { + python, err := exec.LookPath("python3") + if err != nil { + t.Skip("Python 3 is required to exercise the embedded vLLM memory probe") + } + accept := func(name, input string, want deviceMemoryProbe) { + t.Helper() + t.Run(name, func(t *testing.T) { + cmd := exec.Command(python, "-I", "-c", deviceMemoryProbeHarness+"\n"+deviceMemoryProbeScript) + cmd.Stdin = strings.NewReader(input) + output, err := cmd.CombinedOutput() + require.NoError(t, err, "%s", output) + var result deviceMemoryProbe + require.NoError(t, json.Unmarshal(output, &result)) + assert.Equal(t, want, result) + }) + } + accept("discrete CUDA with the same capacity as host RAM", `{"backend":"cuda"}`, + deviceMemoryProbe{Total: 16 * gib, Free: 4 * gib}) + accept("discrete ROCm with the same capacity as host RAM", `{"backend":"rocm"}`, + deviceMemoryProbe{Total: 16 * gib, Free: 4 * gib}) + accept("CUDA with unknown topology keeps device readings", `{"backend":"cuda","unknown_topology":true}`, + deviceMemoryProbe{Total: 16 * gib, Free: 4 * gib}) + accept("verified integrated CUDA uses available host RAM", `{"backend":"cuda","integrated":true,"allow_system":true}`, + deviceMemoryProbe{Total: 16 * gib, Free: 12 * gib, Shared: true}) + accept("integrated CUDA respects the device memory limit", `{"backend":"cuda","integrated":true,"allow_system":true,"total_gib":8}`, + deviceMemoryProbe{Total: 8 * gib, Free: 8 * gib, Shared: true}) + accept("discrete XPU keeps device readings", `{"backend":"xpu"}`, + deviceMemoryProbe{Total: 16 * gib, Free: 4 * gib}) + accept("verified integrated XPU uses available host RAM", `{"backend":"xpu","integrated":true,"allow_system":true}`, + deviceMemoryProbe{Total: 16 * gib, Free: 12 * gib, Shared: true}) + accept("Apple Silicon Metal without a free-memory API", `{"backend":"metal","os":"darwin","arch":"arm64","mps_available":true,"allow_system":true}`, + deviceMemoryProbe{Total: 16 * gib, Free: 12 * gib, Shared: true}) + accept("Apple Silicon preserves the engine available-memory reading", `{"backend":"metal","os":"darwin","arch":"arm64","mps_available":true,"has_device_free":true}`, + deviceMemoryProbe{Total: 16 * gib, Free: 4 * gib, Shared: true}) + accept("CPU explicitly uses system RAM", `{"backend":"cpu","allow_system":true}`, + deviceMemoryProbe{Total: 16 * gib, Free: 12 * gib, Shared: true}) + accept("unknown engine with its own free-memory API", `{"has_device_free":true}`, + deviceMemoryProbe{Total: 16 * gib, Free: 4 * gib}) +} + +func TestDeviceMemoryProbe_RejectsUnavailableMemoryWithoutGuessingTopology(t *testing.T) { + python, err := exec.LookPath("python3") + if err != nil { + t.Skip("Python 3 is required to exercise the embedded vLLM memory probe") + } + for _, tc := range []struct{ name, input, message string }{ + {"unknown device matching host RAM", `{}`, "the selected vLLM backend does not report free memory"}, + {"Intel Mac with Metal is not proof of shared RAM", `{"os":"darwin","arch":"x86_64","mps_available":true}`, "the selected vLLM backend does not report free memory"}, + {"Apple Silicon without Metal available", `{"os":"darwin","arch":"arm64"}`, "the selected vLLM backend does not report free memory"}, + {"discrete CUDA query failure", `{"backend":"cuda","device_memory_error":true}`, "device memory unavailable"}, + {"multiple CUDA devices", `{"backend":"cuda","device_count":2}`, "multiple CUDA devices require engine-owned memory sizing"}, + } { + t.Run(tc.name, func(t *testing.T) { + cmd := exec.Command(python, "-I", "-c", deviceMemoryProbeHarness+"\n"+deviceMemoryProbeScript) + cmd.Stdin = strings.NewReader(tc.input) + output, err := cmd.CombinedOutput() + require.Error(t, err) + assert.Contains(t, string(output), tc.message) + }) + } +} diff --git a/services/nvpair-engine-manager/devicememory_test.go b/services/nvpair-engine-manager/devicememory_test.go new file mode 100644 index 000000000..ee2d49ce9 --- /dev/null +++ b/services/nvpair-engine-manager/devicememory_test.go @@ -0,0 +1,107 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "errors" + "testing" +) + +func withDeviceMemoryProviders(t *testing.T, native func() (int64, int64, bool), probe func(string) (deviceMemoryProbe, error)) { + t.Helper() + oldNative := nativeDeviceMemory + oldProbe := probeVLLMDeviceMemory + nativeDeviceMemory = native + probeVLLMDeviceMemory = probe + t.Cleanup(func() { + nativeDeviceMemory = oldNative + probeVLLMDeviceMemory = oldProbe + }) +} + +func TestDetectDeviceMemory_PrefersNativeTelemetry(t *testing.T) { + withDeviceMemoryProviders(t, + func() (int64, int64, bool) { return 48 * gib, 32 * gib, true }, + func(string) (deviceMemoryProbe, error) { + t.Fatal("the vLLM probe must not run when native telemetry works") + return deviceMemoryProbe{}, nil + }, + ) + + total, free, ok := detectDeviceMemory("/opt/vllm/bin/vllm") + if !ok || total != 48*gib || free != 32*gib { + t.Fatalf("got total=%d free=%d ok=%v, want native telemetry", total, free, ok) + } +} + +func TestDetectDeviceMemory_FallsBackToTheSelectedVLLMBackend(t *testing.T) { + withDeviceMemoryProviders(t, + func() (int64, int64, bool) { return 0, 0, false }, + func(bin string) (deviceMemoryProbe, error) { + if bin != "/opt/vllm/bin/vllm" { + t.Fatalf("probe bin = %q", bin) + } + return deviceMemoryProbe{Total: 64 * gib, Free: 40 * gib}, nil + }, + ) + + total, free, ok := detectDeviceMemory("/opt/vllm/bin/vllm") + if !ok || total != 64*gib || free != 40*gib { + t.Fatalf("got total=%d free=%d ok=%v, want the vLLM backend reading", total, free, ok) + } +} + +func TestDetectDeviceMemory_UsesVRAMWhenNvidiaSMIReportsNA(t *testing.T) { + withDeviceMemoryProviders(t, + func() (int64, int64, bool) { return gpuMemoryFromQuery("[N/A], [N/A]\n") }, + func(string) (deviceMemoryProbe, error) { + return deviceMemoryProbe{Total: 16 * gib, Free: 4 * gib, Shared: false}, nil + }, + ) + + total, free, ok := detectDeviceMemory("vllm") + if !ok || total != 16*gib || free != 4*gib { + t.Fatalf("got total=%d free=%d ok=%v, want the discrete device's VRAM", total, free, ok) + } +} + +func TestDetectDeviceMemory_ReservesStartupHeadroomOnSharedMemory(t *testing.T) { + withDeviceMemoryProviders(t, + func() (int64, int64, bool) { return 0, 0, false }, + func(string) (deviceMemoryProbe, error) { + return deviceMemoryProbe{Total: 64 * gib, Free: 20 * gib, Shared: true}, nil + }, + ) + + _, free, ok := detectDeviceMemory("vllm") + if !ok || free != 20*gib-unifiedMemoryHeadroom { + t.Fatalf("free = %d ok=%v, want shared memory less startup headroom", free, ok) + } +} + +func TestDetectDeviceMemory_LeavesSizingToVLLMWhenEveryProviderFails(t *testing.T) { + withDeviceMemoryProviders(t, + func() (int64, int64, bool) { return 0, 0, false }, + func(string) (deviceMemoryProbe, error) { return deviceMemoryProbe{}, errors.New("unsupported backend") }, + ) + + if _, _, ok := detectDeviceMemory("vllm"); ok { + t.Fatal("a made-up memory limit is less safe than leaving sizing to vLLM") + } +} + +func TestParseDeviceMemoryProbe_ReadsTheLastOutputLine(t *testing.T) { + reading, err := parseDeviceMemoryProbe([]byte("vLLM startup log\n{\"total\":68719476736,\"free\":42949672960,\"shared\":true}\n")) + if err != nil || reading.Total != 64*gib || reading.Free != 40*gib || !reading.Shared { + t.Fatalf("reading = %+v err=%v", reading, err) + } +} + +func TestParseDeviceMemoryProbe_RejectsImpossibleReadings(t *testing.T) { + for _, output := range []string{"", "not json", `{"total":0,"free":0}`, `{"total":10,"free":11}`} { + if _, err := parseDeviceMemoryProbe([]byte(output)); err == nil { + t.Errorf("accepted %q", output) + } + } +} diff --git a/services/nvpair-engine-manager/executor.go b/services/nvpair-engine-manager/executor.go index 0c107d2ee..89d3149e8 100644 --- a/services/nvpair-engine-manager/executor.go +++ b/services/nvpair-engine-manager/executor.go @@ -102,7 +102,7 @@ type Executor struct { // streaming handlers) in addition to the local engine:install-progress // notification path. See progress.go. progress *progressHub - baseDir string // user-scoped install base + baseDir string // default user-scoped install base // desired persists explicit per-engine ON/OFF intent. Runtime state remains // in-memory; shutdown cleanup must not rewrite this store. desired *desiredStateStore @@ -253,7 +253,7 @@ func (e *Executor) state(engine string) (*engineState, error) { plat: plat, logs: newLogBuffer(), port: plat.Runtime.Port, - installDir: filepath.Join(e.baseDir, engine), + installDir: engineInstallDir(e.baseDir, engine, plat.InstallDir), } if plat.ModelsDir != "" { st.modelsDir = expandPath(plat.ModelsDir) @@ -262,6 +262,13 @@ func (e *Executor) state(engine string) (*engineState, error) { return st, nil } +func engineInstallDir(defaultBase, engine, configured string) string { + if configured != "" { + return filepath.Clean(expandPath(configured)) + } + return filepath.Join(defaultBase, engine) +} + // pathVars are the directory placeholders every templated manifest string can // use. Both are absolute by the time they get here, so a manifest never has to // expand "~" itself — which it could not do, since only argv and a handful of diff --git a/services/nvpair-engine-manager/hfcache_test.go b/services/nvpair-engine-manager/hfcache_test.go index 37bdc393e..d0e981793 100644 --- a/services/nvpair-engine-manager/hfcache_test.go +++ b/services/nvpair-engine-manager/hfcache_test.go @@ -357,7 +357,7 @@ func TestDeleteHFModel_RefusesTheBundledDefaultModel(t *testing.T) { reg := loadWithOverrides(t, t.TempDir()) base := t.TempDir() // The bundled manifest runs `hf` from the vLLM virtualenv. - venvBin := filepath.Join(base, "vllm", "venv", "bin") + venvBin := filepath.Join(home, ".nvpair", "engine-bin", "vllm", "venv", "bin") require.NoError(t, os.MkdirAll(venvBin, 0o755)) require.NoError(t, os.WriteFile(filepath.Join(venvBin, "hf"), []byte(fakeHFCacheRm), 0o755)) e := NewExecutor(reg, NewReporter(nil), func(string, any) {}, base) diff --git a/services/nvpair-engine-manager/launchauto.go b/services/nvpair-engine-manager/launchauto.go index 77d9dd47c..5abfdd88a 100644 --- a/services/nvpair-engine-manager/launchauto.go +++ b/services/nvpair-engine-manager/launchauto.go @@ -34,9 +34,24 @@ const ( minAutoUtil = 0.15 ) -// gpuMemory reports memory only when the launch has one unambiguous GPU. +// Shared-memory runtimes need room for their own startup allocations. +const unifiedMemoryHeadroom = 3 * gib + +var deviceMemory = detectDeviceMemory + +// gpuMemory leaves explicit CUDA device selection to the engine. // It is a variable so tests can supply a machine. -var gpuMemory = nvidiaSMIMemory +var gpuMemory = func(bin string, env map[string]string) (int64, int64, bool) { + if _, selected := os.LookupEnv("CUDA_VISIBLE_DEVICES"); selected { + return 0, 0, false + } + for name := range env { + if environmentKey(name) == "CUDA_VISIBLE_DEVICES" { + return 0, 0, false + } + } + return deviceMemory(bin) +} func nvidiaSMIMemory(env map[string]string) (total, free int64, ok bool) { // CUDA ordinals can differ from nvidia-smi's physical-device order. Leave @@ -55,15 +70,12 @@ func nvidiaSMIMemory(env map[string]string) (total, free int64, ok bool) { if err != nil { return 0, 0, false } - return parseGPUMemory(string(out), systemMemory) + return gpuMemoryFromQuery(string(out)) } -// parseGPUMemory reads a single GPU's memory.total,memory.free query, in MiB. -// Multiple devices are ambiguous without reproducing CUDA's enumeration, so -// the engine handles their memory sizing. On a unified-memory machine such as -// DGX Spark the GPU shares system RAM and nvidia-smi reports [N/A]; system -// memory then stands in, as it does in nvpair-node-info and vLLM itself. -func parseGPUMemory(out string, system func() (total, available int64, ok bool)) (total, free int64, ok bool) { +// gpuMemoryFromQuery accepts one unambiguous device with valid VRAM readings. +// Missing telemetry requires the runtime probe to establish memory topology. +func gpuMemoryFromQuery(out string) (total, free int64, ok bool) { line := strings.TrimSpace(out) if strings.ContainsAny(line, "\r\n") { return 0, 0, false @@ -72,59 +84,14 @@ func parseGPUMemory(out string, system func() (total, available int64, ok bool)) if len(parts) != 2 { return 0, 0, false } - if nvidiaSMINotApplicable(parts[0]) { - return system() - } t, errT := strconv.ParseInt(strings.TrimSpace(parts[0]), 10, 64) f, errF := strconv.ParseInt(strings.TrimSpace(parts[1]), 10, 64) - if errT != nil || errF != nil || t <= 0 { + if errT != nil || errF != nil || t <= 0 || f < 0 || f > t || t > math.MaxInt64/(1024*1024) { return 0, 0, false } return t * 1024 * 1024, f * 1024 * 1024, true } -// nvidiaSMINotApplicable reports an nvidia-smi field that holds no number, such -// as the [N/A] or [Not Supported] a unified-memory GPU gives for its memory. -func nvidiaSMINotApplicable(field string) bool { - value := strings.ToLower(strings.Trim(strings.TrimSpace(field), "[]")) - return value == "n/a" || value == "not supported" -} - -// systemMemory reports total and available system memory from /proc/meminfo. -func systemMemory() (total, available int64, ok bool) { - raw, err := os.ReadFile("/proc/meminfo") - if err != nil { - return 0, 0, false - } - return parseMeminfo(string(raw)) -} - -// parseMeminfo reads MemTotal and MemAvailable, which /proc/meminfo gives in kB. -// Available rather than free counts the page cache the kernel can reclaim. -func parseMeminfo(text string) (total, available int64, ok bool) { - for _, line := range strings.Split(text, "\n") { - key, rest, found := strings.Cut(line, ":") - if !found { - continue - } - fields := strings.Fields(rest) - if len(fields) == 0 { - continue - } - kb, err := strconv.ParseInt(fields[0], 10, 64) - if err != nil { - continue - } - switch key { - case "MemTotal": - total = kb * 1024 - case "MemAvailable": - available = kb * 1024 - } - } - return total, available, total > 0 && available > 0 -} - // applyAutoLaunch appends the arguments the manifest's auto_launch asks for. // It runs after the user's launch has been validated and never changes an // argument the user supplied, so anything set in the engine settings wins. @@ -138,7 +105,7 @@ func applyAutoLaunch(ctx context.Context, engine string, rt Runtime, command lau if rt.AutoLaunch == nil || rt.AutoLaunch.Kind != autoLaunchHFContext { return command } - extra, note := planHFContext(command.Args, hub, int64(rt.AutoLaunch.ContextTokens), fetch, command.Env) + extra, note := planHFContext(command.Args, hub, int64(rt.AutoLaunch.ContextTokens), fetch, command.Env, command.Bin) if tools, parser := planToolCalling(ctx, command.Args, hub, command.Bin, fetch); len(tools) > 0 { extra = append(extra, tools...) note = strings.TrimPrefix(note+"; tool calls parsed with "+parser, "; ") @@ -377,7 +344,7 @@ func autoGPUShare(need, total, free int64) (float64, bool) { // is not in the cache yet is read through fetch, when given, since vLLM // downloads it on this start. Without its size and config, only the memory // share is bounded. -func planHFContext(args []string, hubDir string, target int64, fetch hfModelFetcher, env map[string]string) (extra []string, note string) { +func planHFContext(args []string, hubDir string, target int64, fetch hfModelFetcher, env map[string]string, bin string) (extra []string, note string) { if _, ok := launchFlagValue(args, "--tensor-parallel-size"); ok { return nil, "" } @@ -403,7 +370,7 @@ func planHFContext(args []string, hubDir string, target int64, fetch hfModelFetc var total, free int64 var haveGPU bool if !userDtype && !userCacheDtype { - total, free, haveGPU = gpuMemory(env) + total, free, haveGPU = gpuMemory(bin, env) } found, known := findLaunchModel(args, hubDir, fetch) if !known { diff --git a/services/nvpair-engine-manager/launchauto_test.go b/services/nvpair-engine-manager/launchauto_test.go index 28f94050e..d27c8f20c 100644 --- a/services/nvpair-engine-manager/launchauto_test.go +++ b/services/nvpair-engine-manager/launchauto_test.go @@ -51,14 +51,14 @@ func pendingModel(t *testing.T, cache, id, config string, weightBytes int64) hfM func withGPU(t *testing.T, totalGiB, freeGiB int64) { t.Helper() old := gpuMemory - gpuMemory = func(map[string]string) (int64, int64, bool) { return totalGiB * gib, freeGiB * gib, true } + gpuMemory = func(string, map[string]string) (int64, int64, bool) { return totalGiB * gib, freeGiB * gib, true } t.Cleanup(func() { gpuMemory = old }) } func withoutGPU(t *testing.T) { t.Helper() old := gpuMemory - gpuMemory = func(map[string]string) (int64, int64, bool) { return 0, 0, false } + gpuMemory = func(string, map[string]string) (int64, int64, bool) { return 0, 0, false } t.Cleanup(func() { gpuMemory = old }) } @@ -147,7 +147,7 @@ func TestAutoContext_SizesTheModelNamedLast(t *testing.T) { // use it rather than fall back to the uncached first one. cache := cachedModel(t, "Qwen/Qwen3-8B", qwen8BConfig, 16_400_000_000) - extra, _ := planHFContext([]string{"serve", "--model", "org/not-cached", "--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"serve", "--model", "org/not-cached", "--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil, "") assert.Equal(t, "65536", flagOf(t, extra, "--max-model-len"), "sized from Qwen3-8B's config: %v", extra) } @@ -156,7 +156,7 @@ func TestAutoContext_ExtendsAShortModelToTheTarget(t *testing.T) { withGPU(t, 48, 48) cache := cachedModel(t, "Qwen/Qwen3-8B", qwen8BConfig, 16_400_000_000) - extra, _ := planHFContext([]string{"serve", "--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"serve", "--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil, "") assert.Equal(t, "65536", flagOf(t, extra, "--max-model-len"), "the model alone stops at 40960") var overrides struct { @@ -182,7 +182,7 @@ func TestAutoContext_CapsALongModelAtTheTarget(t *testing.T) { cfg := strings.Replace(qwen8BConfig, "40960", "262144", 1) cache := cachedModel(t, "Qwen/Long", cfg, 16_400_000_000) - extra, _ := planHFContext([]string{"--model", "Qwen/Long"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Long"}, cache, 65536, nil, nil, "") assert.Equal(t, "65536", flagOf(t, extra, "--max-model-len"), "the target, not the 262144 native window") assert.NotContains(t, extra, "--hf-overrides", "a model that already supports the target needs no rope override") @@ -193,7 +193,7 @@ func TestAutoContext_LeavesModelsWithRopeScalingAlone(t *testing.T) { cfg := strings.Replace(qwen8BConfig, `"torch_dtype"`, `"rope_scaling":{"rope_type":"yarn","factor":4.0},"torch_dtype"`, 1) cache := cachedModel(t, "Qwen/Scaled", cfg, 16_400_000_000) - extra, _ := planHFContext([]string{"--model", "Qwen/Scaled"}, cache, 131072, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Scaled"}, cache, 131072, nil, nil, "") assert.NotContains(t, extra, "--hf-overrides", "must not override a model's own rope scaling") assert.Equal(t, "40960", flagOf(t, extra, "--max-model-len")) @@ -203,7 +203,7 @@ func TestAutoContext_ShrinksToWhatFitsInFreeMemory(t *testing.T) { withGPU(t, 48, 22) cache := cachedModel(t, "Qwen/Qwen3-8B", qwen8BConfig, 16_400_000_000) - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil, "") n, err := strconv.ParseInt(flagOf(t, extra, "--max-model-len"), 10, 64) require.NoError(t, err) @@ -221,7 +221,7 @@ func TestAutoContext_KeepsTheShareWithinFreeMemoryOnALargeBusyGPU(t *testing.T) withGPU(t, 48, 7) cache := cachedModel(t, "Qwen/Qwen3-0.6B", qwen8BConfig, 1_200_000_000) - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-0.6B"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-0.6B"}, cache, 65536, nil, nil, "") assert.InDelta(t, 0.12, utilOf(t, extra), 0.0001) assert.LessOrEqual(t, utilOf(t, extra)*48, 6.0, "the share must fit the 6 GiB free beyond the margin") @@ -232,14 +232,14 @@ func TestAutoContext_NeverOverridesWhatTheUserSet(t *testing.T) { cache := cachedModel(t, "Qwen/Qwen3-8B", qwen8BConfig, 16_400_000_000) t.Run("the user's context", func(t *testing.T) { - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", "--max-model-len=8192"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", "--max-model-len=8192"}, cache, 65536, nil, nil, "") assert.NotContains(t, extra, "--max-model-len") assert.NotContains(t, extra, "--hf-overrides") assert.LessOrEqual(t, utilOf(t, extra), 0.45, "an 8K window needs far less of the GPU") }) t.Run("the user's GPU share", func(t *testing.T) { - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", "--gpu-memory-utilization", "0.5"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", "--gpu-memory-utilization", "0.5"}, cache, 65536, nil, nil, "") assert.NotContains(t, extra, "--gpu-memory-utilization") n, err := strconv.ParseInt(flagOf(t, extra, "--max-model-len"), 10, 64) @@ -248,7 +248,7 @@ func TestAutoContext_NeverOverridesWhatTheUserSet(t *testing.T) { assert.GreaterOrEqual(t, n, int64(4096)) }) t.Run("both", func(t *testing.T) { - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", "--max-model-len", "4096", "--gpu-memory-utilization", "0.3"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", "--max-model-len", "4096", "--gpu-memory-utilization", "0.3"}, cache, 65536, nil, nil, "") assert.Empty(t, extra) }) @@ -259,7 +259,7 @@ func TestAutoContext_SkipsShardedLaunches(t *testing.T) { cache := cachedModel(t, "Qwen/Qwen3-8B", qwen8BConfig, 16_400_000_000) test := func(flag string) { t.Run(flag, func(t *testing.T) { - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", flag, "2"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B", flag, "2"}, cache, 65536, nil, nil, "") assert.Empty(t, extra, "sizing a sharded launch needs more than GPU 0") }) @@ -271,7 +271,7 @@ func TestAutoContext_SkipsShardedLaunches(t *testing.T) { func TestAutoContext_GivesAModelNotInTheCacheWhatIsFree(t *testing.T) { withGPU(t, 48, 20) - extra, _ := planHFContext([]string{"--model", "org/not-downloaded"}, t.TempDir(), 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "org/not-downloaded"}, t.TempDir(), 65536, nil, nil, "") assert.NotContains(t, extra, "--max-model-len", "an unknown model's limits are unknown") assert.LessOrEqual(t, utilOf(t, extra), 19.0/48.0+0.001, "must leave a margin of the 20 GiB free") @@ -281,7 +281,7 @@ func TestAutoContext_BoundsTheWindowWithoutAGPUReading(t *testing.T) { withoutGPU(t) cache := cachedModel(t, "Qwen/Qwen3-8B", qwen8BConfig, 16_400_000_000) - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, cache, 65536, nil, nil, "") assert.Equal(t, "65536", flagOf(t, extra, "--max-model-len"), "the window does not depend on the GPU reading") assert.NotContains(t, extra, "--gpu-memory-utilization", "without a reading the GPU share is left to vLLM") @@ -292,7 +292,7 @@ func TestAutoContext_WrapsOverridesForAMultimodalConfig(t *testing.T) { cfg := `{"architectures":["Qwen3VLForConditionalGeneration"],"text_config":` + qwen8BConfig + `}` cache := cachedModel(t, "Qwen/VL", cfg, 16_400_000_000) - extra, _ := planHFContext([]string{"--model", "Qwen/VL"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/VL"}, cache, 65536, nil, nil, "") var overrides map[string]json.RawMessage require.NoError(t, json.Unmarshal([]byte(flagOf(t, extra, "--hf-overrides")), &overrides)) @@ -309,8 +309,8 @@ func TestAutoContext_SizesByTheWeightsVLLMLoadsNotTheDiskTheSnapshotUses(t *test repo := writeHFRepo(t, both, "Qwen/Qwen3-8B", map[string]string{"pytorch_model.bin": "x"}) require.NoError(t, os.Truncate(filepath.Join(repo, "blobs", "blob-pytorch_model.bin"), 16_400_000_000)) - alone, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, single, 65536, nil, nil) - withCopy, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, both, 65536, nil, nil) + alone, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, single, 65536, nil, nil, "") + withCopy, _ := planHFContext([]string{"--model", "Qwen/Qwen3-8B"}, both, 65536, nil, nil, "") assert.Equal(t, alone, withCopy) models, err := scanHFCache(both) @@ -372,18 +372,18 @@ func TestAutoContext_ReadsTheRevisionVLLMLoads(t *testing.T) { cache := hfRevisions(t) t.Run("main", func(t *testing.T) { - extra, _ := planHFContext([]string{"--model", "org/m"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "org/m"}, cache, 65536, nil, nil, "") assert.Equal(t, "65536", flagOf(t, extra, "--max-model-len")) assert.NotContains(t, extra, "--hf-overrides", "main's 131072-token window needs no rope override") }) t.Run("an explicit older revision", func(t *testing.T) { - extra, _ := planHFContext([]string{"--model", "org/m", "--revision", "aaa111"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "org/m", "--revision", "aaa111"}, cache, 65536, nil, nil, "") assert.Contains(t, extra, "--hf-overrides", "aaa111's 8192-token window is extended to the target") }) t.Run("a revision that is not cached", func(t *testing.T) { - extra, _ := planHFContext([]string{"--model", "org/m", "--revision", "ccc333"}, cache, 65536, nil, nil) + extra, _ := planHFContext([]string{"--model", "org/m", "--revision", "ccc333"}, cache, 65536, nil, nil, "") assert.NotContains(t, extra, "--max-model-len", "no other revision's config is read") }) @@ -486,42 +486,13 @@ func TestPlanToolCalling_KeepsTheParserTheUserChose(t *testing.T) { assert.Equal(t, []string{"--enable-auto-tool-choice"}, extra, "only the enable flag beside the user's parser") } -func TestParseGPUMemory_ReadsADiscreteGPUAndFallsBackOnUnifiedMemory(t *testing.T) { - system := func() (int64, int64, bool) { return 128 * gib, 100 * gib, true } - reads := func(name, out string, wantTotal, wantFree int64) { - t.Run(name, func(t *testing.T) { - total, free, ok := parseGPUMemory(out, system) - - require.True(t, ok) - assert.Equal(t, wantTotal, total) - assert.Equal(t, wantFree, free) - }) - } - unreadable := func(name, out string) { - t.Run(name, func(t *testing.T) { - _, _, ok := parseGPUMemory(out, system) - +func TestGPUMemoryFromQuery_RejectsAmbiguousOrImpossibleReadings(t *testing.T) { + for _, output := range []string{"24576, 20000\n49140, 48600\n", "error\n", "", "16, -1", "16, 17", "0, 0"} { + t.Run(output, func(t *testing.T) { + _, _, ok := gpuMemoryFromQuery(output) assert.False(t, ok) }) } - reads("discrete GPU, in MiB", "49140, 48600\n", 49140<<20, 48600<<20) - unreadable("multiple GPUs without an identified CUDA device", "24576, 20000\n49140, 48600\n") - reads("DGX Spark reports N/A", "[N/A], [N/A]\n", 128*gib, 100*gib) - reads("not supported", "[Not Supported], [Not Supported]\n", 128*gib, 100*gib) - unreadable("malformed", "error\n") - unreadable("empty", "") -} - -func TestParseMeminfo_ReadsTotalAndAvailable(t *testing.T) { - text := "MemTotal: 130567000 kB\nMemFree: 2000000 kB\nMemAvailable: 104000000 kB\nBuffers: 100 kB\n" - - total, available, ok := parseMeminfo(text) - - require.True(t, ok) - assert.Equal(t, int64(130567000*1024), total) - assert.Equal(t, int64(104000000*1024), available) - _, _, ok = parseMeminfo("MemTotal: 100 kB\n") - assert.False(t, ok, "a reading without MemAvailable must not count as known") } // qwen35_4BConfig is the part of Qwen/Qwen3.5-4B's config.json that sizing @@ -537,7 +508,7 @@ func TestAutoContext_FirstStartOfAnUncachedModelGetsRoomForItsWeights(t *testing withGPU(t, 16, 15) cache := t.TempDir() - extra, note := planHFContext([]string{"--model", "Qwen/Qwen3.5-4B"}, cache, 65536, pendingModel(t, cache, "Qwen/Qwen3.5-4B", qwen35_4BConfig, qwen35_4BBytes), nil) + extra, note := planHFContext([]string{"--model", "Qwen/Qwen3.5-4B"}, cache, 65536, pendingModel(t, cache, "Qwen/Qwen3.5-4B", qwen35_4BConfig, qwen35_4BBytes), nil, "") weightsAndOverhead := float64(qwen35_4BBytes)*weightsSlack + float64(vllmOverhead) assert.GreaterOrEqual(t, utilOf(t, extra)*16*float64(gib), weightsAndOverhead, "the share must hold the weights and overhead: %v", extra) @@ -554,7 +525,7 @@ func TestAutoContext_DoesNotFetchACachedModel(t *testing.T) { return hfPendingModel{}, false } - extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3.5-8B", "--model", "Qwen/Qwen3-8B"}, cache, 65536, fetch, nil) + extra, _ := planHFContext([]string{"--model", "Qwen/Qwen3.5-8B", "--model", "Qwen/Qwen3-8B"}, cache, 65536, fetch, nil, "") assert.Equal(t, "65536", flagOf(t, extra, "--max-model-len")) } @@ -785,7 +756,7 @@ func TestFetchHFMetadata_FallsBackWhenHuggingFaceCannotAnswer(t *testing.T) { withGPU(t, 16, 15) extra, _ := planHFContext([]string{"--model", "org/model"}, f.hub, 65536, func() (hfPendingModel, bool) { return f.fetch(t, "org/model", "main") - }, nil) + }, nil, "") assert.InDelta(t, 0.87, utilOf(t, extra), 0.0001) assert.NotContains(t, extra, "--max-model-len") }) @@ -907,7 +878,7 @@ func TestAutoContext_KeepsTheNativeLimitWithUserConfigOverrides(t *testing.T) { } args := []string{"--model", "Qwen/Qwen3-8B", "--hf-overrides", `{"attention_dropout":0.0}`} - extra, _ := planHFContext(args, t.TempDir(), 65536, fetch, nil) + extra, _ := planHFContext(args, t.TempDir(), 65536, fetch, nil, "") assert.Equal(t, "40960", flagOf(t, extra, "--max-model-len")) assert.NotContains(t, extra, "--hf-overrides") @@ -917,7 +888,7 @@ func TestApplyAutoLaunch_UsesTheEngineDeviceEnvironment(t *testing.T) { env := map[string]string{"CUDA_VISIBLE_DEVICES": "1"} old := gpuMemory var received map[string]string - gpuMemory = func(got map[string]string) (int64, int64, bool) { + gpuMemory = func(_ string, got map[string]string) (int64, int64, bool) { received = got return 0, 0, false } @@ -965,7 +936,7 @@ func TestAutoContext_LeavesExplicitDtypeMemorySizingToTheEngine(t *testing.T) { t.Run(name, func(t *testing.T) { args := []string{"--model", "Qwen/Qwen3-8B", flag, value} - extra, _ := planHFContext(args, t.TempDir(), 65536, fetch, nil) + extra, _ := planHFContext(args, t.TempDir(), 65536, fetch, nil, "") assert.NotContains(t, extra, "--gpu-memory-utilization") assert.Equal(t, "65536", flagOf(t, extra, "--max-model-len")) @@ -986,7 +957,7 @@ func TestAutoContext_PreservesTheUnderscoreContextLimit(t *testing.T) { } args := []string{"--model", "Qwen/Qwen3-8B", "--max_model_len", "8192"} - extra, _ := planHFContext(args, t.TempDir(), 65536, fetch, nil) + extra, _ := planHFContext(args, t.TempDir(), 65536, fetch, nil, "") assert.Empty(t, extra, "the explicit limit must prevent automatic context and rope overrides") } diff --git a/services/nvpair-engine-manager/launchauto_unified_test.go b/services/nvpair-engine-manager/launchauto_unified_test.go new file mode 100644 index 000000000..96fe4e11a --- /dev/null +++ b/services/nvpair-engine-manager/launchauto_unified_test.go @@ -0,0 +1,102 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +package main + +import "testing" + +// A DGX Spark (GB10) as seen while a model load failed: 121.69 GiB of unified +// memory of which about 41.7 GiB was available. The GPU reports "[N/A]" for its +// own memory, so the system's figures are the only ones there are. +const ( + sparkTotal = int64(127600524) * 1024 + sparkAvail = int64(43684000) * 1024 +) + +const llama8BFP8Config = `{"max_position_embeddings":131072,"num_hidden_layers":32,"num_attention_heads":32,"num_key_value_heads":8,"head_dim":128,"torch_dtype":"bfloat16","rope_scaling":{"rope_type":"llama3","factor":8.0}}` + +// The runtime confirms shared memory after native memory telemetry is missing. +func withUnifiedMemoryGPU(t *testing.T, total, available int64) { + t.Helper() + withDeviceMemoryProviders(t, + func() (int64, int64, bool) { return gpuMemoryFromQuery("[N/A], [N/A]\n") }, + func(string) (deviceMemoryProbe, error) { + return deviceMemoryProbe{Total: total, Free: available, Shared: true}, nil + }, + ) + old := deviceMemory + deviceMemory = detectDeviceMemory + t.Cleanup(func() { deviceMemory = old }) +} + +func TestGPUMemoryFromQuery_ReadsADiscreteGPU(t *testing.T) { + total, free, ok := gpuMemoryFromQuery("49140, 40000\n") + if !ok || total != 49140*1024*1024 || free != 40000*1024*1024 { + t.Fatalf("got total=%d free=%d ok=%v, want the two MiB figures in bytes", total, free, ok) + } +} + +func TestGPUMemoryFromQuery_RejectsMissingValuesWithoutAssumingSharedMemory(t *testing.T) { + for _, tc := range []struct{ name, output string }{ + {"bracketed N/A, as nvidia-smi prints with nounits", "[N/A], [N/A]\n"}, + {"bare N/A", "N/A, N/A\n"}, + {"Not Supported", "[Not Supported], [Not Supported]\n"}, + } { + t.Run(tc.name, func(t *testing.T) { + total, free, ok := gpuMemoryFromQuery(tc.output) + if ok || total != 0 || free != 0 { + t.Fatalf("got total=%d free=%d ok=%v; missing telemetry must not become a host RAM reading", total, free, ok) + } + }) + } +} + +func TestGPUMemoryFromQuery_IsUnavailableWhenOnlyOneFigureIsMissing(t *testing.T) { + if _, _, ok := gpuMemoryFromQuery("[N/A], 1000\n"); ok { + t.Fatal("one missing figure is not a unified-memory GPU; the reading is unusable") + } +} + +func TestDetectDeviceMemory_ReportsNoFreeMemoryBelowSharedStartupHeadroom(t *testing.T) { + withUnifiedMemoryGPU(t, sparkTotal, gib) + _, free, ok := detectDeviceMemory("vllm") + if !ok || free != 0 { + t.Fatalf("free = %d ok=%v, want 0 rather than a negative figure", free, ok) + } +} + +// The failure behind the report: with about 41 GiB available, the launch asked +// vLLM for nothing, so it wanted its default 0.92 of 121.69 GiB (111.95 GiB) and +// stopped before readiness. +func TestPlanHFContext_CapsTheGPUShareOnAUnifiedMemoryMachine(t *testing.T) { + withUnifiedMemoryGPU(t, sparkTotal, sparkAvail) + cache := cachedModel(t, "nvidia/Llama-3.1-8B-Instruct-FP8", llama8BFP8Config, 9_100_000_000) + extra, _ := planHFContext([]string{"serve", "--model", "nvidia/Llama-3.1-8B-Instruct-FP8"}, cache, 65536, nil, nil, "") + + util := utilOf(t, extra) + request := util * float64(sparkTotal) + if limit := float64(sparkAvail - unifiedMemoryHeadroom - freeMargin); request > limit { + t.Fatalf("--gpu-memory-utilization %.2f asks for %.1f GiB but only %.1f GiB can be claimed", util, request/float64(gib), limit/float64(gib)) + } + if got := flagOf(t, extra, "--max-model-len"); got != "65536" { + t.Fatalf("max-model-len = %q, want 65536 (it fits in the memory that is free): %v", got, extra) + } +} + +func TestPlanHFContext_CapsTheGPUShareOfAModelNotYetDownloadedOnAUnifiedMemoryMachine(t *testing.T) { + withUnifiedMemoryGPU(t, sparkTotal, sparkAvail) + extra, _ := planHFContext([]string{"serve", "--model", "nvidia/Llama-3.1-8B-Instruct-FP8"}, t.TempDir(), 65536, nil, nil, "") + + util := utilOf(t, extra) + if limit := float64(sparkAvail-unifiedMemoryHeadroom-freeMargin) / float64(sparkTotal); util > limit { + t.Fatalf("--gpu-memory-utilization %.2f is above the %.2f that is free: %v", util, limit, extra) + } +} + +func TestPlanHFContext_LeavesTheShareToVLLMWhenAUnifiedMemoryMachineHasNothingFree(t *testing.T) { + withUnifiedMemoryGPU(t, sparkTotal, gib) + extra, _ := planHFContext([]string{"serve", "--model", "nvidia/Llama-3.1-8B-Instruct-FP8"}, t.TempDir(), 65536, nil, nil, "") + if v, _ := launchFlagValue(extra, "--gpu-memory-utilization"); v != "" { + t.Fatalf("no share can be sized from nothing free; got %q in %v", v, extra) + } +} diff --git a/services/nvpair-engine-manager/manifests/vllm.json b/services/nvpair-engine-manager/manifests/vllm.json index 7bf0f3fc6..356ab4a9c 100644 --- a/services/nvpair-engine-manager/manifests/vllm.json +++ b/services/nvpair-engine-manager/manifests/vllm.json @@ -13,6 +13,7 @@ "vllm-install", "{download}", "{install_dir}" ] }, + "install_dir": "~/.nvpair/engine-bin/vllm", "models_dir": "~/.cache/huggingface", "uninstall": { "remove": ["{install_dir}"] }, "runtime": { diff --git a/services/nvpair-engine-manager/modelsdir_test.go b/services/nvpair-engine-manager/modelsdir_test.go index db2f1dff4..0c122bb73 100644 --- a/services/nvpair-engine-manager/modelsdir_test.go +++ b/services/nvpair-engine-manager/modelsdir_test.go @@ -16,15 +16,14 @@ import ( "nvpair-shared/appdir" ) -// TestBundledModelStoresOutliveTheAppDataRoot is the regression guard for the -// uninstall that deleted downloaded models. The app-level "remove all data" -// uninstall deletes the whole app data root, so a model store anywhere inside it -// is destroyed even though no engine uninstall touched it — which is exactly -// what llama.cpp's cache did when it sat beside the install directory. +// TestBundledModelStores_RemainOutsideManagedApplicationPaths is the regression +// guard for the uninstall that deleted downloaded models. App reset deletes the +// whole app data root and every declared external install directory, so a model +// store inside either is destroyed even though no engine uninstall touched it. // // The app data root always ends in the vendor/product segments, so this holds // the line for every target platform rather than only the test host's. -func TestBundledModelStores_RemainOutsideApplicationDataRoot(t *testing.T) { +func TestBundledModelStores_RemainOutsideManagedApplicationPaths(t *testing.T) { forbidden := []string{"nvidia corporation", "personal ai router"} for name, manifest := range bundledManifestSet(t) { for key, platform := range manifest.Platforms { @@ -37,6 +36,9 @@ func TestBundledModelStores_RemainOutsideApplicationDataRoot(t *testing.T) { break } } + if platform.InstallDir != "" { + assert.False(t, pathWithinRoot(expandPath(platform.InstallDir), expandPath(platform.ModelsDir)), "%s/%s: models_dir %q resolves under managed install_dir %q", name, key, platform.ModelsDir, platform.InstallDir) + } } platform, ok := manifest.HostPlatform() if !ok { diff --git a/services/nvpair-engine-manager/proc.go b/services/nvpair-engine-manager/proc.go index 70f3bd27c..94d412625 100644 --- a/services/nvpair-engine-manager/proc.go +++ b/services/nvpair-engine-manager/proc.go @@ -14,6 +14,49 @@ import ( "time" ) +// processEnvironment is the environment an engine process starts with: the +// current one with env merged over it, and the engine's own Python virtual +// environment first on PATH. An engine installed in a virtual environment runs +// its helper tools by name (vLLM's FlashInfer compiles kernels with ninja), and +// those are in the environment's bin directory, which the service's own PATH +// does not include. It returns nil when there is nothing to add, so the process +// inherits the environment unchanged. +func processEnvironment(bin string, env map[string]string) []string { + venvBin := venvBinDir(bin) + if len(env) == 0 && venvBin == "" { + return nil + } + merged := os.Environ() + for k, v := range env { + merged = append(merged, k+"="+v) + } + if venvBin != "" { + path, ok := env["PATH"] + if !ok { + path = os.Getenv("PATH") + } + merged = append(merged, "PATH="+venvBin+string(os.PathListSeparator)+path) + } + return merged +} + +// venvBinDir returns the directory holding bin when it belongs to a Python +// virtual environment, which has a pyvenv.cfg beside its bin directory, and "" +// when it does not. A symbolic link such as ~/.local/bin/vllm is followed. +func venvBinDir(bin string) string { + candidates := []string{bin} + if resolved, err := filepath.EvalSymlinks(bin); err == nil && resolved != bin { + candidates = append(candidates, resolved) + } + for _, candidate := range candidates { + dir := filepath.Dir(candidate) + if _, err := os.Stat(filepath.Join(filepath.Dir(dir), "pyvenv.cfg")); err == nil { + return dir + } + } + return "" +} + // managedProc is a spawned engine process with its stdout/stderr // captured line-by-line and a `done` channel that closes when it // exits. The OS-specific primitives it relies on (console hiding, @@ -30,12 +73,7 @@ type managedProc struct { // closes once the process exits. func startManagedProc(bin string, args []string, env map[string]string, onLine func(stream, line string)) (*managedProc, error) { cmd := exec.Command(bin, args...) - if len(env) > 0 { - cmd.Env = os.Environ() - for k, v := range env { - cmd.Env = append(cmd.Env, k+"="+v) - } - } + cmd.Env = processEnvironment(bin, env) configureSysProcAttr(cmd) stdout, err := cmd.StdoutPipe() diff --git a/services/nvpair-engine-manager/proc_venv_test.go b/services/nvpair-engine-manager/proc_venv_test.go new file mode 100644 index 000000000..45131415f --- /dev/null +++ b/services/nvpair-engine-manager/proc_venv_test.go @@ -0,0 +1,124 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//go:build linux + +package main + +import ( + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +// fakeVenv lays out a Python virtual environment: a pyvenv.cfg beside a bin +// directory holding the given executable scripts. It returns the bin directory. +func fakeVenv(t *testing.T, scripts map[string]string) string { + t.Helper() + root := t.TempDir() + bin := filepath.Join(root, "venv", "bin") + if err := os.MkdirAll(bin, 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "venv", "pyvenv.cfg"), []byte("home = /usr/bin\n"), 0o644); err != nil { + t.Fatal(err) + } + for name, body := range scripts { + if err := os.WriteFile(filepath.Join(bin, name), []byte(body), 0o755); err != nil { + t.Fatal(err) + } + } + return bin +} + +// lastValue is the value the process will see for key, which is the last entry +// naming it. +func lastValue(environment []string, key string) (string, bool) { + value, found := "", false + for _, entry := range environment { + if k, v, ok := strings.Cut(entry, "="); ok && k == key { + value, found = v, true + } + } + return value, found +} + +func TestProcessEnvironment_PutsTheVirtualEnvironmentFirstOnPath(t *testing.T) { + t.Setenv("PATH", "/usr/bin:/bin") + bin := fakeVenv(t, map[string]string{"vllm": "#!/bin/sh\n"}) + + got, _ := lastValue(processEnvironment(filepath.Join(bin, "vllm"), nil), "PATH") + + if want := bin + ":/usr/bin:/bin"; got != want { + t.Fatalf("PATH = %q, want the venv's bin directory ahead of the existing PATH (%q)", got, want) + } +} + +func TestProcessEnvironment_InheritsTheEnvironmentForAnEngineOutsideAVirtualEnvironment(t *testing.T) { + dir := t.TempDir() + if got := processEnvironment(filepath.Join(dir, "ollama"), nil); got != nil { + t.Fatalf("environment = %v, want nil so the process inherits it unchanged", got) + } +} + +func TestProcessEnvironment_KeepsTheManifestVariablesAndTheirPath(t *testing.T) { + bin := fakeVenv(t, map[string]string{"vllm": "#!/bin/sh\n"}) + + environment := processEnvironment(filepath.Join(bin, "vllm"), map[string]string{"HF_HOME": "/models", "PATH": "/custom"}) + + if got, _ := lastValue(environment, "HF_HOME"); got != "/models" { + t.Fatalf("HF_HOME = %q, want /models", got) + } + if got, _ := lastValue(environment, "PATH"); got != bin+":/custom" { + t.Fatalf("PATH = %q, want the venv's bin directory ahead of the PATH the manifest set", got) + } +} + +func TestProcessEnvironment_FollowsALinkIntoTheVirtualEnvironment(t *testing.T) { + bin := fakeVenv(t, map[string]string{"vllm": "#!/bin/sh\n"}) + shim := filepath.Join(t.TempDir(), "vllm") + if err := os.Symlink(filepath.Join(bin, "vllm"), shim); err != nil { + t.Fatal(err) + } + + if got := venvBinDir(shim); got != bin { + t.Fatalf("venvBinDir = %q, want %q: a link such as ~/.local/bin/vllm belongs to the venv it points into", got, bin) + } +} + +// vLLM's FlashInfer compiles kernels by running ninja by name, and ninja is in +// the virtual environment's bin directory. Before the engine was started with +// that directory on PATH, loading a model that needs a kernel stopped with +// "No such file or directory: 'ninja'". +func TestStartManagedProc_FindsToolsInTheEnginesVirtualEnvironment(t *testing.T) { + t.Setenv("PATH", "/usr/bin:/bin") + bin := fakeVenv(t, map[string]string{ + "engine": "#!/bin/sh\nexec ninja\n", + "ninja": "#!/bin/sh\necho ninja-was-found\n", + }) + + var mu sync.Mutex + var lines []string + proc, err := startManagedProc(filepath.Join(bin, "engine"), nil, nil, func(_, line string) { + mu.Lock() + defer mu.Unlock() + lines = append(lines, line) + }) + if err != nil { + t.Fatal(err) + } + select { + case <-proc.done: + case <-time.After(10 * time.Second): + t.Fatal("the engine did not exit") + } + + mu.Lock() + defer mu.Unlock() + if got := strings.Join(lines, "\n"); !strings.Contains(got, "ninja-was-found") { + t.Fatalf("engine output = %q, want it to have run the venv's ninja", got) + } +} diff --git a/services/nvpair-engine-manager/provenance.go b/services/nvpair-engine-manager/provenance.go index 023421e72..04c8d665b 100644 --- a/services/nvpair-engine-manager/provenance.go +++ b/services/nvpair-engine-manager/provenance.go @@ -125,7 +125,13 @@ func (e *Executor) UninstallManaged(ctx context.Context) []ManagedUninstall { slog.Warn("stopped removing engines before finishing", "err", err, "remaining", len(names)-len(results)) return results } - installDir := filepath.Join(e.baseDir, engine) + st, err := e.state(engine) + if err != nil { + slog.Info("leaving engine in place; it is not supported on this host", "engine", engine, "err", err) + results = append(results, ManagedUninstall{Engine: engine}) + continue + } + installDir := st.installDir if !installedByPAIR(installDir) { slog.Info("leaving engine in place; PAIR did not install it", "engine", engine) results = append(results, ManagedUninstall{Engine: engine}) diff --git a/services/nvpair-engine-manager/provenance_test.go b/services/nvpair-engine-manager/provenance_test.go index ecf8d2c31..67032a52a 100644 --- a/services/nvpair-engine-manager/provenance_test.go +++ b/services/nvpair-engine-manager/provenance_test.go @@ -105,6 +105,27 @@ func TestUninstallManaged_RemovesOnlyWhatPAIRInstalled(t *testing.T) { assert.False(t, installedByPAIR(installDir), "install marker not cleared after a successful removal") } +func TestUninstallManaged_RemovesEngineMarkedInDeclaredInstallDirectory(t *testing.T) { + vendorRoot := t.TempDir() + binary, weights := vendorEngineOnDisk(t, vendorRoot) + manifest := vendorEngineManifest(vendorRoot, filepath.Join(vendorRoot, "models"), []string{vendorRoot}) + platformKey := runtime.GOOS + "/" + runtime.GOARCH + platform := manifest.Platforms[platformKey] + platform.InstallDir = filepath.Join(t.TempDir(), "managed", "vendor") + manifest.Platforms[platformKey] = platform + ex := managedExecutor(t, manifest) + require.NoError(t, writeInstallMarker(platform.InstallDir, "vendor")) + + results := ex.UninstallManaged(context.Background()) + + require.Len(t, results, 1, "declared install directory outcome") + assert.True(t, results[0].Removed, "marked engine was not removed") + assert.Empty(t, results[0].Error, "marked engine removal") + assert.NoFileExists(t, binary, "engine binary survived") + assert.FileExists(t, weights, "removal deleted downloaded models") + assert.False(t, installedByPAIR(platform.InstallDir), "install marker survived removal") +} + // TestUninstallManagedKeepsTheMarkerWhenRemovalFails is the regression guard for // losing ownership. The marker is the only record that an engine is PAIR's, so // clearing it after a failure would leave the engine installed and every future diff --git a/services/nvpair-engine-manager/registry.go b/services/nvpair-engine-manager/registry.go index cccf69057..f1db7792b 100644 --- a/services/nvpair-engine-manager/registry.go +++ b/services/nvpair-engine-manager/registry.go @@ -87,15 +87,19 @@ type Manifest struct { // first; OS-specific Go code is the fallback only for primitives a // manifest cannot express (process termination, console hiding). type Platform struct { - Detect []string `json:"detect,omitempty"` - Install *Install `json:"install,omitempty"` - Uninstall *Uninstall `json:"uninstall,omitempty"` - Runtime Runtime `json:"runtime"` + // InstallDir overrides the default app-data-root engine directory. It is a + // literal user path with the same ~ and OS environment expansion as detect + // paths; its resolved value is exposed to the manifest as {install_dir}. + InstallDir string `json:"install_dir,omitempty"` + Detect []string `json:"detect,omitempty"` + Install *Install `json:"install,omitempty"` + Uninstall *Uninstall `json:"uninstall,omitempty"` + Runtime Runtime `json:"runtime"` // ModelsDir is the engine's on-disk model store, referenced elsewhere as // {models_dir}. Nothing this service removes may delete it: Uninstall skips - // it, and the app-level data purge never reaches it because validation - // rejects a models_dir that is a placeholder (and therefore cannot be - // derived from the install directory, which lives under the app data root). + // it, validation keeps it outside an explicit managed install directory, and + // the bundled-manifest tests keep it outside the app data root. App reset + // removes both kinds of managed application path. // Declare the engine's real default — the path it would use on its own — // so PAIR and a user's own copy of the engine share one library. ModelsDir string `json:"models_dir,omitempty"` @@ -716,9 +720,9 @@ func (r *Registry) mergeOntoBundled(override []byte) ([]byte, error) { } // applyPlatformDefaults implements manifest-level shared defaults: a -// top-level "detect" / "install" / "uninstall" / "runtime" / "models_dir" -// block is merged into every entry under "platforms", with the per-platform block -// winning key-by-key. Nested objects (runtime, runtime.env, …) merge +// top-level "install_dir" / "detect" / "install" / "uninstall" / "runtime" / +// "models_dir" block is merged into every entry under "platforms", with the +// per-platform block winning key-by-key. Nested objects (runtime, runtime.env, …) merge // recursively; arrays and scalars are replaced wholesale. The merge runs // at the JSON level before unmarshalling, so an omitted key truly // inherits the default while an explicit zero value (e.g. "port": 0) @@ -731,7 +735,7 @@ func applyPlatformDefaults(data []byte) ([]byte, error) { return nil, fmt.Errorf("invalid JSON: %w", err) } base := map[string]any{} - for _, k := range []string{"detect", "install", "uninstall", "runtime", "models_dir"} { + for _, k := range []string{"install_dir", "detect", "install", "uninstall", "runtime", "models_dir"} { raw, ok := top[k] if !ok { continue @@ -877,6 +881,24 @@ func (p *Platform) validate(key string) error { if source := p.Runtime.ModelSource; source != "" && source != "huggingface" && source != "modelscope" { return fmt.Errorf("platform %q: invalid model_source", key) } + if p.InstallDir != "" { + if strings.TrimSpace(p.InstallDir) == "" { + return fmt.Errorf("platform %q: install_dir cannot be blank", key) + } + literalPath := p.InstallDir + if !strings.HasPrefix(key, "windows/") { + literalPath = unixEnvRe.ReplaceAllString(literalPath, "") + } + if strings.ContainsAny(literalPath, "{}") { + return fmt.Errorf("platform %q: install_dir %q must be a literal path, not a template", key, p.InstallDir) + } + if p.ModelsDir != "" && pathWithinRoot( + strings.ToLower(filepath.Clean(expandPath(p.InstallDir))), + strings.ToLower(filepath.Clean(expandPath(p.ModelsDir))), + ) { + return fmt.Errorf("platform %q: models_dir %q cannot be inside install_dir %q — app reset removes managed engine payloads", key, p.ModelsDir, p.InstallDir) + } + } if auto := p.Runtime.AutoLaunch; auto != nil { if auto.Kind != autoLaunchHFContext || auto.ContextTokens < 1024 { return fmt.Errorf("platform %q: auto_launch needs kind %q and context_tokens of at least 1024", key, autoLaunchHFContext) @@ -977,9 +999,8 @@ func (p *Platform) validate(key string) error { } } // A models_dir derived from {install_dir} is the mistake this field exists to - // prevent: the install directory lives under the app data root, so the - // app-level "remove all data" uninstall would take the weights with it even - // though no engine uninstall ever touched them. Keeping it literal means the + // prevent: app reset removes every managed install directory, so it would take + // the weights too even though no engine uninstall ever touched them. Keeping it literal means the // only way to point the model store at the data root is to type that path // out, which review will catch. if strings.Contains(p.ModelsDir, "{") { diff --git a/services/nvpair-engine-manager/registry_test.go b/services/nvpair-engine-manager/registry_test.go index 17179a69c..ca15f3cdb 100644 --- a/services/nvpair-engine-manager/registry_test.go +++ b/services/nvpair-engine-manager/registry_test.go @@ -54,6 +54,35 @@ func TestValidate_AcceptsValidManifest(t *testing.T) { require.NoError(t, m.Validate(), "valid manifest rejected") } +func TestValidate_RejectsTemplatedInstallDirectory(t *testing.T) { + m := validManifest() + platform := m.Platforms["linux/amd64"] + platform.InstallDir = "{models_dir}/engine" + m.Platforms["linux/amd64"] = platform + + require.ErrorContains(t, m.Validate(), "install_dir", "templated install directory accepted") +} + +func TestValidate_AcceptsEnvironmentReferenceInInstallDirectory(t *testing.T) { + t.Setenv("NVPAIR_INSTALL_ROOT", t.TempDir()) + m := validManifest() + platform := m.Platforms["linux/amd64"] + platform.InstallDir = "${NVPAIR_INSTALL_ROOT}/engine" + m.Platforms["linux/amd64"] = platform + + require.NoError(t, m.Validate(), "supported braced environment reference rejected") +} + +func TestValidate_RejectsModelStoreInsideDeclaredInstallDirectory(t *testing.T) { + m := validManifest() + platform := m.Platforms["linux/amd64"] + platform.InstallDir = "~/.nvpair/engine-bin/demo" + platform.ModelsDir = "~/.nvpair/engine-bin/demo/models" + m.Platforms["linux/amd64"] = platform + + require.ErrorContains(t, m.Validate(), "models_dir", "model store inside managed install directory accepted") +} + func TestValidate_AcceptsCommandModeAndCmdAction(t *testing.T) { m := validManifest() p := m.Platforms["linux/amd64"] @@ -440,7 +469,7 @@ func TestLoadRegistryMissingDir_IsSkipped(t *testing.T) { } // TestApplyPlatformDefaults covers the shared-base merge: a top-level -// runtime/install block is inherited by each platform, per-platform keys +// runtime/install block and install directory are inherited by each platform, per-platform keys // override, nested objects (env) deep-merge, and a platform can override // a shared scalar. func TestApplyPlatformDefaults_InheritsBaseFieldsAndKeepsPlatformOverrides(t *testing.T) { @@ -449,6 +478,7 @@ func TestApplyPlatformDefaults_InheritsBaseFieldsAndKeepsPlatformOverrides(t *te "engine": "demo", "display_name": "Demo", "manifest_version": 1, + "install_dir": "~/.nvpair/engine-bin/demo", "install": { "mode": "user" }, "runtime": { "args": ["serve"], @@ -464,6 +494,7 @@ func TestApplyPlatformDefaults_InheritsBaseFieldsAndKeepsPlatformOverrides(t *te "runtime": { "bin": "{install_dir}/bin/demo", "env": { "EXTRA": "2" } } }, "linux/arm64": { + "install_dir": "/opt/nvpair/demo", "detect": ["{install_dir}/bin/demo"], "install": { "fetch": { "url": "https://x/demo.tgz" }, "run": ["tar", "xf", "{download}", "-C", "{install_dir}"] }, "uninstall": { "run": ["rm", "-rf", "{install_dir}"] }, @@ -489,6 +520,7 @@ func TestApplyPlatformDefaults_InheritsBaseFieldsAndKeepsPlatformOverrides(t *te assert.Equal(t, "user", amd.Install.ModeOrDefault(), "amd64 install: expected base mode + platform fetch") assert.NotNil(t, amd.Install.Fetch, "amd64 install: expected base mode + platform fetch") } + assert.Equal(t, "~/.nvpair/engine-bin/demo", amd.InstallDir, "amd64 install_dir: expected inherited path") // per-platform override applied: assert.Equal(t, "{install_dir}/bin/demo", amd.Runtime.Bin, "amd64 bin override missing") // nested object (env) deep-merges base + platform keys: @@ -500,6 +532,7 @@ func TestApplyPlatformDefaults_InheritsBaseFieldsAndKeepsPlatformOverrides(t *te require.True(t, ok, "linux/arm64 missing") assert.Equal(t, 5678, arm.Runtime.Port, "arm64 port override expected 5678") assert.Equal(t, "1", arm.Runtime.Env["SHARED"], "arm64 should still inherit base env SHARED") + assert.Equal(t, "/opt/nvpair/demo", arm.InstallDir, "arm64 install_dir override missing") } // TestBundledManifestsMerge guards the actual shipped manifests: they must @@ -724,6 +757,8 @@ func TestBundledVLLMManifest_PinsTheInstallAndSizesTheLaunchOnEachPlatform(t *te // Our managed install must win over an external one. require.GreaterOrEqual(t, len(platform.Detect), 2, "detect must list the managed install and an external one") assert.True(t, strings.HasPrefix(platform.Detect[0], "{install_dir}/"), "detect order = %v, want the managed install first", platform.Detect) + assert.Equal(t, "~/.nvpair/engine-bin/vllm", platform.InstallDir, "use the space-free managed vLLM directory") + assert.False(t, strings.ContainsAny(platform.InstallDir, " \t\r\n"), "the managed install path must not contain whitespace") require.NotNil(t, platform.Install) require.NotNil(t, platform.Install.Fetch, "the uv bootstrap must be a fetch") require.NotEmpty(t, platform.Install.Fetch.SHA256, "the uv bootstrap must be sha256-pinned") @@ -746,11 +781,10 @@ func TestBundledVLLMManifest_PinsTheInstallAndSizesTheLaunchOnEachPlatform(t *te require.NotNil(t, rt.EditableLaunch) assert.Equal(t, []string{"serve"}, rt.EditableLaunch.FixedArgs, "launch must start with the fixed serve subcommand") assert.Equal(t, "~/.cache/huggingface", platform.ModelsDir, "models_dir must be the Hugging Face cache") - // vLLM compiles FlashInfer sampling kernels at first start and that build - // cannot cope with the spaces in PAIR's app-data path - // ("Nvidia Corporation/Personal AI Router"), so it must stay off. It also - // keeps a system CUDA toolkit and C++ compiler from being required. + // Keep the optional sampler off so ordinary models do not need a CUDA + // toolkit solely for sampling. Other kernels can use the space-free path. assert.Equal(t, "0", rt.Env["VLLM_USE_FLASHINFER_SAMPLER"], "the FlashInfer sampler must be disabled") + assert.NotContains(t, rt.Env, "VLLM_DISABLED_KERNELS", "kernels must stay enabled with the space-free install") // vLLM reserves 92% of GPU memory by default and refuses a context above // the model's own limit. Both are decided per model at start, so the // manifest must not pin them in the saved launch. diff --git a/services/nvpair-engine-manager/spec.md b/services/nvpair-engine-manager/spec.md index 309ec554f..8e639233a 100644 --- a/services/nvpair-engine-manager/spec.md +++ b/services/nvpair-engine-manager/spec.md @@ -49,6 +49,10 @@ extensibility story for an open-source product. and must not be started or adopted because its `/models` contract does not expose router residency state. - Per-engine stdout/stderr log capture and structured operational error records, surfaced via the errors pipeline. +- vLLM memory sizing uses native device telemetry or the installed runtime. + Missing telemetry never implies shared memory: only verified integrated + CUDA/ROCm or XPU devices, Apple Silicon Metal, or an explicit CPU engine use + system RAM. Shared-memory readings reserve 3 GiB for runtime startup. - A normalized node-level model list (`engine:models`): union of every running engine's `list_models`, name-extracted via each action's declarative `result` spec, plus the per-engine set of models loaded in memory (`loadedByEngine`, from each engine's `loaded_models` action). An engine with `advertised_models: "loaded"` (vLLM) advertises only its served model in `modelsByEngine` and reports its `list_models` inventory as `downloadedByEngine`, read even while it is stopped; its `loadedByEngine` names models the way that inventory does. A successful explicit empty inventory remains an engine key with `[]`; a missing/malformed/failed inventory omits that engine key instead of being mislabeled as authoritative empty. A watcher polls these lists and pushes `engine:models-changed` when one changes (explicit load/unload, JIT auto-load, TTL/idle eviction, a pull or a delete). - Expose all of the above over the `engine:*` JSON-RPC surface to whatever orchestrates the service, plus an optional plain-HTTP LAN endpoint (`--http-port`, `GET /v1/models`) that serves the model list to a peer's discovery daemon (the list moved off the size-limited mDNS TXT onto HTTP). @@ -236,7 +240,7 @@ fails it with the peer's error message. `~/Library/Application Support/Nvidia Corporation/Personal AI Router` on macOS) plus bundled `manifests/*.json`. Each engine's model store is declared as `models_dir` outside that data dir (`~/.ollama`, `~/.llamacpp`, - `~/.lmstudio/models`) so neither an engine uninstall nor the app-level data + `~/.lmstudio/models`, `~/.cache/huggingface`) so neither an engine uninstall nor the app-level data purge removes downloaded models; clearing one is the user's own choice, made per model through `delete_model`. @@ -247,8 +251,9 @@ fails it with the peer's error message. wherever the filesystem says so. Removal unlinks a symlink or junction rather than following it, and refuses a target that is the store or inside it. - One persisted file: `engine-bin//installed-by-pair.json`, written after - an install this service performed. It is the only record distinguishing an + One persisted file: `installed-by-pair.json` in the engine's managed install + directory (normally `engine-bin/` under app data, or the platform's + explicit `install_dir`), written after an install this service performed. It is the only record distinguishing an install PAIR made from one the user made themselves, which an engine whose vendor script chooses its own destination cannot be told apart any other way, and it gates both `engine:uninstall` for such an engine and