From c412291168cf96716be13f31725227b35b89a3d7 Mon Sep 17 00:00:00 2001 From: dmmdea <4965968+dmmdea@users.noreply.github.com> Date: Sat, 7 Nov 2026 22:48:10 -0500 Subject: [PATCH 1/3] node-info: list Axera AX650N accelerator cards on Linux Detect PCI vendor 0x1f4b cards and list each as a kind "npu" accelerator row, not inference-ready and outside GPU telemetry, like the Coral and RKNPU rows. A single axcl-smi table call every 5 s (5 s timeout, never overlapping) feeds the card name, CMM pool capacity and usage, NPU utilization and temperature; the last good table is published for at most 20 s. A "--" NPU reading is reported as utilization_unavailable, never as 0. Without the CLI the card is still listed from PCI. Power is not reported by the card and is omitted. gpuStat gains VRAMTotal so a sampler can supply a capacity the static inventory does not know. Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: dmmdea <4965968+dmmdea@users.noreply.github.com> --- services/nvpair-node-info/README.md | 27 +- .../nvpair-node-info/accel_axera_linux.go | 434 ++++++++++++++++++ services/nvpair-node-info/accel_linux.go | 6 +- services/nvpair-node-info/main.go | 3 + services/nvpair-node-info/stats.go | 6 +- services/nvpair-node-info/stats_linux.go | 1 + 6 files changed, 473 insertions(+), 4 deletions(-) create mode 100644 services/nvpair-node-info/accel_axera_linux.go diff --git a/services/nvpair-node-info/README.md b/services/nvpair-node-info/README.md index 9ec84a710..3fbc5ccda 100644 --- a/services/nvpair-node-info/README.md +++ b/services/nvpair-node-info/README.md @@ -105,7 +105,7 @@ Field notes: - `power_watts` is what a device is drawing in whole watts, published only where the hardware meters itself: an NVIDIA GPU (`nvidia-smi`'s `power.draw`, Linux and Windows), an AMD discrete GPU (the amdgpu hwmon's `PPT` input), and on `cpu` the package power — derived from the processor's energy counter, or on a Linux AMD APU host from that APU's `PPT` input, which meters the whole socket. Every other row has no meter and carries no such field — see **Power draw** below for the full table and for why a Linux host normally reports no `cpu.power_watts`. - All dynamic fields and the `cpu` / `memory` objects use `omitempty`: a value the service couldn't read is dropped from the JSON entirely rather than reported as a misleading literal zero. A genuinely idle CPU renders the same as "unknown" — that ambiguity is intentional and benign. - `utilization_unavailable` is `true` on a device row that has **no busy counter** this service can read, and absent everywhere else. It exists because `utilization_percent` is `omitempty`: a measured idle `0` is absent on the wire too, so the absence alone cannot tell "idle" from "cannot tell", and a client that renders both as "0 %" claims an idle device nobody measured. Set on a Hailo module for which neither the inference process's activity file nor the `nvpair-sensors` I/O-activity estimate gives a current figure (HailoRT has no busy counter on Windows; see [Utilization from the activity file](#utilization-from-the-activity-file) and [Utilization from the I/O-activity estimate](#utilization-from-the-io-activity-estimate)), on a Linux Intel GPU whose driver publishes no idle-residency counter and, until its second sample, on one that does (i915/xe keep their engine-busy counters in a root-gated PMU; the figure is derived from RC6 / GT idle residency instead, see [Linux Intel (i915 / xe sysfs)](#linux-intel-i915--xe-sysfs)), and on a Mali GPU or RKNPU row whose sampler has not read its counter — before the first read, or after three consecutive failed reads, rather than freezing the last value. A client must show such a row's usage as unknown; a row without the flag keeps its historical meaning, so an idle GPU on a node that predates the field still reads 0 %. -- `inference_ready` is `false` on a device no engine PAIR runs can use — an Arm Mali GPU, an RKNPU, a Google Coral Edge TPU, a Hailo module, a Linux Intel integrated GPU — and absent on every other row, which makes no claim either way. It travels per row rather than as a node-level id list because clients re-sort the rows and build their own ids. The desktop derives the node's inference-ready list from it, so a board whose only devices are a Mali GPU and an NPU shows its CPU and RAM instead of two idle GPU rings. +- `inference_ready` is `false` on a device no engine PAIR runs can use — an Arm Mali GPU, an RKNPU, a Google Coral Edge TPU, an Axera AX650N card, a Hailo module, a Linux Intel integrated GPU — and absent on every other row, which makes no claim either way. It travels per row rather than as a node-level id list because clients re-sort the rows and build their own ids. The desktop derives the node's inference-ready list from it, so a board whose only devices are a Mali GPU and an NPU shows its CPU and RAM instead of two idle GPU rings. - `memory.total_bytes` and `memory.used_bytes` are on **one base**: the memory the operating system manages, read from the same source the used figure is — `/proc/meminfo` `MemTotal` on Linux (used = `MemTotal - MemAvailable`), `GlobalMemoryStatusEx` `TotalPhys` on Windows (used = `TotalPhys - AvailPhys`), gopsutil's Mach readers on macOS. The Linux Intel, Mali, RKNPU and NVIDIA UMA rows use this same total as their `vram_bytes` ceiling; a Windows integrated GPU does not (see `memory_pool` below), because this base excludes the firmware-reserved memory an APU's graphics carve-out lives in. Installed DIMM capacity is not used: on Windows it counts hardware-reserved memory that can never appear as used, and on Linux it is not readable unprivileged. The Linux total used to be the count of online memory blocks times the block size, which counts the blocks around the PCI hole and at the top of RAM in full: a 64 GiB desktop with 2 GiB blocks published 66 GiB against a `MemTotal` of 62.7 GiB, and RAM % read two points low. - `vram_bytes` is reported through DXGI on Windows, `nvidia-smi` on Linux, and IORegistry on macOS. On a unified-memory NVIDIA GPU such as DGX Spark, Linux uses total physical system memory for `vram_bytes` and the independently sampled system-memory usage for `vram_used_bytes`. On Apple Silicon, `vram_bytes` is total physical unified memory and `vram_used_bytes` is the GPU driver's mapped allocation (`Alloc system memory`), not whole-system RAM usage or the momentarily active subset. - `memory_pool` is absent on a device with memory of its own and `"unified"` on one whose `vram_bytes` is a pool it shares with the host. A client must not label a unified capacity "VRAM", and must not assume such a row reports usage — see **Unified memory rows** below. On Windows an Intel or AMD integrated adapter is classified by the same PCI id table the Linux inventory uses (`nvpair-shared/gpunames`), and an id the table does not list by DXCore's `IsIntegrated` adapter property; such a row's `vram_bytes` is DXGI's `DedicatedVideoMemory + SharedSystemMemory` and its `vram_used_bytes` is PDH's `Dedicated Usage + Shared Usage` — the Windows counterpart of amdgpu's VRAM + GTT on Linux. The dedicated half alone is only the firmware's stolen aperture (128 MB on a UHD 630) or an APU's carve-out, and the host memory total would lose that carve-out (it excludes firmware-reserved memory), so neither is used as the ceiling. Without a `Shared Usage` sample the row publishes no used figure rather than the dedicated half alone. A startup row the OS could not classify takes the integrated classification from a later re-detection; a later pass that cannot say does not undo it. @@ -130,6 +130,7 @@ Several devices in this inventory have no memory of their own: an Intel or AMD i | Intel integrated GPU | i915/xe expose no power attribute to an unprivileged reader | no | | Arm Mali GPU, RKNPU | no meter in the driver | no | | Google Coral Edge TPU | the gasket/apex driver keeps no power attribute | no | +| Axera AX650N card | `axcl-smi` reports `-- / --` for power | no | | Hailo-8L | HailoRT's `measure-power` is unsupported on that module | no | | Apple Silicon GPU | IOAccelerator publishes no power figure | no | @@ -405,6 +406,30 @@ GOOS=linux GOARCH=arm64 go test -c -o rockchip_arm64.test # cross-compile NVPAIR_LIVE_ROCKCHIP=1 ./rockchip_arm64.test -test.v # on the board ``` +### Linux Axera (AX650N accelerator cards via `axcl-smi`) + +An Axera AX650N-based card (for example the M.2 "AX8850"; PCI vendor `0x1f4b`, device `0x0650`, driven by the vendor's `axcl_host` modules) is listed as an accelerator row — `kind:"npu"`, `inference_ready:false` — alongside a Coral Edge TPU or an RKNPU. No engine PAIR runs has an Axera backend, so the row never counts toward GPU pressure and never marks `telemetryValid`. + +The card has its own SoC, DRAM and operating system, so the host kernel exposes no counters for it. What is read: + +| What | Source | +| ---- | ------ | +| Presence, one row per card | `/sys/bus/pci/devices/*` with vendor `0x1f4b`. Listed even when the vendor CLI is missing or failing; such a row is named from the PCI ids (`"Axera AX650N"`), carries `utilization_unavailable:true`, and has no memory or temperature. | +| Name | The card name `axcl-smi` prints plus the chip where known: `"Axera AX8850 (AX650N)"`. | +| `vram_bytes` / `vram_used_bytes` | The **CMM** pool from the `CMM-Usage` column — the contiguous memory NPU models and media buffers allocate from (2944 MiB on an AX8850). The card's `Memory-Usage` column is the card's own Linux RAM and is not reported. The pool belongs to the card, so there is no `memory_pool:"unified"`. | +| `utilization_percent` | The table's **NPU** column. Measured on an AX8850 running a YOLO11s model in a loop: 0 % idle, 89-93 % under load, back to 0 % when the run ended. `axcl-smi info --npu` is not used: it prints `NPU Usage (%) : --` at idle and a differently shaped `vnpu-Non : 92%%` line under load. A `--` in the column means no reading and sets `utilization_unavailable:true`; it is never published as 0. | +| `temperature_celsius` | The `Temp` field of the same table, whole degrees. | + +One `axcl-smi` table call (about 0.6 s; it opens a session on the card) runs every **5 s** on its own goroutine, with a 5 s timeout and never overlapping the previous call, and one call is also made at startup so the first response already has numbers. A response uses the last good table only while it is at most 20 s old; after that the memory, utilization and temperature are dropped rather than frozen. A single failed call (the CLI returns no table if an operator runs a second `axcl-smi` at the same moment) is therefore invisible, and a warning is logged only after three consecutive failures. The CLI's own `file sink initialization failed` message on stderr is ignored. + +**Power is a ceiling.** The card reports none (`Pwr:Usage/Cap` is `-- / --`), so `power_watts` is omitted, and there is no fan. + +The CLI is expected at `/usr/bin/axcl/axcl-smi` (the AXCL package's install path) and must be runnable by the user the service runs as. + +```json +{"name": "Axera AX8850 (AX650N)", "vram_bytes": 3087007744, "vram_used_bytes": 35651584, "utilization_percent": 91, "kind": "npu", "temperature_celsius": 37, "inference_ready": false} +``` + ## Shutdown The service shuts down on: diff --git a/services/nvpair-node-info/accel_axera_linux.go b/services/nvpair-node-info/accel_axera_linux.go new file mode 100644 index 000000000..beb78aaf9 --- /dev/null +++ b/services/nvpair-node-info/accel_axera_linux.go @@ -0,0 +1,434 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//go:build linux + +package main + +import ( + "context" + "log/slog" + "os" + "os/exec" + "path/filepath" + "regexp" + "strconv" + "strings" + "sync" + "sync/atomic" + "time" + + "nvpair-shared/noderec" +) + +// Axera AX650N accelerator cards (the M.2 "AX8850" and its relatives) inventory +// and sampling. +// +// The card is a PCIe endpoint (vendor 0x1f4b, class "Multimedia video +// controller") driven by the vendor's axcl_host / ax_pcie_host_dev modules. It +// has its own SoC, its own 4 GB of DRAM and its own Linux, so the host kernel +// exposes no counters for it: the only supported readout is the vendor CLI +// axcl-smi. The row is listed in the same inventory as a Coral Edge TPU or an +// RKNPU, with Kind = noderec.GPUKindAccelerator: an inference accelerator PAIR's +// engines cannot run on, so it never counts toward GPU pressure and never marks +// GPU telemetry fresh. +// +// Inventory comes from PCI (every vendor 0x1f4b device), so the card is listed +// even when axcl-smi is missing or failing — then it carries a name and +// utilization_unavailable, and no memory or temperature. +// +// The dynamic numbers come from ONE axcl-smi table call every axeraSampleInterval +// (a call takes ~0.6 s and opens a session on the card, so it is not run on the +// collector's 1 s tick). The table is the single source because it is the one +// place every figure is truthful: +// +// - utilization: the NPU column. Measured under a looping model run, it reads +// 0 % idle and 89-92 % under load. "axcl-smi info --npu" is NOT used: it +// prints "NPU Usage (%) : --" at idle and a differently shaped +// "vnpu-Non : 92%%" line under load, so it cannot tell idle from unreadable. +// A "--" in the table's NPU column is treated as no reading, never as 0. +// - memory: the CMM pool (Contiguous Memory Manager), which is what NPU models +// and media buffers allocate from. The card's "Memory-Usage" column is its +// own Linux RAM, not the NPU's, so it is not reported. The pool belongs to +// the card, so the row is not memory_pool:"unified". +// - temperature: the card temperature, whole degrees. +// - power: the card reports none ("-- / --"), so power_watts is omitted. +// +// The sampler keeps its last good table for axeraFreshFor; once that is older the +// dynamic fields are dropped, so a wedged CLI never freezes a stale figure. + +const ( + axeraPCIVendor = "1f4b" + axeraPCIDevicesDir = "/sys/bus/pci/devices" + axeraStatsPrefix = "axera:" + + axeraSMIPath = "/usr/bin/axcl/axcl-smi" + axeraSampleInterval = 5 * time.Second + axeraSMITimeout = 5 * time.Second + // axeraFreshFor is how old a good sample may be and still be published: + // four sample intervals, so a single slow or failed call does not blank the + // row but a dead CLI does. + axeraFreshFor = 20 * time.Second +) + +// axeraProductNames maps a PCI "vendor:device" id pair to the display name used +// when axcl-smi gives none. Unknown ids get a generic label carrying the ids. +var axeraProductNames = map[string]string{ + "1f4b:0650": "Axera AX650N", +} + +// axeraChips maps the card name axcl-smi prints to the SoC on it. +var axeraChips = map[string]string{ + "AX8850": "AX650N", +} + +// axeraDevice is one Axera PCI function. +type axeraDevice struct { + bdf string // lower-case PCI address, e.g. 0000:04:00.0 + vendor string + device string +} + +func axeraStatsKey(bdf string) string { return axeraStatsPrefix + strings.ToLower(bdf) } + +// listAxeraDevices scans the PCI bus for Axera functions, sorted by address. +func listAxeraDevices(dir string) []axeraDevice { + var out []axeraDevice + for _, name := range sortedDirNames(dir) { + vendor := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(readSysfs(filepath.Join(dir, name, "vendor")))), "0x") + if vendor != axeraPCIVendor { + continue + } + device := strings.TrimPrefix(strings.ToLower(strings.TrimSpace(readSysfs(filepath.Join(dir, name, "device")))), "0x") + out = append(out, axeraDevice{bdf: strings.ToLower(name), vendor: vendor, device: device}) + } + return out +} + +// axeraPresenceName is the name of a card axcl-smi has not (yet) described. +func axeraPresenceName(vendor, device string) string { + if name, ok := axeraProductNames[vendor+":"+device]; ok { + return name + } + if device == "" { + return "Axera accelerator" + } + return "Axera accelerator (" + vendor + ":" + device + ")" +} + +// axeraCardName is "Axera AX8850 (AX650N)" from the name axcl-smi prints. +func axeraCardName(cardName string) string { + cardName = strings.TrimSpace(cardName) + if cardName == "" { + return "" + } + name := "Axera " + cardName + if chip, ok := axeraChips[cardName]; ok { + name += " (" + chip + ")" + } + return name +} + +// axeraRow is the inventory row. Capacity, usage, utilization and temperature +// are filled from the sampler by buildResponse (statsKey join); until the first +// good sample, and whenever the sample is stale, the row says +// utilization_unavailable. It is not inference-ready: no engine PAIR runs has an +// Axera backend. +func axeraRow(dev axeraDevice, name string) GPUInfo { + if name == "" { + name = axeraPresenceName(dev.vendor, dev.device) + } + return GPUInfo{ + Name: name, + Kind: noderec.GPUKindAccelerator, + InferenceReady: notInferenceReady(), + statsKey: axeraStatsKey(dev.bdf), + utilizationNeedsSample: true, + } +} + +// axclCard is one card block of the axcl-smi table. +type axclCard struct { + index int + name string + busID string // lower-case PCI address + tempC uint32 + tempOK bool + npuPct uint32 + npuOK bool // false for "--": no reading, not idle + cmmUsed uint64 + cmmTotal uint64 + cmmOK bool +} + +var ( + axclBusIDPattern = regexp.MustCompile(`^[0-9a-fA-F]{4}:[0-9a-fA-F]{2}:[0-9a-fA-F]{2}\.[0-7]$`) + axclTempPattern = regexp.MustCompile(`(\d+)\s*C\b`) + axclUtilPattern = regexp.MustCompile(`^(\d+|--)\s*%?\s+(\d+|--)\s*%?$`) + axclMemPattern = regexp.MustCompile(`^(\d+)\s*([KMG]iB)\s*/\s*(\d+)\s*([KMG]iB)$`) +) + +func axclUnitBytes(unit string) uint64 { + switch unit { + case "KiB": + return 1 << 10 + case "MiB": + return 1 << 20 + case "GiB": + return 1 << 30 + } + return 0 +} + +// parseAxclSmiTable decodes the card blocks of the default axcl-smi table: +// +// | 0 AX8850 V3.16.0 | 0000:04:00.0 | 151 MiB / 945 MiB | +// | -- 33C -- / -- | 0% 0% | 18 MiB / 2944 MiB | +// +// A card is two consecutive three-cell rows: the first has an index/name/firmware +// cell and a PCI address, the second the fan/temperature/power cell, the CPU and +// NPU percentages and the CMM usage. Rows with another cell count (the header, +// the process list) are skipped, so a card with processes on it parses the same. +// Fields that do not parse are left unset (…OK false) rather than zeroed. +func parseAxclSmiTable(s string) []axclCard { + var cards []axclCard + var cur *axclCard + for _, line := range strings.Split(s, "\n") { + line = strings.TrimSpace(line) + if len(line) < 2 || line[0] != '|' || line[len(line)-1] != '|' { + cur = nil + continue + } + cells := strings.Split(line[1:len(line)-1], "|") + if len(cells) != 3 { + cur = nil + continue + } + for i := range cells { + cells[i] = strings.TrimSpace(cells[i]) + } + if axclBusIDPattern.MatchString(cells[1]) { + f := strings.Fields(cells[0]) + if len(f) < 2 { + cur = nil + continue + } + idx, err := strconv.Atoi(f[0]) + if err != nil { + cur = nil + continue + } + rest := f[1:] + if len(rest) > 1 { + rest = rest[:len(rest)-1] // trailing firmware version + } + cards = append(cards, axclCard{ + index: idx, + name: strings.Join(rest, " "), + busID: strings.ToLower(cells[1]), + }) + cur = &cards[len(cards)-1] + continue + } + if cur == nil { + continue + } + // Second row of the block. + if m := axclTempPattern.FindStringSubmatch(cells[0]); m != nil { + if v, err := strconv.ParseUint(m[1], 10, 32); err == nil { + cur.tempC, cur.tempOK = uint32(v), true + } + } + if m := axclUtilPattern.FindStringSubmatch(cells[1]); m != nil && m[2] != "--" { + if v, err := strconv.ParseUint(m[2], 10, 32); err == nil { + if v > 100 { + v = 100 + } + cur.npuPct, cur.npuOK = uint32(v), true + } + } + if m := axclMemPattern.FindStringSubmatch(cells[2]); m != nil { + used, e1 := strconv.ParseUint(m[1], 10, 64) + total, e2 := strconv.ParseUint(m[3], 10, 64) + if e1 == nil && e2 == nil && axclUnitBytes(m[2]) > 0 && axclUnitBytes(m[4]) > 0 { + cur.cmmUsed = used * axclUnitBytes(m[2]) + cur.cmmTotal = total * axclUnitBytes(m[4]) + cur.cmmOK = cur.cmmTotal > 0 + } + } + cur = nil + } + return cards +} + +// axeraSnapshot is one published table: the time it was read and, per PCI +// address, the card's name and stat. +type axeraSnapshot struct { + at time.Time + stats map[string]gpuStat + names map[string]string +} + +// axeraSampler runs axcl-smi on its own goroutine, one call at a time (the loop +// is the single-flight guard: a call that overruns delays the next, it never +// overlaps it), and publishes the latest parsed table atomically. +type axeraSampler struct { + smiPath string + latest atomic.Pointer[axeraSnapshot] + fails int // consecutive failed calls; sampler goroutine only + warned bool // sampler goroutine only + stop chan struct{} + done chan struct{} +} + +func (s *axeraSampler) sample() { + ctx, cancel := context.WithTimeout(context.Background(), axeraSMITimeout) + defer cancel() + cmd := exec.CommandContext(ctx, s.smiPath) + // The CLI prints a harmless log-sink permission error on stderr; stdout is + // the table. WaitDelay bounds the wait if the killed CLI left a child + // holding the pipe. + cmd.WaitDelay = time.Second + out, err := cmd.Output() + cards := parseAxclSmiTable(string(out)) + if err != nil || len(cards) == 0 { + // A second axcl-smi session running at the same moment (an operator at a + // shell) makes one call return no table; that is not a fault, and the + // last good table stays published for axeraFreshFor. Warn only when it + // persists for as long as the table stays fresh. + s.fails++ + slog.Debug("axcl-smi gave no usable table", "path", s.smiPath, "err", err, "cards", len(cards), "consecutive", s.fails) + if s.fails >= 3 && !s.warned { + s.warned = true + slog.Warn("axcl-smi has given no usable table for several samples; Axera card telemetry is omitted until it does", + "path", s.smiPath, "err", err, "cards", len(cards)) + } + return + } + s.fails = 0 + if s.warned { + s.warned = false + slog.Info("axcl-smi recovered", "cards", len(cards)) + } + snap := &axeraSnapshot{ + at: time.Now(), + stats: make(map[string]gpuStat, len(cards)), + names: make(map[string]string, len(cards)), + } + for _, c := range cards { + st := gpuStat{UtilizationPct: c.npuPct, UtilizationKnown: c.npuOK} + if c.tempOK { + st.TemperatureC = c.tempC + } + if c.cmmOK { + st.VRAMTotal, st.VRAMUsed = c.cmmTotal, c.cmmUsed + } + snap.stats[axeraStatsKey(c.busID)] = st + snap.names[axeraStatsKey(c.busID)] = axeraCardName(c.name) + } + s.latest.Store(snap) +} + +func (s *axeraSampler) run() { + defer close(s.done) + ticker := time.NewTicker(axeraSampleInterval) + defer ticker.Stop() + // The first table was taken synchronously by detection, so wait one interval. + for { + select { + case <-s.stop: + return + case <-ticker.C: + s.sample() + } + } +} + +// fresh returns the last good snapshot while it is recent enough to publish. +func (s *axeraSampler) fresh(now time.Time) (*axeraSnapshot, bool) { + p := s.latest.Load() + if p == nil || now.Sub(p.at) > axeraFreshFor { + return nil, false + } + return p, true +} + +func (s *axeraSampler) Stop() { + close(s.stop) + <-s.done +} + +// axeraSamplerSet holds the process-wide sampler, started once by detection and +// living as long as the inventory rows it feeds (the process). +var axeraSamplerSet struct { + mu sync.Mutex + s *axeraSampler +} + +func runningAxeraSampler() *axeraSampler { + axeraSamplerSet.mu.Lock() + defer axeraSamplerSet.mu.Unlock() + return axeraSamplerSet.s +} + +// detectAxeraCards returns one accelerator row per Axera PCI function and, when +// the vendor CLI is installed, starts the sampler. A host without an Axera card +// returns nil without touching the CLI. +func detectAxeraCards() []GPUInfo { + devs := listAxeraDevices(axeraPCIDevicesDir) + if len(devs) == 0 { + return nil + } + rows := make([]GPUInfo, 0, len(devs)) + for _, d := range devs { + rows = append(rows, axeraRow(d, "")) + slog.Debug("Axera card detected", "bdf", d.bdf, "device", d.vendor+":"+d.device) + } + if info, err := os.Stat(axeraSMIPath); err != nil || info.IsDir() { + slog.Info("Axera card present but axcl-smi is not installed; memory, utilization and temperature will be omitted", + "path", axeraSMIPath) + return rows + } + axeraSamplerSet.mu.Lock() + defer axeraSamplerSet.mu.Unlock() + if axeraSamplerSet.s != nil { + return rows + } + // One synchronous table at startup (~0.6 s, like nvidia-smi's first query) + // so the rows are named from it and the first response already has numbers. + s := &axeraSampler{smiPath: axeraSMIPath, stop: make(chan struct{}), done: make(chan struct{})} + s.sample() + if snap := s.latest.Load(); snap != nil { + for i := range rows { + if name := snap.names[rows[i].statsKey]; name != "" { + rows[i].Name = name + } + } + } + axeraSamplerSet.s = s + go s.run() + return rows +} + +// mergeAxeraStats folds the sampler's last fresh table into the snapshot's GPU +// map under each card's statsKey. Like the other accelerator samplers it leaves +// GPUSampledAt untouched, and it clones the map first because on a stale-preserve +// tick snap.GPU aliases the previously published one. +func mergeAxeraStats(snap *statsSnapshot) { + s := runningAxeraSampler() + if s == nil { + return + } + p, ok := s.fresh(time.Now()) + if !ok { + return + } + merged := make(map[string]gpuStat, len(snap.GPU)+len(p.stats)) + for k, v := range snap.GPU { + merged[k] = v + } + for k, v := range p.stats { + merged[k] = v + } + snap.GPU = merged +} diff --git a/services/nvpair-node-info/accel_linux.go b/services/nvpair-node-info/accel_linux.go index 3047fa34f..94ab1d4eb 100644 --- a/services/nvpair-node-info/accel_linux.go +++ b/services/nvpair-node-info/accel_linux.go @@ -85,7 +85,8 @@ func listApexDevices() []apexDevice { return devs } -// detectAccelerators returns one inventory row per apex device. The name is +// detectAccelerators returns one inventory row per apex device (plus any Axera +// card, see accel_axera_linux.go). The name is // resolved from the PCI ids behind the class entry's `device` link; the // dynamic fields are filled by buildResponse from the sampler keyed by // accelStatsKey. @@ -96,7 +97,8 @@ func detectAccelerators() []GPUInfo { device := readSysfs(filepath.Join(dev.dir, "device", "device")) out = append(out, apexRow(apexProductName(vendor, device), dev.name)) } - return out + // Axera AX650N cards (accel_axera_linux.go) join the same accelerator list. + return append(out, detectAxeraCards()...) } // apexRow is one Edge TPU's inventory row. Its utilization is a real diff --git a/services/nvpair-node-info/main.go b/services/nvpair-node-info/main.go index ac9b65ca3..6e99df59e 100644 --- a/services/nvpair-node-info/main.go +++ b/services/nvpair-node-info/main.go @@ -308,6 +308,9 @@ func buildResponseAt(gpus []GPUInfo, cpuStatic *CPUInfo, memTotal uint64, snap s default: gpu.VramUsedBytes = s.VRAMUsed } + if s.VRAMTotal > 0 { + gpu.VramBytes = s.VRAMTotal + } gpu.UtilizationPercent = s.UtilizationPct if s.UtilizationKnown { gpu.UtilizationSource = s.UtilizationSource diff --git a/services/nvpair-node-info/stats.go b/services/nvpair-node-info/stats.go index 5b60a32f2..08f7ed762 100644 --- a/services/nvpair-node-info/stats.go +++ b/services/nvpair-node-info/stats.go @@ -19,7 +19,11 @@ import ( // way; a row that has no source at all says so with // GPUInfo.UtilizationUnavailable instead of relying on the absence. type gpuStat struct { - VRAMUsed uint64 + VRAMUsed uint64 + // VRAMTotal is the device's memory capacity when only the sampler knows it + // (an Axera card's CMM pool, which exists only in the vendor CLI's output); + // zero leaves the row's static vram_bytes alone. + VRAMTotal uint64 UtilizationPct uint32 // TemperatureC is filled only by sources that expose a device thermal // readout (the Linux accelerator sampler); zero means unavailable. diff --git a/services/nvpair-node-info/stats_linux.go b/services/nvpair-node-info/stats_linux.go index 022889062..9c5ae04ec 100644 --- a/services/nvpair-node-info/stats_linux.go +++ b/services/nvpair-node-info/stats_linux.go @@ -214,6 +214,7 @@ func (c *statsCollector) decodeSnapshot() *statsSnapshot { applyGPUStats(previous, snap, gpu, sampledAt) c.mergeAccelStats(snap) mergeRockchipStats(snap) + mergeAxeraStats(snap) mergeIntelStats(c.intelUtil, snap) return snap } From 5fdf70f658813279813191c5ea344f9c9c0b1c4c Mon Sep 17 00:00:00 2001 From: dmmdea <4965968+dmmdea@users.noreply.github.com> Date: Sat, 7 Nov 2026 22:48:13 -0500 Subject: [PATCH 2/3] node-info 0.27.2: GPU temperature on R620+ drivers, stable DXGI out-params Two Windows node-info defects seen on a three-card host: - No GPU temperature at all. NVIDIA-SMI 621.14 prints temperature.gpu with decimals ("78.76"); the integer-only parse rejected every row, so every card lost its temperature while power and utilization (parsed as floats/ints) stayed. parseCelsius (beside parseWatts) accepts whole and fractional degrees and rounds; the Linux collector uses it too. - Intermittent "IDXGIFactory1::EnumAdapters1 failed; stopping enumeration idx=0 hr=0x00000000" (hundreds per day). vtableCall receives out-parameters as uintptr(unsafe.Pointer(&local)); without //go:uintptrescapes the local stays on the goroutine stack, and a stack move inside the call makes DXGI write to the old address, so the caller sees S_OK with a nil adapter and the pass reports no adapters (the temperature join and inventory refresh then work from an empty list). go build -gcflags=-m: before "adapter does not escape", after "moved to heap: adapter". Same directive as syscall.(*LazyProc).Call. Signed-off-by: dmmdea <4965968+dmmdea@users.noreply.github.com> --- services/nvpair-node-info/gpu_windows.go | 12 ++++++++++++ services/nvpair-node-info/gputemp_windows.go | 4 ++-- services/nvpair-node-info/stats.go | 14 ++++++++++++++ services/nvpair-node-info/stats_linux.go | 9 +++++---- services/versions.json | 2 +- 5 files changed, 34 insertions(+), 7 deletions(-) diff --git a/services/nvpair-node-info/gpu_windows.go b/services/nvpair-node-info/gpu_windows.go index 9f598bb5f..95456d4ca 100644 --- a/services/nvpair-node-info/gpu_windows.go +++ b/services/nvpair-node-info/gpu_windows.go @@ -109,6 +109,18 @@ const ( // The first machine word of a COM object points at its method table; each // table slot is a function pointer. The implicit `this` (= iface) is // prepended to args automatically. +// +// Callers pass out-parameters as uintptr(unsafe.Pointer(&local)). Without the +// directive below that conversion hides the pointer from escape analysis, so +// the local stays on the goroutine stack; if the stack grows inside this call +// (append, SyscallN) the runtime copies it and the system writes the result to +// the old address. The caller then sees S_OK with its local still nil — the +// intermittent "EnumAdapters1 failed ... hr=0x00000000" warning, which emptied +// the adapter list and the temperature join for that pass. uintptrescapes moves +// such pointees to the heap and keeps them alive for the call, exactly as +// syscall.(*LazyProc).Call and x/sys/windows.(*Proc).Call do. +// +//go:uintptrescapes func vtableCall(iface unsafe.Pointer, index int, args ...uintptr) uintptr { vtbl := *(**[64]uintptr)(iface) fn := vtbl[index] diff --git a/services/nvpair-node-info/gputemp_windows.go b/services/nvpair-node-info/gputemp_windows.go index 140379343..1b866d343 100644 --- a/services/nvpair-node-info/gputemp_windows.go +++ b/services/nvpair-node-info/gputemp_windows.go @@ -233,8 +233,8 @@ func parseNvidiaSensors(out string) map[string]gpuSensorSample { continue } var sample gpuSensorSample - if c, err := strconv.ParseUint(fields[1], 10, 32); err == nil { - sample.TemperatureC = uint32(c) + if c, ok := parseCelsius(fields[1]); ok { + sample.TemperatureC = c } if len(fields) >= 3 { if w, ok := parseWatts(fields[2]); ok { diff --git a/services/nvpair-node-info/stats.go b/services/nvpair-node-info/stats.go index 08f7ed762..a15605688 100644 --- a/services/nvpair-node-info/stats.go +++ b/services/nvpair-node-info/stats.go @@ -128,6 +128,20 @@ func parseWatts(s string) (float64, bool) { return math.Round(v), true } +// parseCelsius decodes nvidia-smi's temperature.gpu column (-nounits) to whole +// degrees Celsius. Drivers have printed whole degrees ("61") for years, but the +// R620+ Windows driver prints fractional degrees ("78.76"); an integer-only +// parse dropped every reading on such a host, so the card showed no +// temperature at all. [N/A] and garbage report unavailable (the field is then +// omitted), and the value is rounded like power draw. +func parseCelsius(s string) (uint32, bool) { + v, err := strconv.ParseFloat(strings.TrimSpace(s), 64) + if err != nil || math.IsNaN(v) || math.IsInf(v, 0) || v < 0 || v > 1000 { + return 0, false + } + return uint32(math.Round(v)), true +} + // parseEngineInstance pulls the adapter-LUID key and the engine-type tag // out of a PDH "GPU Engine" counter instance name, which looks like: // diff --git a/services/nvpair-node-info/stats_linux.go b/services/nvpair-node-info/stats_linux.go index 9c5ae04ec..02b3ee285 100644 --- a/services/nvpair-node-info/stats_linux.go +++ b/services/nvpair-node-info/stats_linux.go @@ -443,11 +443,12 @@ func parseNvidiaDynamic(out string) (map[string]gpuStat, int) { if mib, err := strconv.ParseUint(fields[2], 10, 64); err == nil { stat.VRAMUsed = mib * 1024 * 1024 } - // temperature.gpu is whole degrees Celsius; [N/A] (some virtual or - // headless SKUs) and a missing column leave it zero (omitted). + // temperature.gpu is degrees Celsius, whole on most drivers and + // fractional on newer ones (parseCelsius rounds both); [N/A] (some + // virtual or headless SKUs) and a missing column leave it zero (omitted). if len(fields) >= 4 { - if c, err := strconv.ParseUint(fields[3], 10, 32); err == nil { - stat.TemperatureC = uint32(c) + if c, ok := parseCelsius(fields[3]); ok { + stat.TemperatureC = c } } // power.draw is watts with two decimals under -nounits, rounded to diff --git a/services/versions.json b/services/versions.json index 277e30117..efa0f7bbd 100644 --- a/services/versions.json +++ b/services/versions.json @@ -3,7 +3,7 @@ "services": "2.2.0", "components": { "nvpair-proxy": "1.4.1", - "nvpair-node-info": "0.27.1", + "nvpair-node-info": "0.27.2", "nvpair-sensors": "0.5.0", "nvpair-node-scanner": "0.22.1", "nvpair-manual-nodes": "0.12.1", From 8e0281f35df76158da2b6282f8a0035a80f26a51 Mon Sep 17 00:00:00 2001 From: dmmdea <4965968+dmmdea@users.noreply.github.com> Date: Sat, 7 Nov 2026 22:48:16 -0500 Subject: [PATCH 3/3] node-info 0.28.0: Axera AX650N accelerator rows + R620+ GPU temperature fix Signed-off-by: dmmdea <4965968+dmmdea@users.noreply.github.com> --- services/versions.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/services/versions.json b/services/versions.json index efa0f7bbd..e3a05bbc1 100644 --- a/services/versions.json +++ b/services/versions.json @@ -3,7 +3,7 @@ "services": "2.2.0", "components": { "nvpair-proxy": "1.4.1", - "nvpair-node-info": "0.27.2", + "nvpair-node-info": "0.28.0", "nvpair-sensors": "0.5.0", "nvpair-node-scanner": "0.22.1", "nvpair-manual-nodes": "0.12.1",