issue/1553 - fix: let the default runtime honour INFINICORE_DEFAULT_DEVICE_INDEX - #1554
Open
JoeZhang-0x000 wants to merge 1 commit into
Open
Conversation
ContextImpl built its default Runtime on device index 0 whatever card the process actually uses, so any process working on another card also opened a whole extra runtime on card 0 -- primary context, a stream, an infiniop handle and two allocators -- on a card it never computes on. In a tensor-parallel job that accumulates: measured with vLLM at TP=4, card 0 carried four processes against two for the reference backend, and on MACA that made TP>=4 hang in an unbounded mxkwCreateQueueBlock retry. default_device_index() now reads INFINICORE_DEFAULT_DEVICE_INDEX so the embedder can name the card this process will use. Unset, invalid or out-of-range values all fall back to 0, matching the previous behaviour exactly. Two supporting changes: getCurrentRuntime() scans the row for the first non-null runtime instead of assuming index 0, since the default one no longer has to live there; and setDevice() no longer calls getCurrentRuntime() merely to compare, because on a cold thread that eagerly builds the default card's runtime just to answer a comparison it is about to overwrite -- which is why calling setDevice() early could not avoid the stray runtime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Author
|
我手动检验过,详细见#1553 里面描述的问题,在沐曦机器上会出现错误。怀疑和沐曦驱动有关系。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联 issue: #1553
问题
ContextImpl的构造函数把默认 runtime 的设备号硬编码为0,所以任何使用非 0 号卡的进程,都会在 0 号卡上额外建一整套 runtime(primary context + 一条流 + 一个 infiniop handle + 两个分配器),尽管它从不在那张卡上计算。PyTorch 没有这个行为。复现方式和实测数据见 #1553。
改动
只动
src/infinicore/context/context_impl.cc,三处:新增
default_device_index(),读环境变量INFINICORE_DEFAULT_DEVICE_INDEX决定默认 runtime 建在哪张卡上,构造函数用它取代硬编码的0。变量未设置、取值非法或越界时一律回落到 0,与现行行为逐字一致,因此对现有用法完全向后兼容。getCurrentRuntime()的惰性初始化改为扫描整行找第一个非空 runtime,不再假定下标0。这是第 1 处的必要配套:默认 runtime 不再必然位于 0,否则这里会读到 nullptr 并一路回落到 CPU runtime。setDevice()不再为了做一次比较而调用getCurrentRuntime(),改为直接判断current_runtime_是否为空。这一处独立成立:在冷线程上,setDevice(cuda:N)的第一行原本就会先把默认卡的 runtime 惰性建出来,仅仅是为了回答一个马上会被覆盖的比较 —— 这也是调用方「提前set_device()」无法规避该问题的原因。嵌入方只需在第一次调用 InfiniCore 之前,把本进程要用的卡号写进
INFINICORE_DEFAULT_DEVICE_INDEX。不新增除
<cstdlib>外的依赖,不改任何公开 API,不动构建配置。验证
环境:MetaX C550(8 卡)/ MACA 3.8.0.23 / PyTorch 2.10.0 / vLLM 0.22.0。
单进程隔离,绑定到 3 号卡:
INFINICORE_DEFAULT_DEVICE_INDEX=3端到端,通过 vLLM 张量并行,运行中用
mx-smi统计 0 号卡上的进程数:vllm-metax(对照)修复后 16 组张量并行用例(3 个模型 × TP 1~8 × 两种后端)全部有效;此前 TP>=4 的用例无一能跑完。
格式化:
clang-format 21.1.8对本文件--dry-run --Werror通过。说明
MACA 驱动为何会因这些多余的 runtime 而超时,我们没有定论,本 PR 也不做任何机制上的声称。已排除的有:不是显存(多余的 runtime 占 0 MiB,把
gpu_memory_utilization降到 0.5 照样挂)、不是每卡 context 数量(8 个进程同时在一张卡上持有 context 正常)、不是每卡流的容量(单进程建 4096 条流正常)。合成负载也复现不出该失败。本 PR 依据的是因果证据:去掉这个多余的 runtime,TP>=4 就从必然卡死变成全部跑通。 而且无论驱动侧的上限是多少,一个进程都不应该在它从不使用的卡上建 runtime。
我们也考虑过在构造函数里用
infinirtGetDevice()自动探测当前设备,这样就不需要环境变量。但它返回的是 InfiniRT 自己维护的CURRENT_DEVICE_ID,而非驱动层的当前设备,在 InfiniRT 被显式setDevice之前拿不到宿主框架已经设好的卡号。要做到自动探测需要新增直达驱动的接口(各后端已有hcGetDevice/cudaGetDevice之类的调用),改动面较大。如果维护者更倾向那个方向,我们乐意按该思路重做。