Skip to content

[CI] Add Intel GPU Kubernetes support to the pipeline generator - #857

Open
zxd1997066 wants to merge 2 commits into
vllm-project:mainfrom
zxd1997066:xiangdong/k8s
Open

zxd1997066 wants to merge 2 commits into
vllm-project:mainfrom
zxd1997066:xiangdong/k8s

Conversation

@zxd1997066

Copy link
Copy Markdown

Summary

Add intel-gpu-k8s support to the pipeline generator to enable the Intel GPU CI migration to Kubernetes.

Changes

  • Recognize intel_gpu_k8s and route jobs to the intel-gpu-k8s queue.
  • Add an Intel-specific Kubernetes pod template using XPU images and GPU resource claims.
  • Support an optional resource_claim_template, defaulting to b70-{num_devices}.
  • Scale CPU, host memory, and shared-memory resources with the requested GPU count.
  • Skip NVIDIA-specific setup and kernel/function recording for Intel GPU Kubernetes jobs.

Validation

Verified in vllm-project/vllm#63311, Results match the intel-gpu test run.

@zxd1997066

zxd1997066 commented Nov 3, 2026 •

Copy link
Copy Markdown
Author

Hi @khluu , Could you please take a look at this? We are preparing to migrate Intel CI to Kubernetes.

@Thangnguyenvn98 Thangnguyenvn98 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, and for validating it in vllm-project/vllm#63311. A few things before we merge, mostly so the generator stays small for the people maintaining it:

  1. Workspace path (biggest one). The XPU image keeps the source at /workspace/vllm, and agent-stack-k8s mounts its own /workspace volume over it, which is why the pod needs the copy-vllm-src init container. The generator's automatic sharding also expects the source at /vllm-workspace (runtime_shard.CONTAINER_WORKSPACE), so with the current layout these steps can never be auto-sharded. If the XPU test image put the source at /vllm-workspace like docker/Dockerfile does, the init container, the workingDir and PYTHONPATH overrides, and the working_dir line on every Intel step could all go, and the steps could use automatic_shard. Would that work on your side?
  2. Tests. Please add a test for an intel_gpu_k8s step, following buildkite/tests/pipeline_generator/test_l4_k8s.py: queue, claim template name, image tag, CPU/memory/shm for 1 and 4 GPUs, no NVIDIA setup or recorder commands, and the timeout not raised by the recording factor.
  3. resource_claim_template. Every step in #63311 sets it to the default b70-{num_devices}, so the field isn't needed yet. Could we drop it and add it when a step needs a different template?

Smaller suggestions are inline.

],
"initContainers": [
{
"name": "copy-vllm-src",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mounts a volume named workspace that the template doesn't define, so it depends on agent-stack-k8s injecting it. If the init container stays (see point 1 in the summary), a one-line comment saying so would help the next reader.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the init container as part of the workspace change

step.resource_claim_template or f"b70-{num_gpus}"
)
for container in pod_spec["containers"] + pod_spec["initContainers"]:
container["image"] = f"{image}-xpu"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other image suffixes (-cpu, -arm64, -torch-nightly) are added in utils_lib/docker_utils.get_image. Could -xpu live there too (get_image(xpu=True))? Then the tag logic stays in one place; as it is, a torch-nightly config would ask for <commit>-torch-nightly-xpu, which image_build_xpu.sh never builds.

@zxd1997066 zxd1997066 Nov 15, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the -xpu suffix handling into get_image(xpu=True). It now uses the -xpu tag without the -torch-nightly suffix, matching the XPU image build.

plugin = copy.deepcopy(intel_gpu_k8s_plugin_template)
pod_spec = plugin["kubernetes"]["podSpec"]
pod_spec["resourceClaims"][0]["resourceClaimTemplateName"] = (
step.resource_claim_template or f"b70-{num_gpus}"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See point 3: with the field gone this becomes f"b70-{num_gpus}".

Comment thread buildkite/pipeline_generator/step.py Outdated
device: Optional[str] = None
agent_tags: Optional[Dict[str, str]] = None
num_devices: Optional[int] = None
resource_claim_template: Optional[str] = Field(default=None, min_length=1)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See point 3.

enabled = kernrec_enabled()
return (
enabled
and step.device != DeviceType.INTEL_GPU_K8S

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These checks are needed because _recording_timeout is called with the default "nvidia" profile, not because the recorder predicates are wrong. A small helper that returns the setup profile for a step ("amd", "none" for intel_gpu_k8s, else "nvidia"), used both here at :1246 and for the timeout, would let both device checks go. It also avoids a subtle case: an AMD mirror keeps its parent's device, so a check on step.device would also turn off ROCm recording for an Intel step's AMD mirror.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a setup profile helper and used it for command setup and timeout calculation. Intel K8s uses none, while AMD mirrors explicitly use amd regardless of the parent step’s device. The device checks in the recorder predicates have been removed.

# command step
step_commands = _prepare_commands(step, variables_to_inject)
setup_profile: SetupProfile = (
"none" if step.device == DeviceType.INTEL_GPU_K8S else "nvidia"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the comment at :494.

"""
return (
fnrec_enabled()
and step.device != DeviceType.INTEL_GPU_K8S

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the comment at :494.

Signed-off-by: zengxian <xiangdong.zeng@intel.com>
Signed-off-by: zengxian <xiangdong.zeng@intel.com>
@zxd1997066

zxd1997066 commented Nov 15, 2026 •

Copy link
Copy Markdown
Author

Thanks for this, and for validating it in vllm-project/vllm#63311. A few things before we merge, mostly so the generator stays small for the people maintaining it:

  1. Workspace path (biggest one). The XPU image keeps the source at /workspace/vllm, and agent-stack-k8s mounts its own /workspace volume over it, which is why the pod needs the copy-vllm-src init container. The generator's automatic sharding also expects the source at /vllm-workspace (runtime_shard.CONTAINER_WORKSPACE), so with the current layout these steps can never be auto-sharded. If the XPU test image put the source at /vllm-workspace like docker/Dockerfile does, the init container, the workingDir and PYTHONPATH overrides, and the working_dir line on every Intel step could all go, and the steps could use automatic_shard. Would that work on your side?
  2. Tests. Please add a test for an intel_gpu_k8s step, following buildkite/tests/pipeline_generator/test_l4_k8s.py: queue, claim template name, image tag, CPU/memory/shm for 1 and 4 GPUs, no NVIDIA setup or recorder commands, and the timeout not raised by the recording factor.
  3. resource_claim_template. Every step in #63311 sets it to the default b70-{num_devices}, so the field isn't needed yet. Could we drop it and add it when a step needs a different template?

Smaller suggestions are inline.

Thanks for the review!

  1. Workspace: Yes, this works on our side. I updated the XPU image to use /vllm-workspace [XPU][CI]Update working_dir from /workspace/vllm to /vllm-workspace/ vllm#66973 and removed the source-copy init container and the workingDir/PYTHONPATH overrides from the Intel K8s template. Automatic sharding is not enabled yet; we can address that separately after the migration.
  2. Tests: Added the test in buildkite/tests/pipeline_generator/test_intel_k8s.py
  3. Resource claim template: Removed the optional resource_claim_template field for now, since the current steps use b70-{num_devices}. We plan to introduce additional templates, and will add the override back when those are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants