Repository navigation
[CI] Add Intel GPU Kubernetes support to the pipeline generator - #857
zxd1997066 wants to merge 2 commits into
Conversation
|
Hi @khluu , Could you please take a look at this? We are preparing to migrate Intel CI to Kubernetes. |
dd4e96a to
c2a5204
Compare
Thangnguyenvn98
left a comment
There was a problem hiding this comment.
Thanks for this, and for validating it in vllm-project/vllm#63311. A few things before we merge, mostly so the generator stays small for the people maintaining it:
- Workspace path (biggest one). The XPU image keeps the source at
/workspace/vllm, and agent-stack-k8s mounts its own/workspacevolume over it, which is why the pod needs thecopy-vllm-srcinit container. The generator's automatic sharding also expects the source at/vllm-workspace(runtime_shard.CONTAINER_WORKSPACE), so with the current layout these steps can never be auto-sharded. If the XPU test image put the source at/vllm-workspacelikedocker/Dockerfiledoes, the init container, theworkingDirandPYTHONPATHoverrides, and theworking_dirline on every Intel step could all go, and the steps could useautomatic_shard. Would that work on your side? - Tests. Please add a test for an
intel_gpu_k8sstep, followingbuildkite/tests/pipeline_generator/test_l4_k8s.py: queue, claim template name, image tag, CPU/memory/shm for 1 and 4 GPUs, no NVIDIA setup or recorder commands, and the timeout not raised by the recording factor. resource_claim_template. Every step in #63311 sets it to the defaultb70-{num_devices}, so the field isn't needed yet. Could we drop it and add it when a step needs a different template?
Smaller suggestions are inline.
| ], | ||
| "initContainers": [ | ||
| { | ||
| "name": "copy-vllm-src", |
There was a problem hiding this comment.
This mounts a volume named workspace that the template doesn't define, so it depends on agent-stack-k8s injecting it. If the init container stays (see point 1 in the summary), a one-line comment saying so would help the next reader.
There was a problem hiding this comment.
Removed the init container as part of the workspace change
| step.resource_claim_template or f"b70-{num_gpus}" | ||
| ) | ||
| for container in pod_spec["containers"] + pod_spec["initContainers"]: | ||
| container["image"] = f"{image}-xpu" |
There was a problem hiding this comment.
The other image suffixes (-cpu, -arm64, -torch-nightly) are added in utils_lib/docker_utils.get_image. Could -xpu live there too (get_image(xpu=True))? Then the tag logic stays in one place; as it is, a torch-nightly config would ask for <commit>-torch-nightly-xpu, which image_build_xpu.sh never builds.
There was a problem hiding this comment.
Moved the -xpu suffix handling into get_image(xpu=True). It now uses the -xpu tag without the -torch-nightly suffix, matching the XPU image build.
| plugin = copy.deepcopy(intel_gpu_k8s_plugin_template) | ||
| pod_spec = plugin["kubernetes"]["podSpec"] | ||
| pod_spec["resourceClaims"][0]["resourceClaimTemplateName"] = ( | ||
| step.resource_claim_template or f"b70-{num_gpus}" |
There was a problem hiding this comment.
See point 3: with the field gone this becomes f"b70-{num_gpus}".
| device: Optional[str] = None | ||
| agent_tags: Optional[Dict[str, str]] = None | ||
| num_devices: Optional[int] = None | ||
| resource_claim_template: Optional[str] = Field(default=None, min_length=1) |
| enabled = kernrec_enabled() | ||
| return ( | ||
| enabled | ||
| and step.device != DeviceType.INTEL_GPU_K8S |
There was a problem hiding this comment.
These checks are needed because _recording_timeout is called with the default "nvidia" profile, not because the recorder predicates are wrong. A small helper that returns the setup profile for a step ("amd", "none" for intel_gpu_k8s, else "nvidia"), used both here at :1246 and for the timeout, would let both device checks go. It also avoids a subtle case: an AMD mirror keeps its parent's device, so a check on step.device would also turn off ROCm recording for an Intel step's AMD mirror.
There was a problem hiding this comment.
Added a setup profile helper and used it for command setup and timeout calculation. Intel K8s uses none, while AMD mirrors explicitly use amd regardless of the parent step’s device. The device checks in the recorder predicates have been removed.
| # command step | ||
| step_commands = _prepare_commands(step, variables_to_inject) | ||
| setup_profile: SetupProfile = ( | ||
| "none" if step.device == DeviceType.INTEL_GPU_K8S else "nvidia" |
There was a problem hiding this comment.
See the comment at :494.
| """ | ||
| return ( | ||
| fnrec_enabled() | ||
| and step.device != DeviceType.INTEL_GPU_K8S |
There was a problem hiding this comment.
Same as the comment at :494.
ce5f2a0 to
e3b76ba
Compare
Signed-off-by: zengxian <xiangdong.zeng@intel.com>
e3b76ba to
793a99d
Compare
Thanks for the review!
|
Summary
Add
intel-gpu-k8ssupport to the pipeline generator to enable the Intel GPU CI migration to Kubernetes.Changes
intel_gpu_k8sand route jobs to theintel-gpu-k8squeue.resource_claim_template, defaulting tob70-{num_devices}.Validation
Verified in vllm-project/vllm#63311, Results match the
intel-gputest run.