Skip to content

ze: resolve command-queue ordinal to its engine type (fixes #506) - #526

Merged
TApplencourt merged 3 commits into
argonne-lcf:develfrom
DonAurelio:fix-506-command-queue-ordinal-type
Aug 5, 2026
Merged

ze: resolve command-queue ordinal to its engine type (fixes #506)#526
TApplencourt merged 3 commits into
argonne-lcf:develfrom
DonAurelio:fix-506-command-queue-ordinal-type

Conversation

@DonAurelio

@DonAurelio DonAurelio commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Problem

Level Zero device operations are tagged with the command queue they ran on as {ordinal, index}. The ordinal alone does not say whether the queue is a compute engine or a copy engine — that mapping lives in the device's command-queue-group properties (zeDeviceGetCommandQueueGroupProperties), which THAPI did not record. So a trace could show:

zeCommandListAppendMemoryCopy(H2D)[{ordinal: 1, index: 0}]

with no way to tell it ran on the copy engine. This is #506.

Fix

  1. Record a new curated event, lttng_ust_ze_properties:command_queue_group, carrying the per-device group-properties array (queried in the existing per-device property dump, reusing the already-dlsym'd entry point).
  2. Resolve it in the ze interval analysis: join each op's ordinal to that array and append the engine capability flags.

The group flags field is a bitmask (COMPUTE | COPY | COOPERATIVE_KERNELS | METRICS), and several bits are typically set at once (e.g. ordinal 0 = 0x7), so every set bit is decoded into a list rather than collapsed to one type; unrecognised bits are reported as UNKNOWN.

After the fix:

zeCommandListAppendMemoryCopy(H2D)[{ordinal: 1, index: 0, flags: [COPY]}]
zeCommandListAppendMemoryCopy(H2D)[{ordinal: 0, index: 0, flags: [COMPUTE, COPY, COOPERATIVE_KERNELS]}]

Validation

Recorded two raw traces (compute ordinal 0, copy ordinal 1) on an Intel Data Center GPU Max 1550 (Aurora), then replayed the same trace files through an unmodified baseline build (before) and the patched build (after). Only the analyzer differs, so every difference is attributable to this change:

trace before after
ordinal 0 {ordinal: 0, index: 0} {ordinal: 0, index: 0, flags: [COMPUTE, COPY, COOPERATIVE_KERNELS]}
ordinal 1 {ordinal: 1, index: 0} {ordinal: 1, index: 0, flags: [COPY]}

Timings and byte counts are byte-identical before vs after (2.96us / 5.44us, 4.10kB) — the change adds a label without perturbing measurement.

Changes

5 edits, all in backends/ze/:

  • ze_events.yaml — declare the new event
  • tracer_ze_helpers.include.c — emit it
  • btx_zeinterval_callbacks.hpp — analysis-state map (device → ordinal-indexed group vector)
  • btx_zeinterval_callbacks.cpp — populate/register callback + decode flags at the emit site

Fixes #506

🤖 Generated with Claude Code

@DonAurelio
DonAurelio force-pushed the fix-506-command-queue-ordinal-type branch from 8f46a52 to 58d2935 Compare August 4, 2026 18:58
…cf#506)

Level Zero device operations are labelled with the command queue they ran
on as {ordinal, index}. The ordinal alone does not indicate whether the
queue is a compute or a copy engine; that mapping lives in the device's
command-queue-group properties, which THAPI did not record.

Record a new curated event, lttng_ust_ze_properties:command_queue_group,
carrying the per-device group-properties array, and in the ze interval
analysis join each op's ordinal to that array. The group's capability
flags field is a bitmask (COMPUTE|COPY|COOPERATIVE_KERNELS|METRICS, often
several bits set at once), so decode every set bit into a list rather than
collapse to a single type; unrecognised bits are reported as UNKNOWN.

Device ops now print e.g.
  zeCommandListAppendMemoryCopy(H2D)[{ordinal: 1, index: 0, flags: [COPY]}]
  zeCommandListAppendMemoryCopy(H2D)[{ordinal: 0, index: 0, flags: [COMPUTE, COPY, COOPERATIVE_KERNELS]}]

A missing table or out-of-range ordinal appends no flags, preserving the
pre-fix output. Timings and byte counts are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@DonAurelio
DonAurelio force-pushed the fix-506-command-queue-ordinal-type branch from 58d2935 to 5f69443 Compare August 4, 2026 19:31
The lint CI runs yamlfmt (go install ...@latest) on any changed YAML file.
Current yamlfmt strips intra-bracket spaces ([ x ] -> [x]) and the leading
document marker, but ze_events.yaml predates that default and carried 137
such violations across the whole file. Since the yamlfmt job only runs when
a PR touches a .yaml file, the debt stayed latent until this change.

This commit is a pure reformat (whitespace only, no semantic change,
verified by comparing space-stripped content) so the yamlfmt check passes.
Kept separate from the functional fix so it can be reverted independently.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@TApplencourt

Copy link
Copy Markdown
Collaborator

Can you rebase? The CI should pass now.

@DonAurelio

Copy link
Copy Markdown
Collaborator Author

It seems the CI is failing now becasue I am adding a new trace point.

@TApplencourt

Copy link
Copy Markdown
Collaborator

Yep, no problem. Will review and merge!

@TApplencourt

Copy link
Copy Markdown
Collaborator

I guess at some point we may want to refactor to have always some dump_device_group_property this will allow to have cleaner early exit aka

if != ZE_SUCESS
     return
```
But good enough for now.

Thanks a lot!     

@TApplencourt
TApplencourt merged commit 27ff14a into argonne-lcf:devel Aug 5, 2026
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missing the Type of Compute Ordinal

2 participants