dimention not match

[billbliss3]

It is an awesome work.

However, I encountered an error during training.

[rank0]: File "/Nexus/nuplan_extent/planning/training/modeling/models/nexus.py", line 331, in _generate_diffusion_forcing_times
[rank0]: diffusion_times = torch.where(~valid_mask, torch.full_like(diffusion_times, 1), diffusion_times)
[rank0]: RuntimeError: The size of tensor a (42) must match the size of tensor b (21) at non-singleton dimension 2

[zhouyunsong]

Thank you, would you mind sharing more details so we can try to reproduce the issue on our end? Also, if possible, could you help identify the potential cause? You're very welcome to submit a new merge request if you'd like.

[billbliss3]

您好，您的来件已收到，我会尽快给您回复的。 Thanks for you e-mail.~~ I've got it and I will reply it as soon as possible .

[Kguo-cs]

    scene_tensor_validity = np.concatenate([scene_tensor_validity, keep_mask.astype(scene_tensor_validity.dtype)], axis=-1) in waymo_nexus_feature_builder.py (line 269)

This lead to scene_tensor_validity with dimension 42, It is different from diffusion_times with dimension 21. Why is different?

[billbliss3]

您好，您的来件已收到，我会尽快给您回复的。 Thanks for you e-mail.~~ I've got it and I will reply it as soon as possible .

[1450081092]

Hello @zhouyunsong (and other contributors)!

Thanks again for your excellent work. I've been closely following this Issue #5, as it addresses the core problem I also encountered while studying and using the code.

Synthesizing the initial dimension mismatch error (42 vs 21) raised by billbliss3 and the key line of code (np.concatenate...) pointed out by Kguo-cs, our community (or rather, those of us studying this project) has developed a deeper hypothesis that we'd like to ask for your confirmation on.

Our understanding is as follows:

• The 42-dimensional validity tensor is indeed created by concatenating a 21-dim validity_mask and a 21-dim keep_mask during the data preprocessing stage.

• However, in the core training loop (forward_train), this pre-computed keep_mask (the latter 21 dimensions) appears to be ignored. Instead, the code dynamically and randomly generates new conditioning masks (task_mask and control_mask) to achieve a more flexible and diverse training process.

• The pre-processed static keep_mask seems to only be split and used in the specific evaluation function sim_agents_batched_rollout(nuplan_extent/planning/training/modeling/models/nexus.py，def sim_agents_batched_rollout(self, input_features)).

if 'test' in self._downstream_task:
            # all valid tokens in scene_tensor should be kept in test time
            # keep_mask = scene_tensor_features.validity.unsqueeze(-1).expand(-1, -1, -1, scene_tensor_features.tensor.shape[-1])
            valid_mask = scene_tensor_features.validity
            valid_mask,keep_mask = valid_mask.chunk(2,dim=-1)
            keep_mask = keep_mask.unsqueeze(-1).repeat(1,1,1,scene_tensor_features.tensor.shape[-1])
            # diffusion model only generates tokens where valid_mask is 1, 
            # in waymo, agents present at current timestep(5th) should be predicted in future timesteps(8s@2Hz)
            # valid_mask = scene_tensor_features.validity.clone()
            valid_mask[:, :, 5:] = (scene_tensor_features.validity[:, :, 4:5] == 1).expand(-1, -1, 16)

So, we'd like to ask if our understanding of this design choice is correct: is this situation the result of the project's evolution, where a more flexible, dynamic mask generation strategy is used for training, while the static keep_mask is retained for compatibility with specific evaluation tasks like the Sim Agents Challenge?

If this understanding is correct, it would be a great help to us learners and would clarify much of the confusion about the data flow.

We look forward to your reply. Thank you again for your time and your open-source contribution!

[Y-vic]

Thank you for the excellent and insightful analysis. You are entirely correct—this design was a pragmatic workaround on our part, and we appreciate you highlighting its lack of clarity.

To summarize the logic: the static keep_mask is ignored during training in favor of dynamic masks for better model robustness. It is concatenated solely to meet the deterministic evaluation requirements of the Waymo benchmark in a direct manner, avoiding a more complex data pipeline.

Your contribution here is very valuable for the community and for our future development. Thank you.

[1450081092]

That's wonderful, thank you for the detailed and transparent explanation! We're thrilled that our analysis was on the right track.

Understanding the design rationale, especially the need for a practical solution for evaluation, is incredibly helpful for everyone studying this project. Your engagement with the community is truly valuable.

Looking forward to the project's future developments. Thanks again!