questions about command generation during inference time

[qiangt]

Hi, authors, I have a question regarding how the command is generated, i.e., turn-left, turn-right, go-forward. I read the related function get_sdc_planning_label in projects/mmdet3d_plugin/datasets/data_utils/trajectory_api.py

My understanding is that: to decide whether the command should be turn left or right, it compares the horizontal distance shift (x) between the first and last checkpoint (6 in total by default). My issue is that even during the inference time, the code is using the ground truth trajectories from future 6 frames. However, if we think about the real situation, we cannot read the ground truth trajectories from future since the car hasn't drive to the future yet. This implementation makes the UniAD only works for dataset has labels and cannot be directly tested in unlabelled dataset, even at inference time.

I have two questions:

1. Is my understanding correct? I might misunderstand the codes
2. What will be your thoughts on how to generate this command in the real scenario where no label is available? I know we might be able to get the command from navigations but navigation is normally much coarser and considered to be global planning. 2 seconds time is normally falling into the local planning range.

[YTEP-ZHI]

@qiangt Good question. High-level command is used to help self-car make decisions among multi-modal future choices (e.g., you can either trun left or keep straight forward in a crossroad). It's similar to GPS guidance in real-world scenarios, where the GPS system gives a high-level guidance towards the destination and the drive/self-car still need to make low-level(detailed) controls by themselves to follow that command/guidance. Moreover, in closed-loop simulation (e.g., carla), the target point in a near future will be given to the self-car to plan towards that direction at each timestep, which works similarly with the high-level command.

We've also found that, once your model is well-trained with commands, it learns the correspondence between command and planned trajectories. During inference, given the same scenario and different commands, the model will predict reasonable yet completely different trajectories related to each command.

[qiangt]

@YTEP-ZHI Thanks for your explanation. I'll close the issue.