How to generate ground truth lable for occupancy prediction?

[haibao-yu]

Thanks for you great work!

As demonstrated in this paper, the occupancy model's ultimate output is a binary segmentation that indicates whether each BEV grid is free or occupied. The original nuScenes dataset does not provide an occupancy label. I am interested to know if there is any documentation available that explains the process of generating the ground truth label for occupancy prediction.

[YTEP-ZHI]

Hi, @haibao-yu. Please refer to this file for occupancy ground-truth generation: https://github.com/OpenDriveLab/UniAD/blob/main/projects/mmdet3d_plugin/datasets/pipelines/occflow_label.py.
Briefly speaking, we draw the annotated 3d boxes of vehicles on a bev plane while ignoring the height of boxes. Pixels within boxes are filled with a "1" to indicate occupied regions, otherwise filled with "0" to indicate empty space.

As each ground-truth is a sequence of multi-frame future occupancy, the coordinates of all frames in the sequence should be transformed to align with the first frame.

[haibao-yu]

Thank you for your response. Based on the information provided, it seems that there is no direct connection between the usage of the Occupancy labels in UniAD and the Occupancy challenge described at https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction. Additionally, I am interested to know if BEV Occupancy is considered a sub-task of BEV detection in UniAD, considering that the occupancy outputs focuses solely on vehicle boxes.

[YTEP-ZHI]

@haibao-yu Yes, the 3d occupancy described in https://github.com/CVPR2023-3D-Occupancy-Prediction/CVPR2023-3D-Occupancy-Prediction has major differences with BEV occupancy in UniAD.

• The 3d occupancy challenge considers both dynamic agents and static elements in the scene, as long as there're lidar points on these objects. However, in BEV occupancy, only vehicles are emphasized without the awareness of static objects (trees, buildings).
• UniAD predicts multi-step future BEV occupancies while 3d occupancy challenge only requires predicting the result of the curent timestep.

BEV occupancy also varies from detection:

• Output format: BEV occupancy is similar to 2d segmentation (of a BEV plane), while detections are collections of sparse attributes, indicating both spatial and dynamic information.
• Timestep: BEV occupancy -- Multi-step in the future, Detection -- Single-step of current state
  Due to these differences, we do not consider BEV occupancy prediction as a sub-task of detection, and the model designs of these two tasks are different.

[haibao-yu]

Thank you for your quick response.

Regarding the second concern about the relationship between BEV occupancy and BEV detection, I would like to explore the inclusion of motion forecasting information. Specifically, when we combine the detection frame, how does the predicted future trajectory of the dynamic target differ from the concept of occupancy in this context?

The reason I am interested in this question is that, in my perspective, the occupancy task seems to complement the 3D detection and related prediction tasks, such as abnormal object detection and corner case detection. However, it appears that in UniAD, BEV occupancy can be directly obtained through detection and motion forecasting.

[YTEP-ZHI]

the occupancy task seems to complement the 3D detection and related prediction tasks, such as abnormal object detection and corner case detection.

@haibao-yu Yes, this is also one of our motivations for introducing occupancy in UniAD. Additionally, the occupancy is per-grid supervised (supervise on the whole BEV plane), much denser than the training of detection (supervise on some sparse vectors in low dimension). Therefore, we think the resulting per-grid probability of occupancy is more reliable than that of detection results. For example, in detection results, a small digital error in the position/heading angle of an output vector would lead to significant deviations from the real-world state.

However, it appears that in UniAD, BEV occupancy can be directly obtained through detection and motion forecasting.

Yes, you can actually extract the future occupancy via some post-progresses on the detection and motion results. However, both detection and motion are trained with sparse supervision on low-dimension vectors, where a small digital error would lead to significant deviations in real world as I mentioned before. By combining their predictions together to obtain occupancy, their errors will be further accumulated $E = E_{track} \cdot E_{motion}$, making the results more unreliable.

That's why we directly learn occupancy with per-grid dense supervision.

[haibao-yu]

the occupancy task seems to complement the 3D detection and related prediction tasks, such as abnormal object detection and corner case detection.

@haibao-yu Yes, this is also one of our motivations for introducing occupancy in UniAD. Additionally, the occupancy is per-grid supervised (supervise on the whole BEV plane), much denser than the training of detection (supervise on some sparse vectors in low dimension). Therefore, we think the resulting per-grid probability of occupancy is more reliable than that of detection results. For example, in detection results, a small digital error in the position/heading angle of an output vector would lead to significant deviations from the real-world state.

However, it appears that in UniAD, BEV occupancy can be directly obtained through detection and motion forecasting.

Yes, you can actually extract the future occupancy via some post-progresses on the detection and motion results. However, both detection and motion are trained with sparse supervision on low-dimension vectors, where a small digital error would lead to significant deviations in real world as I mentioned before. By combining their predictions together to obtain occupancy, their errors will be further accumulated E=Etrack⋅Emotion, making the results more unreliable.

That's why we directly learn occupancy with per-grid dense supervision.

Very thanks for your response.

[MPCheng-ZW]

Thanks for your great job!

What are the connections and differences between occupancy prediction and occupancy flow prediction? Which one is employed in your paper?

Additionally, could you recommend some papers on BEV occupancy prediction?

Thanks again! 谢谢~