[Feature] Support FocalFormer3D on nuScenes - #3148
Open
Yerdana2002 wants to merge 1 commit into
Open
Yerdana2002 wants to merge 1 commit into
Yerdana2002 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
Motivation
FocalFormer3D (ICCV 2023) targets false negatives in 3D detection, which are
the failure mode that matters most for autonomous driving. It uses a
multi-stage heatmap encoder that masks out whatever earlier stages already
found, so each stage is pushed toward the objects the previous ones missed,
then rescores the accumulated queries with a deformable transformer decoder to
control the false positives that harvesting hard instances introduces.
There is no mmdet3d 1.x implementation. The official release
(NVlabs/FocalFormer3D) targets mmdet3d 0.x and does not run on the current
codebase. This PR ports it to
projects/for nuScenes.Modification
Adds
projects/FocalFormer3D/:FocalFormer3Ddetector,FocalEncoderneck andFocalDecoderhead,ported from the official release to the mmengine APIs (
loss/predict,MODELS.build,mmdet3d.registry,mmdet3d.structures).TransFusionBBoxCoder,HungarianAssigner3D,HeuristicAssigner3Dand theFadinghook, which dropsObjectSampleon the schedule upstream uses.FFLoadMultiViewImageandFFImageAug3Dfor the LiDAR-camera pipeline,adapted from
projects/BEVFusionand registered under names nothing elseuses, so both projects can coexist.
projects/BEVFusion:a LiDAR model, a LiDAR-camera model, and DeformFormer3D, which is the
single-stage variant the LiDAR model initialises from.
tools/convert_focalformer_ckpt.py, which remaps the released checkpointsfrom the mmcv 0.x decoder key layout to mmdet 3.x.
The released checkpoints were trained under the mmdet3d 0.x nuScenes box
convention, which stores dimensions as (w, l, h) and the yaw offset by
-yaw - pi/2relative to what 1.x writes. Rather than retrain the outputlayers, the detector converts at the boundary: ground truth is converted on the
way into
lossand predictions are converted back inadd_pred_to_datasample. The yaw map is its own inverse, so both directionsuse the same expression and the round trip is the identity.
BC-breaking
No. Everything is new and lives under
projects/. No existing file ismodified.
Use cases
nuScenes 3D detection, LiDAR-only and LiDAR-camera.
Verified on the nuScenes validation split with the released LiDAR checkpoint
after conversion: 70.4 NDS and 65.8 mAP, against 70.9 and 66.4 reported
upstream. The remaining gap comes from differences between the 0.x and 1.x
data pipelines rather than from the model.
Waymo support will follow in a separate PR.
Checklist
a
projects/contribution and follows the same testing scope as theexisting projects there.
included at
projects/FocalFormer3D/README.md.