Skip to content

[Feature] Support FocalFormer3D on nuScenes - #3148

Open
Yerdana2002 wants to merge 1 commit into
open-mmlab:dev-1.xfrom
Yerdana2002:focalformer3d
Open

Yerdana2002 wants to merge 1 commit into
open-mmlab:dev-1.xfrom
Yerdana2002:focalformer3d

Conversation

@Yerdana2002

Copy link
Copy Markdown

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

FocalFormer3D (ICCV 2023) targets false negatives in 3D detection, which are
the failure mode that matters most for autonomous driving. It uses a
multi-stage heatmap encoder that masks out whatever earlier stages already
found, so each stage is pushed toward the objects the previous ones missed,
then rescores the accumulated queries with a deformable transformer decoder to
control the false positives that harvesting hard instances introduces.

There is no mmdet3d 1.x implementation. The official release
(NVlabs/FocalFormer3D) targets mmdet3d 0.x and does not run on the current
codebase. This PR ports it to projects/ for nuScenes.

Modification

Adds projects/FocalFormer3D/:

  • FocalFormer3D detector, FocalEncoder neck and FocalDecoder head,
    ported from the official release to the mmengine APIs (loss / predict,
    MODELS.build, mmdet3d.registry, mmdet3d.structures).
  • TransFusionBBoxCoder, HungarianAssigner3D, HeuristicAssigner3D and the
    Fading hook, which drops ObjectSample on the schedule upstream uses.
  • FFLoadMultiViewImage and FFImageAug3D for the LiDAR-camera pipeline,
    adapted from projects/BEVFusion and registered under names nothing else
    uses, so both projects can coexist.
  • Three configs following the naming convention used by projects/BEVFusion:
    a LiDAR model, a LiDAR-camera model, and DeformFormer3D, which is the
    single-stage variant the LiDAR model initialises from.
  • tools/convert_focalformer_ckpt.py, which remaps the released checkpoints
    from the mmcv 0.x decoder key layout to mmdet 3.x.

The released checkpoints were trained under the mmdet3d 0.x nuScenes box
convention, which stores dimensions as (w, l, h) and the yaw offset by
-yaw - pi/2 relative to what 1.x writes. Rather than retrain the output
layers, the detector converts at the boundary: ground truth is converted on the
way into loss and predictions are converted back in
add_pred_to_datasample. The yaw map is its own inverse, so both directions
use the same expression and the round trip is the identity.

BC-breaking

No. Everything is new and lives under projects/. No existing file is
modified.

Use cases

nuScenes 3D detection, LiDAR-only and LiDAR-camera.

Verified on the nuScenes validation split with the released LiDAR checkpoint
after conversion: 70.4 NDS and 65.8 mAP, against 70.9 and 66.4 reported
upstream. The remaining gap comes from differences between the 0.x and 1.x
data pipelines rather than from the model.

Waymo support will follow in a separate PR.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. Not applicable, this is
    a projects/ contribution and follows the same testing scope as the
    existing projects there.
  3. If this PR introduces a new feature, docs have been added. A full README is
    included at projects/FocalFormer3D/README.md.
  4. The documentation has been modified accordingly.

@CLAassistant

CLAassistant commented Sep 6, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants