百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
S

SuperFusion

> 编程语言
开源

[ICRA 2024] SuperFusion: 多级别激光雷达-摄像头融合技术,用于远程高清地图生成

392 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

[ICRA 2024] SuperFusion: 多级别激光雷达-摄像头融合技术,用于远程高清地图生成

SuperFusion

This repository contains the implementation of the ICRA 2024 paper:

SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation

Hao Dong, Xianjing Zhang, Jintao Xu, Rui Ai, Weihao Gu, Huimin Lu, Juho Kannala and Xieyuanli Chen

Link to the arXiv version of the paper is available.

Update: We have a new survey paper on Multimodal Adaptation and Generalization

Pipeline overview of SuperFusion. Our method fuses camera and LiDAR data in three levels: the data-level fusion fuses depth information from LiDAR to improve the accuracy of image depth estimation, the feature-level fusion uses cross-attention for long-range LiDAR BEV feature prediction with the guidance of image features, and the BEV-level fusion aligns two branches to generate high-quality fused BEV features. Finally, the fused BEV features can support different heads, including semantic segmentation, instance embedding, and direction prediction, and finally post-processed to generate the HD map prediction.

Abstract

High-definition (HD) semantic map generation of the environment is an essential component of autonomous driving. Existing methods have achieved good performance in this task by fusing different sensor modalities, such as LiDAR and camera. However, current works are based on raw data or network feature-level fusion and only consider short-range HD map generation, limiting their deployment to realistic autonomous driving applications. In this paper, we focus on the task of building the HD maps in both short ranges, i.e., within 30 m, and also predicting long-range HD maps up to 90 m, which is required by downstream path planning and control tasks to improve the smoothness and safety of autonomous driving. To this end, we propose a novel network named SuperFusion, exploiting the fusion of LiDAR and camera data at multiple levels. We use LiDAR depth to improve image depth estimation and use image features to guide long-range LiDAR feature prediction. We benchmark our SuperFusion on the nuScenes dataset and a self-recorded dataset and show that it outperforms the state-of-the-art baseline methods with large margins on all intervals. Additionally, we apply the generated HD map to a downstream path planning task, demonstrating that the long-range HD maps predicted by our method can lead to better path planning for autonomous vehicles.

Code

Prepare

  1. Download nuScenes dataset (Full dataset and Map expansion) and unzip files. The folder should be like

  2. Install dependencies by running

bash
pip install -r requirement.txt
  1. Download the pretrained DeepLabV3 model and place within the checkpoints directory

  2. Download the pretrained SuperFusion model and place within the runs directory

Training

bash
python train.py --instance_seg --direction_pred --depth_sup --dataroot /path/to/nuScenes/ --pretrained --add_depth_channel

Evaluation

  1. Evaluate IoU
bash
python evaluate_iou_split.py --dataroot /path/to/nuScenes/ --modelf runs/model.pt --instance_seg --direction_pred --depth_sup --add_depth_channel --pretrained
  1. Evaluate CD and AP
bash
python export_pred_to_json.py --dataroot /path/to/nuScenes/ --modelf runs/model.pt --depth_sup --add_depth_channel --pretrained
bash
python evaluate_json_split.py --result_path output.json --dataroot /path/to/nuScenes/

Visualization

bash
python vis_prediction_gt.py --instance_seg --direction_pred --dataroot /path/to/nuScenes/
bash
python vis_prediction.py --modelf runs/model.pt --instance_seg --direction_pred --depth_sup --pretrained --add_depth_channel --version v1.0-trainval --dataroot /path/to/nuScenes/

Long-range HD map generation on nuScenes dataset

Instance detection results on nuScenes dataset.

More examples of self-recorded dataset

Long-range HD map generation on self-recorded dataset

Instance detection results on self-recorded dataset.

Citation

If you use our implementation in your academic work, please cite the corresponding paper:

@article{dong2022SuperFusion,
	author   = {Hao Dong and Xianjing Zhang and Jintao Xu and Rui Ai and Weihao Gu and Huimin Lu and Juho Kannala and Xieyuanli Chen},
	title    = {{SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation}},
	journal  = {arXiv preprint arXiv:2211.15656},
	year     = {2022},
}

Related Projects

SimMMDG: SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization

MOOSA: Towards Multimodal Open-Set Domain Generalization and Adaptation through Self-supervision

AEO: Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization

Survey: Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models

MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

Acknowledgement

Many thanks to the excellent open-source projects HDMapNet,LSS,AlignSeg and CaDDN.

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言