Fast-FoundationStereo

Fast-FoundationStereo: Real-Time Zero-Shot Stereo Matching

This is the official implementation of our paper accepted to CVPR 2026

[Website] [Paper] [Video]

Authors: Bowen Wen, Shaurya Dewan, Stan Birchfield

Abstract

Stereo foundation models achieve strong zero-shot generalization but remain computationally prohibitive for real-time applications. Efficient stereo architectures, on the other hand, sacrifice robustness for speed and require costly per-domain fine-tuning. To bridge this gap, we present Fast-FoundationStereo, a family of architectures that achieve, for the first time, strong zero-shot generalization at real-time frame rate. We employ a divide-and-conquer acceleration strategy with three components: (1) knowledge distillation to compress the hybrid backbone into a single efficient student; (2) blockwise neural architecture search for automatically discovering optimal cost filtering designs under latency budgets, reducing search complexity exponentially; and (3) structured pruning for eliminating redundancy in the iterative refinement module. Furthermore, we introduce an automatic pseudo-labeling pipeline used to curate 1.4M in-the-wild stereo pairs to supplement synthetic training data and facilitate knowledge distillation. The resulting model can run over 10× faster than FoundationStereo while closely matching its zero-shot accuracy, thus establishing a new state-of-the-art among real-time methods.

[NOTE] This model is designed for real-time applications. For offline computation for the best accuracy, please checkout our earlier work FoundationStereo.

Environment setup

• Option 1: Docker

docker build --network host -t ffs -f docker/dockerfile .
bash docker/run_container.sh

• Option 2: pip

conda create -n ffs python=3.12 && conda activate ffs
pip install torch==2.6.0 torchvision==0.21.0 xformers --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt

Weights and Trade-off

Commercial use: Download C-Fast-FoundationStereo under the NVIDIA Open Model Agreement. For commercially licensed training and inference code, see NVIDIA TAO Toolkit.

Research checkpoints: Download from Google Drive and put under the folder weights/ (e.g. ./weights/23-36-37). Below table compares the differences among some representative models of varying sizes from our trained family. They are sorted from slowest to fastest, with accuracy descending, where runtime is profiled on GPU 3090, image size 640x480.

To trade-off speed and accuracy, there are two options:

1. Try with different checkpoints.
2. Tune the config flags (see explanations in the "Run demo" section below).

Run demo

python scripts/run_demo.py --model_dir weights/23-36-37/model_best_bp2_serialize.pth --left_file demo_data/left.png --right_file demo_data/right.png --intrinsic_file demo_data/K.txt --out_dir output/ --remove_invisible 0 --denoise_cloud 1  --scale 1 --get_pc 1 --valid_iters 8 --max_disp 192 --zfar 100

Refer to scripts/run_demo.py for comprehensive list of flags.

Tips:

• The input left and right images should be rectified and undistorted, which means there should not be fisheye kind of lens distortion and the epipolar lines are horizontal between the left/right images. If you obtain images from stereo cameras such as Zed, they usually have handled this for you.
• Do not swap left and right image. The left image should really be obtained from the left-side camera (objects will appear righter in the image).
• We recommend to use PNG files with no lossy compression
• Our method works best on stereo RGB images. However, we have also tested it on monochrome or IR stereo images (e.g. from RealSense D4XX series) and it works well too.
• To get point cloud for your own data, you need to specify the intrinsics. In the intrinsic file in args, 1st line is the flattened 1x9 intrinsic matrix, 2nd line is the baseline (distance) between the left and right camera, unit in meters.
• The model performs better for image width size <1000. You can run with smaller scale, e.g. --scale 0.5 to downsize input image, then upsize the output depth to your need with nearest neighbor interpolation.
• For faster inference, you can reduce the input image resolution by e.g. --scale 0.5, and reduce refine iterations by e.g. --valid_iters 4.
• Note that the 1st time running is slower due to compilation, use a while loop after warm up for live running.

Expect to see results like below:

• Disparity/Depth:

• Point cloud:

ONNX/TRT

For TRT, we recommend first setup env in docker.

Single ONNX

Export the full model as a single ONNX file. This replaces the Triton GWC kernel with ONNX-compatible ops so no intermediate engine split is needed.

python scripts/make_single_onnx.py --model_dir weights/23-36-37/model_best_bp2_serialize.pth --save_path output/ --height 480 --width 640 --valid_iters 8 --max_disp 192

Then convert to a single TRT engine:

trtexec --onnx=output/fast_foundationstereo.onnx --saveEngine=output/fast_foundationstereo.engine --fp16

To run inference with the single ONNX or TRT engine:

python scripts/run_demo_single_trt.py --model_dir output/ --left_file demo_data/left.png --right_file demo_data/right.png --intrinsic_file demo_data/K.txt --out_dir output_demo/ --get_pc 1 --remove_invisible 0 --denoise_cloud 1 --zfar 100

The script auto-detects .engine or .onnx files in --model_dir. To use a specific file, pass --model_file directly.

Note: The single ONNX model expects pre-normalized float32 inputs (ImageNet normalization stripped). The inference script handles this automatically. If integrating into your own pipeline, apply normalization beforehand:

normalized = (pixel - mean) / std
mean = [123.675, 116.28, 103.53]
std  = [ 58.395, 57.12, 57.375]

Two-stage ONNX

The original export splits the model into two ONNX files around the Triton GWC kernel, which runs as an intermediate step between the two TRT engines.

python scripts/make_onnx.py --model_dir weights/23-36-37/model_best_bp2_serialize.pth --save_path output/ --height 448 --width 640 --valid_iters 8 --max_disp 192