A feed-forward 3D foundation model for reconstructing scenes from streaming data
A feed-forward 3D foundation model for reconstructing scenes from streaming data
https://github.com/user-attachments/assets/fe39e095-af2c-4ec9-b68d-a8ba97e505ab
LingBot-Map has focused on:
demo.py)
demo_render/batch_demo.py)preprocess/oxford.py to prepare Oxford Spires data before evaluation.main and run python demo.py --compile ... or python gct_profile.py --backend flashinfer --dtype bf16 --compile to verify on your hardware.--keyframe_interval > 1 silently cached non-keyframes. You should now see better pose and reconstruction quality when running with more than 320 frames.1. Create conda environment
conda create -n lingbot-map python=3.10 -y
conda activate lingbot-map
2. Install PyTorch (CUDA 12.8)
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
PyTorch 2.8.0 is the recommended version because NVIDIA Kaolin (required by the batch rendering pipeline) has prebuilt wheels for
torch-2.8.0_cu128. If you only needdemo.pyyou may use a newer PyTorch, but the batch renderer then requires building Kaolin from source. For other CUDA versions, see PyTorch Get Started.
3. Install lingbot-map
pip install -e .
4. Install FlashInfer (recommended)
FlashInfer provides paged KV cache attention for efficient streaming inference. It is a pure-Python package that JIT-compiles CUDA kernels on first use, so a single wheel works across CUDA/PyTorch versions:
pip install --index-url https://pypi.org/simple flashinfer-python
--index-url https://pypi.org/simpleis only needed if your default pip index is an internal mirror that doesn't haveflashinfer-python. (Optional) For faster first-use, you can additionally install a CUDA-specific JIT cache:pip install flashinfer-jit-cache -f https://flashinfer.ai/whl/cu128/flashinfer-jit-cache/. See FlashInfer installation for details. If FlashInfer is not installed, the model falls back to SDPA (PyTorch native attention) via--use_sdpa.
5. Visualization dependencies (optional)
pip install -e ".[vis]"
🚧 Coming soon: we're training an stronger model that supports longer sequences — stay tuned.
After installation, run your first scene with one command:
python demo.py --model_path /path/to/lingbot-map.pt \
--image_folder example/courthouse --mask_sky
This launches an interactive viser viewer at http://localhost:8080. See Interactive Demo below for the full set of scenes and flags, or jump to Offline Rendering Pipeline for long-sequence batch rendering.
demo.py)Run demo.py for interactive 3D visualization via a browser-based viser viewer (default http://localhost:8080).
We provide three example scenes in example/ that you can run out of the box:
# courthouse scene
python demo.py --model_path /path/to/lingbot-map.pt \
--image_folder example/courthouse --mask_sky
https://github.com/user-attachments/assets/aa10f7ab-8024-43c7-92f8-d56159ec85c8
# University scene
python demo.py --model_path /path/to/lingbot-map.pt \
--image_folder example/university --mask_sky
https://github.com/user-attachments/assets/212a1744-6ff5-4ccf-9bd4-728608248b57
# Loop scene (loop closure trajectory)
python demo.py --model_path /path/to/lingbot-map.pt \
--image_folder example/loop
https://github.com/user-attachments/assets/5ae0a292-b081-40c6-838c-b7c1a0538d75
Sequence is too long for the interactive viser viewer — this clip was rendered with the Offline Rendering Pipeline. See that section for the full command.
We will provide more examples in the follow-up.
Dataset: Download the demo sequences from robbyant/lingbot-map-demo on Hugging Face.
Example run on the dynamic sequence from the dataset above (sky masking on, 4 camera optimization iterations, keyframe every 2 frames):
Run the dynamic sequence with sky masking, 4 camera optimization iterations, and an input stride of 2:
python demo.py \
--image_folder /path/to/dynamic\
--model_path ../../Lingbot-Map/lingbot-map.pt \
--camera_num_iterations 4 \
--mask_sky \
--stride 2
https://github.com/user-attachments/assets/567b6e9b-1cbf-402a-96be-9bab70715ec3
Use --keyframe_interval to reduce KV cache memory by only keeping every N-th frame as a keyframe. Non-keyframe frames still produce predictions but are not stored in the cache. This is useful for long sequences which exceed 320 frames (We train with video RoPE on 320 views, so performance degrades when the KV cache stores more than 320 views. Using a keyframe strategy allows inference over longer sequences.). In demo.py, the keyframe interval is calculated automatically.
Note on inference range. Our method does not perform state resetting by default, so the maximum inference range is bounded by the longest distance seen during training on the dataset. Beyond that distance, state resetting becomes necessary. If you observe pose collapse, switch to windowed mode (
--mode windowed) — in most cases tuning--keyframe_intervalalone is enough and the rest of the windowed parameters can stay at their defaults.
python demo.py --model_path /path/to/lingbot-map.pt \
--video_path video.mp4 --fps 10 \
--mode windowed --window_size 128 --overlap_keyframes 16 --keyframe_interval 2
Sky masking uses an ONNX sky segmentation model to filter out sky points from the reconstructed point cloud, which improves visualization quality for outdoor scenes.
Setup:
# Install onnxruntime (required)
pip install onnxruntime # CPU
# or
pip install onnxruntime-gpu # GPU (faster for large image sets)
By default, root demo.py resolves the sky segmentation model as skyseg.onnx relative to the current working directory. If that default file is missing, it is automatically downloaded from HuggingFace on first use. If the download fails or does not produce a regular file, sky masking stops with a RuntimeError that reports the model path, download URL, cause, and manual setup guidance; it never silently continues without masking.
For manual recovery while keeping the default path, download skyseg.onnx into the directory from which you run demo.py:
wget -O skyseg.onnx https://huggingface.co/JianyuanWang/skyseg/resolve/main/skyseg.onnx
python demo.py --model_path /path/to/checkpoint.pt \
--image_folder /path/to/images/ --mask_sky
Usage with an explicit model path:
To use a model stored elsewhere, pass its absolute path with --sky_model:
python demo.py --model_path /path/to/checkpoint.pt \
--image_folder /path/to/images/ --mask_sky \
--sky_model /absolute/path/to/skyseg.onnx
Sky masks are cached in <image_folder>_sky_masks/ so subsequent runs skip regeneration. You can also specify a custom cache directory with --sky_mask_dir, or save side-by-side mask visualizations with --sky_mask_visualization_dir:
python demo.py --model_path /path/to/checkpoint.pt \
--image_folder /path/to/images/ --mask_sky \
--sky_mask_dir /path/to/cached_masks/ \
--sky_mask_visualization_dir /path/to/mask_viz/
--port
8080
Viser viewer port
--conf_threshold
1.5
Visibility threshold for filtering low-confidence points
--point_size
0.00001
Point cloud point size
--downsample_factor
10
Spatial downsampling for point cloud display