Deep Patch Visual Odometry/SLAM
This repository contains the source code for our papers:
Zachary Teed*, Lahav Lipson*, Jia Deng
Lahav Lipson, Zachary Teed, Jia Deng
@article{teed2023deep,
title={Deep Patch Visual Odometry},
author={Teed, Zachary and Lipson, Lahav and Deng, Jia},
journal={Advances in Neural Information Processing Systems},
year={2023}
}@inproceedings{lipson2024deep,
author={Lipson, Lahav and Teed, Zachary and Deng, Jia},
title={{Deep Patch Visual SLAM}},
booktitle={European Conference on Computer Vision},
year={2024}
}The code was tested on Ubuntu 20/22 and Cuda 11/12.
Clone the repo
git clone https://github.com/princeton-vl/DPVO.git --recursive
cd DPVOCreate and activate the dpvo anaconda environment
conda env create -f environment.yml
conda activate dpvoNext install the DPVO package
wget https://gitlab.com/libeigen/eigen/-/archive/3.4.0/eigen-3.4.0.zip
unzip eigen-3.4.0.zip -d thirdparty
# install DPVO
pip install .
# download models and data (~2GB)
./download_models_and_data.shNote: You will need to have CUDA 11 and CuDNN installed on your system.
./Pangolin/scripts/install_prerequisites.sh recommended
mkdir Pangolin/build && cd Pangolin/build
cmake ..
make -j8
sudo make install
cd ../..pip install ./DPViewerFor installation issues, our Docker Image supports the visualizer.
We provide a classical backend for closing very large loops, which requires extra installation.
Step 1. Install the OpenCV C++ API. On Ubuntu, you can use
sudo apt-get install -y libopencv-devStep 2. Install DBoW2
cd DBoW2
mkdir -p build && cd build
cmake .. # tested with cmake 3.22.1 and gcc/cc 11.4.0 on Ubuntu
make # tested with GNU Make 4.3
sudo make install
cd ../..Step 3. Install the image retrieval
pip install ./DPRetrievalDPVO can be run on any video or image directory with a single command. Note you will need to have installed DPViewer to visualize the reconstructions in real-time. You can also save the completed reconstructions and view them in COLMAP. The pretrained models can be downloaded from google drive models.zip if you have not already run the download script.
python demo.py \
--imagedir= \
--calib= \
--viz # enable visualization
--plot # save trajectory plot
--save_ply # save point cloud as a .ply file
--save_trajectory # save the predicted trajectory as .txt in TUM format
--save_colmap # save point cloud + trajectory in the standard COLMAP text formatpython demo.py --imagedir=movies/IMG_0492.MOV --calib=calib/iphone.txt --stride=5 --plot --vizDownload a sequence from TartanAir (several samples are availabe from download directly from the webpage)
python demo.py --imagedir= --calib=calib/tartan.txt --stride=1 --plot --vizDownload a sequence from EuRoC (download ASL format)
python demo.py --imagedir= --calib=calib/euroc.txt --stride=2 --plot --vizTo run DPVO with a SLAM backend (i.e., DPV-SLAM), add
--opts LOOP_CLOSURE Trueto any evaluate_X.py script or to demo.py
If installed, the classical backend can also be enabled using
--opts CLASSIC_LOOP_CLOSURE TrueWe provide evaluation scripts for TartanAir, EuRoC, TUM-RGBD and ICL-NUIM. Up to date result logs on these datasets can be found in the logs directory.
Results on the validation split and test set can be obtained with the command:
python evaluate_tartan.py --trials=5 --split=validation --plot --save_trajectorypython evaluate_euroc.py --trials=5 --plot --save_trajectorypython evaluate_tum.py --trials=5 --plot --save_trajectorypython evaluate_icl_nuim.py --trials=5 --plot --save_trajectorypython evaluate_kitti.py --trials=5 --plot --save_trajectoryMake sure you have run ./download_models_and_data.sh. Your directory structure should look as follows
├── datasets
├── TartanAir.pickle
├── TartanAir
├── abandonedfactory
├── abandonedfactory_night
├── ...
├── westerndesert
...To train (log files will be written to runs/). Model will be run on the validation split every 10k iterations
python train.py --steps=240000 --lr=0.00008 --name=No open issues yet, or sync has not completed.