MiDaS

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer

This repository contains code to compute depth from a single image. It accompanies our paper:

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, Vladlen Koltun

and our preprint:

Vision Transformers for Dense Prediction
René Ranftl, Alexey Bochkovskiy, Vladlen Koltun

For the latest release MiDaS 3.1, a technical report and video are available.

MiDaS was trained on up to 12 datasets (ReDWeb, DIML, Movies, MegaDepth, WSVD, TartanAir, HRWSI, ApolloScape, BlendedMVS, IRS, KITTI, NYU Depth V2) with multi-objective optimization. The original model that was trained on 5 datasets (MIX 5 in the paper) can be found here. The figure below shows an overview of the different MiDaS models; the bubble size scales with number of parameters.

Setup

1. Pick one or more models and download the corresponding weights to the weights folder:

MiDaS 3.1

• For highest quality: dpt_beit_large_512
• For moderately less quality, but better speed-performance trade-off: dpt_swin2_large_384
• For embedded devices: dpt_swin2_tiny_256, dpt_levit_224
• For inference on Intel CPUs, OpenVINO may be used for the small legacy model: openvino_midas_v21_small .xml, .bin

MiDaS 3.0: Legacy transformer models dpt_large_384 and dpt_hybrid_384

MiDaS 2.1: Legacy convolutional models midas_v21_384 and midas_v21_small_256

1. Set up dependencies:

   conda env create -f environment.yaml
   conda activate midas-py310
   

optional

For the Next-ViT model, execute

git submodule add https://github.com/isl-org/Next-ViT midas/external/next_vit

For the OpenVINO model, install

pip install openvino

Usage

1. Place one or more input images in the folder input.

2. Run the model with

   python run.py --model_type <model_type> --input_path input --output_path output
   

   where <model_type> is chosen from dpt_beit_large_512, dpt_beit_large_384, dpt_beit_base_384, dpt_swin2_large_384, dpt_swin2_base_384, dpt_swin2_tiny_256, dpt_swin_large_384, dpt_next_vit_large_384, dpt_levit_224, dpt_large_384, dpt_hybrid_384, midas_v21_384, midas_v21_small_256, openvino_midas_v21_small_256.

3. The resulting depth maps are written to the output folder.

optional

1. By default, the inference resizes the height of input images to the size of a model to fit into the encoder. This size is given by the numbers in the model names of the accuracy table. Some models do not only support a single inference height but a range of different heights. Feel free to explore different heights by appending the extra command line argument --height. Unsupported height values will throw an error. Note that using this argument may decrease the model accuracy.
2. By default, the inference keeps the aspect ratio of input images when feeding them into the encoder if this is supported by a model (all models except for Swin, Swin2, LeViT). In order to resize to a square resolution, disregarding the aspect ratio while preserving the height, use the command line argument --square.

via Camera

If you want the input images to be grabbed from the camera and shown in a window, leave the input and output paths away and choose a model type as shown above:

python run.py --model_type <model_type> --side

The argument --side is optional and causes both the input RGB image and the output depth map to be shown side-by-side for comparison.

via Docker

1. Make sure you have installed Docker and the NVIDIA Docker runtime.

2. Build the Docker image:

   docker build -t midas .
   

3. Run inference:

   docker run --rm --gpus all -v $PWD/input:/opt/MiDaS/input -v $PWD/output:/opt/MiDaS/output -v $PWD/weights:/opt/MiDaS/weights midas
   

   This command passes through all of your NVIDIA GPUs to the container, mounts the input and output directories and then runs the inference.

via PyTorch Hub

The pretrained model is also available on PyTorch Hub

via TensorFlow or ONNX

See README in the tf subdirectory.

Currently only supports MiDaS v2.1.

via Mobile (iOS / Android)

See README in the mobile subdirectory.

via ROS1 (Robot Operating System)

See README in the ros subdirectory.

Currently only supports MiDaS v2.1. DPT-based models to be added.

Accuracy

We provide a zero-shot error $\epsilon_d$ which is evaluated for 6 different datasets (see paper). Lower error values are better. $\color{green}{\textsf{Overall model quality is represented by the improvement}}$ (Imp.) with respect to MiDaS 3.0 DPTL-384. The models are grouped by the height used for inference, whereas the square training resolution is given by the numbers in the model names. The table also shows the number of parameters (in millions) and the frames per second for inference at the training resolution (for GPU RTX 3090):

MiDaS Model	DIW WHDR	Eth3d AbsRel	Sintel AbsRel	TUM δ1	KITTI δ1	NYUv2 δ1	$\color{green}{\textsf{Imp.}}$ %	Par. M	FPS
Inference height 512
v3.1 BEiTL-512	0.1137	0.0659	0.2366	6.13	11.56*	1.86*	$\color{green}{\textsf{19}}$	345	5.7
v3.1 BEiTL-512$\tiny{\square}$	0.1121	0.0614	0.2090	6.46	5.00*	1.90*	$\color{green}{\textsf{34}}$	345	5.7
Inference height 384
v3.1 BEiTL-512	0.1245	0.0681	0.2176	6.13	6.28*	2.16*	$\color{green}{\textsf{28}}$	345	12
v3.1 Swin2L-384$\tiny{\square}$	0.1106	0.0732	0.2442	8.87	5.84*	2.92*	$\color{green}{\textsf{22}}$	213	41
v3.1 Swin2B-384$\tiny{\square}$	0.1095	0.0790	0.2404	8.93	5.97*	3.28*	$\color{green}{\textsf{22}}$	102	39
v3.1 SwinL-384$\tiny{\square}$	0.1126	0.0853	0.2428	8.74	6.60*	3.34*	$\color{green}{\textsf{17}}$	213	49
v3.1 BEiTL-384	0.1239	0.0667	0.2545	7.17	9.84*	2.21*	$\color{green}{\textsf{17}}$