Dense matching library based on PyTorch
A general dense matching library based on PyTorch.
For any questions, issues or recommendations, please contact Prune at [email protected]
If you're interested in training a probabilistic correspondence network unsupervised (mix of PDCNet and WarpC), check out our recent work Refign, with code. It is also integrated in this code base as UAWarpC.
06/03/2022: We found that significantly better performance and reduced training time are obtained when initializing with bilinear interpolation weights the weights of the transposed convolutions used to upsample the predicted flow fields between the different pyramid levels. We have integrated this initialization as the default. We might provide updated pre-trained weights as well. Alternatively, one can directly simply use bilinear interpolation for upsampling with similar (maybe a bit better) performance, which is also now an option proposed.
Libraries for implementing, training and evaluating dense matching networks. It includes
Common dense matching validation datasets for geometric matching (MegaDepth, RobotCar, ETH3D, HPatches), optical flow (KITTI, Sintel) and semantic matching (TSS, PF-Pascal, PF-Willow, Spair).
Scripts to analyse network performance and obtain standard performance scores for matching and pose estimation.
General building blocks, including deep networks, optimization, feature extraction and utilities.
General training framework for training dense matching networks with
Official implementation of GLU-Net (CVPR 2020), GLU-Net-GOCor (NeurIPS 2020), PWC-Net-GOCor (NeurIPS 2020), PDC-Net (CVPR 2021), WarpC models (ICCV 2021), PWarpC models (CVPR 2022) including trained models and respective results.
The repo contains the implementation of the following matching models. We provide pre-trained model weights, data preparation, evaluation commands, and results for each dataset and method. If you find this library useful, please consider citing the relevant following research papers.
Authors: Prune Truong, Martin Danelljan, Fisher Yu, Luc Van Gool
[Paper] [Website] [Poster] [Video] [Citation]
We propose Probabilistic Warp Consistency, a weakly-supervised learning objective for semantic matching. Our approach directly supervises the dense matching scores predicted by the network, encoded as a conditional probability distribution. We first construct an image triplet by applying a known warp to one of the images in a pair depicting different instances of the same object class. Our probabilistic learning objectives are then derived using the constraints arising from the resulting image triplet. We further account for occlusion and background clutter present in real image pairs by extending our probabilistic output space with a learnable unmatched state. To supervise it, we design an objective between image pairs depicting different object classes. We validate our method by applying it to four recent semantic matching architectures. Our weakly-supervised approach sets a new state-of-the-art on four challenging semantic matching benchmarks. Lastly, we demonstrate that our objective also brings substantial improvements in the strongly-supervised regime, when combined with keypoint annotations.
Authors: Prune Truong, Martin Danelljan, Radu Timofte, Luc Van Gool
Authors: Prune Truong, Martin Danelljan, Fisher Yu, Luc Van Gool
[Paper] [Website] [Poster] [Slides] [Video] [Citation]
| Warp Consistency Graph | Results |
|---|---|
The key challenge in learning dense correspondences lies in the lack of ground-truth matches for real image pairs. While photometric consistency losses provide unsupervised alternatives, they struggle with large appearance changes, which are ubiquitous in geometric and semantic matching tasks. Moreover, methods relying on synthetic training pairs often suffer from poor generalisation to real data. We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression. Our objective is effective even in settings with large appearance and view-point changes. Given a pair of real images, we first construct an image triplet by applying a randomly sampled warp to one of the original images. We derive and analyze all flow-consistency constraints arising between the triplet. From our observations and empirical results, we design a general unsupervised objective employing two of the derived constraints. We validate our warp consistency loss by training three recent dense correspondence networks for the geometric and semantic matching tasks. Our approach sets a new state-of-the-art on several challenging benchmarks, including MegaDepth, RobotCar and TSS.
Authors: Prune Truong, Martin Danelljan, Luc Van Gool, Radu Timofte
[Paper] [Website] [Poster] [Slides] [Video] [Citation]
Dense flow estimation is often inaccurate in the case of large displacements or homogeneous regions. For most applications and down-stream tasks, such as pose estimation, image manipulation, or 3D reconstruction, it is crucial to know when and where to trust the estimated matches. In this work, we aim to estimate a dense flow field relating two images, coupled with a robust pixel-wise confidence map indicating the reliability and accuracy of the prediction. We develop a flexible probabilistic approach that jointly learns the flow prediction and its uncertainty. In particular, we parametrize the predictive distribution as a constrained mixture model, ensuring better modelling of both accurate flow predictions and outliers. Moreover, we develop an architecture and training strategy tailored for robust and generalizable uncertainty prediction in the context of self-supervised training.
Authors: Prune Truong *, Martin Danelljan *, Luc Van Gool, Radu Timofte
[Paper] [Website] [Video] [Citation]
The feature correlation layer serves as a key neural network module in numerous computer vision problems that involve dense correspondences between image pairs. It predicts a correspondence volume by evaluating dense scalar products between feature vectors extracted from pairs of locations in two images. However, this point-to-point feature comparison is insufficient when disambiguating multiple similar regions in an image, severely affecting the performance of the end task. This work proposes GOCor, a fully differentiable dense matching module, acting as a direct replacement to the feature correlation layer. The correspondence volume generated by our module is the result of an internal optimization procedure that explicitly accounts for similar regions in the scene. Moreover, our approach is capable of effectively learning spatial matching priors to resolve further matching ambiguities.
Authors: Prune Truong, Martin Danelljan and Radu Timofte
[Paper] [Website] [Poster] [Oral Video] [Teaser Video] [Citation]
The pre-trained models can be found in the model zoo
Inference runs for torch version >= 1.0
conda create -n dense_matching_env python=3.7
conda activate dense_matching_envpip install numpy opencv-python torch torchvision matplotlib imageio jpeg4py scipy pandas tqdm gdown pycocotools timmNote: CUDA is required to run the code. Indeed, the correlation layer is implemented in CUDA using CuPy, which is why CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided bin
No open issues yet, or sync has not completed.