[NeurIPS 2024] 适用于视频帧插值的可推广隐式运动建模
GIMM-VFI performs generalizable continuous motion modeling and interpolations between two adjacent video frames at arbitrary timesteps.
:open_book: For more visual results of GIMM-VFI, go checkout our project page.
DS_SCALE, which should be a float between 0 and 1 for high-resolution interpolation, such as for 2K and 4K frames.# git clone this repository
git clone https://github.com/GSeanCDAT/GIMM-VFI
cd GIMM-VFI
# create new conda env
conda create -n gimmvfi python=3.7 -y
conda activate gimmvfi
# install other python dependencies
pip install -r requirements.txtGIMM-VFI can be implemented with different flow estimators. As described in our paper, we provide RAFT-based GIMM-VFI-R and FlowFormer-based GIMM-VFI-F in this repo.
Additionally, we release a perceptually enhanced version of GIMM-VFI that incorporates an additional learning objective, the LPIPS loss, during training. Denoted as GIMM-VFI-R-P and GIMM-VFI-F-P, these enhanced variants achieve substantial improvements in perceptual interpolation.
All the model checkpoints can be found from this link. Please put them into ./pretrained_ckpt folder after downloading.
Interpolation demos can be create through the following command:
sh scripts/video_Nx.sh YOUR_PATH_TO_FRAME YOUR_OUTPUT_PATH DS_SCALE N_INTERPThe model variant by default is GIMM-VFI-R-P. You can change the model variant in scripts/video_Nx.sh.
Here is an example usage for 9X interpolation:
sh scripts/video_Nx.sh demo/input_frames demo/output 1 9The expected interpolation output:
DS_SCALE is the downsampling scale factor ranging from 0 to 1. can be adjusted for high-resolution interpolations. For instance, you can perform 8X interpolation for 2K frames using following command:
sh scripts/video_Nx.sh demo/2K_input_frames demo/2K_output 0.5 8The expected interpolation output can be found here.
In our practice, we tested GIMM-VFI on 2K and 4K frames for 8X interpolations on Nvidia V100 GPUs. Following is our settings and the corresponding memory usages:
[2K interpolation] DS_SCALE: 0.5 Memory-Usage: 7932MiB
[4K interpolation] DS_SCALE: 0.25 Memory-Usage: 10922MiBDownload the Vimeo90K, SNU-FILM and X4K1000FPS datasets.
Obtain the motion modeling benchmark datasets, Vimeo-Triplet-Flow (VTF) and Vimeo-Septuplet-Flow (VSF), by extracting optical flows from the Vimeo90K triplet and septuplet test sets using FlowFormer.
The file structure should be like:
├── data
├── SNU-FILM
├── test
├── test-easy.txt
├── test-medium.txt
├── test-hard.txt
├── test-extreme.txt
├── x4k
├── test
├── Type1
├── Type2
├── Type3
├── vimeo90k
├── vimeo_septuplet
├── sequences
├── flow_sequences
├── vimeo_triplet
├── sequences
├── flow_sequencesOn the VTF benchmark:
sh scripts/bm_VTF.shOn the VSF benchmark:
sh scripts/bm_VSF.shOn the SNU-FILM-arb benchmark:
sh scripts/bm_SNU_FILM_arb.shOn the X4K benchmark:
sh scripts/bm_X4K.shThe model variants can be changed inside the shell scripts.
Following is the general command for training:
sh scripts/train.sh YOUR_CONFIG OUTPUT_DIR PRETRAINED_CKPT NUM_GPUSpecifically, you can train GIMM and GIMM-VFI by following the instructions below.
sh scripts/train.sh configs/gimm/gimm.yaml ./work_dirs '' 2sh scripts/train.sh configs/gimmvfi/gimmvfi_r_arb.yaml pretrained_ckpt/gimm.pt 8sh scripts/train.sh configs/gimmvfi/gimmvfi_f_arb.yaml pretrained_ckpt/gimm.pt 8If you find our work interesting or helpful, please leave a star or cite our paper.
@inproceedings{guo2024generalizable,
title={Generalizable Implicit Motion Modeling for Video Frame Interpolation},
author={Guo, Zujin and Li, Wei and Loy, Chen Change},
booktitle={Advances in Neural Information Processing Systems},
year={2024}
}The code is based on GINR-IPC and draws inspiration from several other outstanding works including RAFT, FlowFormer, AMT, softmax-splatting, EMA-VFI, MoTIF and LDMVFI.
暂无开放 Issues,或尚未同步最近议题。