百科.dev
全部条目在线工具AI 编程趋势榜开源项目技术资讯提交条目
登录
< 返回工具列表
S

SAMWISE

> 编程语言
开源

[CVPR 2025 重点展示] "SAMWISE: 为基于文本的视频分割引入智慧技术"

386 stars0 点赞0 次浏览
访问官网GitHub

工具介绍

[CVPR 2025 重点展示] "SAMWISE: 为基于文本的视频分割引入智慧技术"

Welcome to the official repository for "SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation".

In this work, we build upon Segment Anything 2 (SAM2) and make it wiser by infusing natural language understanding and explicit temporal modeling.
No fine-tuning of SAM2 weights.
No reliance on external VLMs for multi-modal interaction.
State-of-the-art performance across multiple benchmarks.
Minimal overhead: just 4.9 M additional parameters!

Read our paper on arXiv

Demo & Project Page

[May 2025] Check out SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation — a unified framework powered by SAM2, supporting points, boxes, scribbles, and masks. No external models, no prompt-specific tweaks. Checkout SANSA

[June 2025] Try SAMWISE on your own data: we’ve added a simple script to run SAMWISE on videos or images using textual prompts. Try SAMWISE on Your Own Data.

SAMWISE in Action

SAMWISE (our model, not the hobbit) segments objects from The Lord of the Rings in zero-shot — no extra training, just living up to its namesake! ‍♂️✨

https://github.com/user-attachments/assets/b582557e-a41a-4eb1-9069-a88dafd3e546

Data Preparation

Before running SAMWISE, set up your dataset: refer to data.md for detailed data preparation.
Once organized, the directory structure should look like this:

SAMWISE/
├── data/
│   ├── ref-youtube-vos/
│   ├── ref-davis/
│   ├── MeViS/
├── datasets/
├── models/
│   ├── sam2/
│   ├── samwise.py
│   ├── ...
...

⚙️ Environment Setup

The code has been tested with Python 3.10 and PyTorch 2.3.1 (with CUDA 11.8). To set up the environment using Conda, run:

bash
conda create --name samwise python=3.10 -y
conda activate samwise
pip install torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt

Referring Video Object Segmentation (RVOS)

Reproducing Our Results: Below, we provide the model weights to replicate the results of our paper.

Dataset Total Parameters Trainable Params J&F Model Zip
MeViS 210 M 4.9 M 49.5 Weights Zip
MeViS - valid_u 210 M 4.9 M 57.1 Weights -
Ref-Youtube-VOS 210 M 4.9 M 69.2 Weights Zip
Ref-Davis 210 M 4.9 M 70.6 Weights -

To evaluate the model on MeViS - valid_u run the following command:

python3 inference_mevis.py --split valid_u --resume=[/path/to/model_weight] --name_exp [name_exp] --HSA --use_cme_head

For Ref-Davis run the following command:

python3 inference_davis.py --resume=[/path/to/model_weight] --name_exp [name_exp]  --HSA --use_cme_head

For MeViS and Ref-Youtube-VOS, upload the zip file to:

  • Ref-Youtube-VOS Competition Server
  • MeViS Competition Server

️ Referring Image Segmentation (RIS)

We also test SAMWISE on the Referring Image Segmentation (RIS) benchmark.

RefCOCO RefCOCO+ RefCOCOg Model
75.6 65.8 66.8 Weights

Run the following to evaluate on RIS:

python3 main_pretrain.py --eval --resume=[/path/to/model_weight] --name_exp [name_exp] --disable_pred_obj_score 

Training and Inference

For step-by-step instructions on training and inference, please refer to the Training and Inference Guide.

This document includes all necessary details on:

  • Training SAMWISE on different datasets
  • Running inference and evaluating performance
  • Submitting results to online benchmarks

▶️ Try SAMWISE on Your Own Data

We provide a simple script to run SAMWISE on your own inputs using natural language prompts.
Supported input types:

  • A single image (.jpg, .png, .jpeg)
  • A video (.mp4)
  • A folder of consecutive video frames (e.g., frame_00001.png, frame_00002.png, ...)

Run the script:

bash
python inference_demo.py --input_path  --text_prompts  

Examples:

bash
# On a single image
python inference_demo.py --input_path assets/example_image.jpg --text_prompts "the dog who is jumping" "the dog on the left" "the person with a yellow jacket"

# On a video
python inference_demo.py --input_path assets/example_video.mp4 --text_prompts "the horse jumping" "the person riding the horse"

# On a folder of consecutive frames
python inference_demo.py --input_path demo_sequence --text_prompts "the horse jumping" "the person riding the horse"

Output:

  • Image input:
    • demo_output//example_image.png
  • Video or sequence of frames:
    • Segmented frames: demo_output//frame_*.png
    • Segmented video: demo_output/.mp4

Acknowledgements

We build upon the amazing work from:

  • Segment Anything 2
  • ReferFormer
  • Fairseq

Citation

@InProceedings{cuttano2025samwise,
    author    = {Cuttano, Claudia and Trivigno, Gabriele and Rosi, Gabriele and Masone, Carlo and Averta, Giuseppe},
    title     = {SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation},
    booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
    month     = {June},
    year      = {2025},
    pages     = {3395-3405}
}

Issues· 0 开放

查看全部 Issues在 GitHub 打开

暂无开放 Issues,或尚未同步最近议题。

> 标签

Python

暂无评论,来聊聊你的看法吧

> 工具信息

发布日期2026年8月1日
最后更新2026年9月17日
分类编程语言
定价开源

> 相关工具

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言