generative-models

Generative Models by Stability AI

News

May 20, 2025

• We are releasing Stable Video 4D 2.0 (SV4D 2.0), an enhanced video-to-4D diffusion model for high-fidelity novel-view video synthesis and 4D asset generation. For research purposes:
  • SV4D 2.0 was trained to generate 48 frames (12 video frames x 4 camera views) at 576x576 resolution, given a 12-frame input video of the same size, ideally consisting of white-background images of a moving object.
  • Compared to our previous 4D model SV4D, SV4D 2.0 can generate videos with higher fidelity, sharper details during motion, and better spatio-temporal consistency. It also generalizes much better to real-world videos. Moreover, it does not rely on refernce multi-view of the first frame generated by SV3D, making it more robust to self-occlusions.
  • To generate longer novel-view videos, we autoregressively generate 12 frames at a time and use the previous generation as conditioning views for the remaining frames.
  • Please check our project page, arxiv paper and video summary for more details.

QUICKSTART :

• python scripts/sampling/simple_video_sample_4d2.py --input_path assets/sv4d_videos/camel.gif --output_folder outputs (after downloading sv4d2.safetensors from HuggingFace into checkpoints/)

To run SV4D 2.0 on a single input video of 21 frames:

• Download SV4D 2.0 model (sv4d2.safetensors) from here to checkpoints/: huggingface-cli download stabilityai/sv4d2.0 sv4d2.safetensors --local-dir checkpoints
• Run inference: python scripts/sampling/simple_video_sample_4d2.py --input_path <path/to/video>
  • input_path : The input video <path/to/video> can be
    • a single video file in gif or mp4 format, such as assets/sv4d_videos/camel.gif, or
    • a folder containing images of video frames in .jpg, .jpeg, or .png format, or
    • a file name pattern matching images of video frames.
  • num_steps : default is 50, can decrease to it to shorten sampling time.
  • elevations_deg : specified elevations (reletive to input view), default is 0.0 (same as input view).
  • Background removal : For input videos with plain background, (optionally) use rembg to remove background and crop video frames by setting --remove_bg=True. To obtain higher quality outputs on real-world input videos with noisy background, try segmenting the foreground object using Clipdrop or SAM2 before running SV4D.
  • Low VRAM environment : To run on GPUs with low VRAM, try setting --encoding_t=1 (of frames encoded at a time) and --decoding_t=1 (of frames decoded at a time) or lower video resolution like --img_size=512.

Notes:

• We also train a 8-view model that generates 5 frames x 8 views at a time (same as SV4D).
  • Download the model from huggingface: huggingface-cli download stabilityai/sv4d2.0 sv4d2_8views.safetensors --local-dir checkpoints
  • Run inference: python scripts/sampling/simple_video_sample_4d2.py --model_path checkpoints/sv4d2_8views.safetensors --input_path assets/sv4d_videos/chest.gif --output_folder outputs
  • The 5x8 model takes 5 frames of input at a time. But the inference scripts for both model take 21-frame video as input by default (same as SV3D and SV4D), we run the model autoregressively until we generate 21 frames.
• Install dependencies before running:

python3.10 -m venv .generativemodels
source .generativemodels/bin/activate
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 # check CUDA version
pip3 install -r requirements/pt2.txt
pip3 install .
pip3 install -e git+https://github.com/Stability-AI/datapipelines.git@main#egg=sdata

July 24, 2024

• We are releasing Stable Video 4D (SV4D), a video-to-4D diffusion model for novel-view video synthesis. For research purposes:
  • SV4D was trained to generate 40 frames (5 video frames x 8 camera views) at 576x576 resolution, given 5 context frames (the input video), and 8 reference views (synthesised from the first frame of the input video, using a multi-view diffusion model like SV3D) of the same size, ideally white-background images with one object.
  • To generate longer novel-view videos (21 frames), we propose a novel sampling method using SV4D, by first sampling 5 anchor frames and then densely sampling the remaining frames while maintaining temporal consistency.
  • To run the community-build gradio demo locally, run python -m scripts.demo.gradio_app_sv4d.
  • Please check our project page, tech report and video summary for more details.

QUICKSTART : python scripts/sampling/simple_video_sample_4d.py --input_path assets/sv4d_videos/test_video1.mp4 --output_folder outputs/sv4d (after downloading sv4d.safetensors and sv3d_u.safetensors from HuggingFace into checkpoints/)

To run SV4D on a single input video of 21 frames:

• Download SV3D models (sv3d_u.safetensors and sv3d_p.safetensors) from here and SV4D model (sv4d.safetensors) from here to checkpoints/
• Run python scripts/sampling/simple_video_sample_4d.py --input_path <path/to/video>
  • input_path : The input video <path/to/video> can be
    • a single video file in gif or mp4 format, such as assets/sv4d_videos/test_video1.mp4, or
    • a folder containing images of video frames in .jpg, .jpeg, or .png format, or
    • a file name pattern matching images of video frames.
  • num_steps : default is 20, can increase to 50 for better quality but longer sampling time.
  • sv3d_version : To specify the SV3D model to generate reference multi-views, set --sv3d_version=sv3d_u for SV3D_u or --sv3d_version=sv3d_p for SV3D_p.
  • elevations_deg : To generate novel-view videos at a specified elevation (default elevation is 10) using SV3D_p (default is SV3D_u), run python scripts/sampling/simple_video_sample_4d.py --input_path assets/sv4d_videos/test_video1.mp4 --sv3d_version sv3d_p --elevations_deg 30.0
  • Background removal : For input videos with plain background, (optionally) use rembg to remove background and crop video frames by setting --remove_bg=True. To obtain higher quality outputs on real-world input videos with noisy background, try segmenting the foreground object using Clipdrop or SAM2 before running SV4D.
  • Low VRAM environment : To run on GPUs with low VRAM, try setting --encoding_t=1 (of frames encoded at a time) and --decoding_t=1 (of frames decoded at a time) or lower video resolution like --img_size=512.

March 18, 2024

• We are releasing SV3D, an image-to-video model for novel multi-view synthesis, for research purposes:
  • SV3D was trained to generate 21 frames at resolution 576x576, given 1 context frame of the same size, ideally a white-background image with one object.
  • SV3D_u: This variant generates orbital videos based on single image inputs without camera conditioning..
  • SV3D_p: Extending the capability of SVD3_u, this variant accommodates both single images and orbital views allowing for the creation of 3D video along specified camera paths.
  • We extend the streamlit demo scripts/demo/video_sampling.py and the standalone python script scripts/sampling/simple_video_sample.py for inference of both models.
  • Please check our project page, tech report and video summary for more details.

To run SV3D_u on a single image:

• Download sv3d_u.safetensors from https://huggingface.co/stabilityai/sv3d to checkpoints/sv3d_u.safetensors
• Run python scripts/sampling/simple_video_sample.py --input_path <path/to/image.png> --version sv3d_u

To run SV3D_p on a single image:

• Download sv3d_p.safetensors from https://huggingface.co/stabilityai/sv3d to checkpoints/sv3d_p.safetensors

1. Generate static orbit at a specified elevation eg. 10.0 : python scripts/sampling/simple_video_sample.py --input_path <path/to/image.png> --version sv3d_p --elevations_deg 10.0
2. Generate dynamic orbit at a specified elevations and azimuths: specify sequences of 21 elevations (in degrees) to elevations_deg ([-90, 90]), and 21 azimuths (in degrees) to azimuths_deg [0, 360] in sorted order from 0 to 360. For example: python scripts/sampling/simple_video_sample.py --input_path <path/to/image.png> --version sv3d_p --elevations_deg [<list of 21 elevations in degrees>] --azimuths_deg [<list of 21 azimuths in degrees>]

To run SVD or SV3D on a streamlit server: streamlit run scripts/demo/video_sampling.py

November 28, 2023

• We are releasing SDXL-Turbo, a lightning fast text-to image model. Alongside the model, we release a technical report
  • Usage:
    • Follow the installation instructions or update the existing environment with pip install streamlit-keyup.
    • Download the weights and place them in the checkpoints/ directory.
    • Run streamlit run scripts/demo/turbo.py.

November 21, 2023

• We are releasing Stable Video Diffusion, an image-to-video model, for research purposes:
  • SVD: This model was trained to generate 14 frames at resolution 576x1024 given a context frame of the same size. We use the standard image encoder from SD 2.1, but replace the decoder with a temporally-aware deflickering decoder.
  • SVD-XT: Same architecture as SVD but finetuned for 25 frame generation.
  • You can run the community-build gradio demo locally by running python -m scripts.demo.gradio_app.
  • We provide a streamlit demo scripts/demo/video_sampling.py and a standalone python script scripts/sampling/simple_video_sample.py for inference of both models.
  • Alongside the model, we release a technical report.

July 26, 2023

• We are releasing two new open models with a permissive CreativeML Open RAIL++-M license (see Inference for file hashes):
  • SDXL-base-1.0: An improved version over SDXL-base-0.9.
  • SDXL-refiner-1.0: An improved version over SDXL-refiner-0.9.

July 4, 2023

• A technical report on SDXL is now available here.

June 22, 2023

• We are releasing two new diffusion models for research purposes:
  • SDXL-base-0.9: The base model was trained on a variety of aspect ratios on images with resolution 1024^2. The base model uses [OpenCLIP-ViT/G](https://github.com/mlfoundation