一个简洁、优雅的框架和教程,用于实时互动的世界模型
A full-stack framework and tutorial for newcomers, rather than a specific model.
minWM is our contribution to the world-model community: a full-stack open-source framework that walks you end-to-end through turning a bidirectional T2V foundation model into an action-conditioned video world model — with example data, runnable scripts, Claude skills capturing our hands-on experience, and onboarding knowledge for newcomers. We hope more researchers and developers join us in growing the community together.
https://github.com/user-attachments/assets/99c25915-7fe7-4a20-a2c4-9d291502fccf
v0.1-legacy).The complete data → training → inference pipeline is open-sourced; every stage exposes input/output checkpoints so you can stop, swap, or fork anywhere.
1.1 Data. We walk you through how to construct training-ready datasets paired with camera poses, and the full data processing pipeline that turns them into latents.
1.2 Training. Including FSDP + sequence parallelism, single-/multi-node training, and the full distillation pipeline from a bidirectional diffusion model to a 4-step AR student:
…
1.3 Inference: 4-step DMD inference for HY Action2V / HY TI2V / Wan Action2V, multi-GPU sequence parallelism, camera-trajectory control via pose strings ("a*4,w*8,s*7") or JSON files
From Scratch: Bidirectional T2V Foundation → Real-Time World Model
The HunyuanVideo 1.5 and Wan 2.1 lines walk through the full 4-stage pipeline — starting from a bidirectional T2V foundation model and ending at a 4-step autoregressive world model.
| Backbone | Architecture | Params | Training | Inference |
|---|---|---|---|---|
| Wan 2.1 | Cross-attention + DiT | 1.3 B | all 4 stages | 4-step DMD |
| HunyuanVideo 1.5 | MMDiT | 8 B | all 4 stages | 4-step DMD |
Both lines share the same trainer / loss / dataset abstractions, so adding a third backbone is structurally a wrapper-and-config exercise.
We are packaging our project experience across the CF / CF++ pipeline as Claude skills, so that an LLM assistant can help users debug failures and integrate new models without reverse-engineering the whole repo.
debug-world-model — collected failure modes from the training pipeline (loss NaN, frame-to-frame jitter, camera drift, memory attenuation, distillation collapse, …). Claude diagnoses likely root causes from your symptoms instead of guessing.integrate-new-backbone — step-by-step recipe for plugging a new video DiT into minWM, grounded in the HunyuanVideo and Wan reference integrations — e.g. "look at how HY does teacher forcing here, do the same for your model there".onboarding-world-model — A third Claude skill aimed at researchers entering the world-model space for the first time. Two parts:Intended audience: graduate students, independent researchers, and junior labs that want to enter the world-model space without spending three months reverse-engineering existing repos.
conda create -n minwm python=3.12 -y
conda activate minwm
pip install -r requirements/base.txt
pip install flash-attn --no-build-isolation
pip install -e . # editable install: makes `import minwm` resolve, no PYTHONPATH
Full requirements, verification, developer setup, and troubleshooting: see
INSTALL.md.Saving checkpoints to a remote object store (
s3://,oss://, or an S3-compatible store like Baidu BOS)? Install the matching fsspec backend — see Remote Checkpoint Storage.
Model Checkpoints (Click to expand)
All weights live under ./ckpts/ after download.
| Checkpoint | Backbone | Stage | Use case | Download |
|---|---|---|---|---|
HunyuanVideo-1.5 (base) |
HY 1.5 | — | Required by both HY pipelines | HF |
HY15/Action2V/bidirectional |
HY 1.5 | Phase 1 SFT | Starting point for HY Action2V Phase 2 | HF |
HY15/Action2V/ar_diffusion_tf |
HY 1.5 | Phase 2 Stage 1 | Teacher Forcing AR diffusion | HF |
HY15/Action2V/causal_ode |
HY 1.5 | Phase 2 Stage 2a (proposed in Causal Forcing) | DMD initialization | HF |
HY15/Action2V/causal_cd |
HY 1.5 | Phase 2 Stage 2b (proposed in Causal Forcing++) | DMD initialization | HF |
HY15/Action2V/dmd |
HY 1.5 | Phase 2 Stage 3 | 4-step real-time inference | HF |
Wan21/Action2V/{bidirectional,ar_diffusion_tf,causal_ode,causal_cd,dmd} |
Wan 2.1 | Same 4 stages | Wan pipeline | HF |
Wan21/Action2V/bidirectional/model_v2.pt |
Wan 2.1 | Phase 1 SFT (v2, retrained data) | Improved Wan Action2V bidirectional | HF |
Wan2.1-T2V-1.3B (base) |
Wan 2.1 | — | Required by Wan pipeline | HF |
…
Checkpoint naming: release names → config names (Click to expand)
The two naming schemes differ, so each downloaded stage needs one directory-level
symlink. The links are relative (so ./ckpts/ stays movable) and re-running is a
no-op. Wan configs load /model.pt; HY configs load / as a
diffusers directory — a single link per stage satisfies both.
Keep the -T: without it, if the stage{N}_* path already exists as a real
directory (e.g. you exported your own checkpoint there), ln would quietly create
a nested link inside it and the config would keep loading the old weights. With
-T you get a loud cannot overwrite directory instead, and nothing is touched.
| Release name (on HF) | Config name (symlink) | Stage |
|---|---|---|
bidirectional |
stage0_bi_sft |
Phase 1 bidirectional SFT |
ar_diffusion_tf |
stage1_ar_tf |
Phase 2 Stage 1 teacher forcing |
causal_ode |
stage2_ar_ode |
Phase 2 Stage 2(a) ODE distillation |
causal_cd |
stage2_ar_cd |
Phase 2 Stage 2(b) consistency distillation |
dmd |
stage3_ar_dmd |
Phase 2 Stage 3 DMD (4-step) |
To link every stage you downloaded, for either model line:
for line in Wan21/Action2V HY15/Action2V; do
( cd ./ckpts/$line 2>/dev/null || exit 0
ln -sfnT bidirectional stage0_bi_sft
ln -sfnT ar_diffusion_tf stage1_ar_tf
ln -sfnT causal_ode stage2_ar_ode
ln -sfnT causal_cd stage2_ar_cd
ln -sfnT dmd stage3_ar_dmd )
done
Dangling links for stages you did not download are harmless — nothing reads them.
Alternatively, skip the links entirely and pass the release path explicitly:
inference.checkpoint=./ckpts/Wan21/Action2V/dmd/model.pt.
All inference goes through one entrypoint — tools/infer_mwm.py — and the loop,
sampler, guidance and step count come from the --config-file, not from CLI flags. So
switching model line or stage means switching the config; the command shape never changes.
The input is a benchmark JSON named by the config key inference.benchmark: a list of
[{id, caption, trajectory}] items (extra keys ignored), each output named {id}.mp4. An
image field makes the item image-to-video (HY); omitting it makes it text-to-video (Wan).
…
Each run writes one .mp4 per sample (Wan Action2V: 832×480, 77 frames @ 16 fps) plus a
manifest.json recording the config and the per-item {prompt, trajectory, seed, video}.
Every config value is overridable inline as a dotlist key=value: inference.checkpoint= /
inference.output_dir= override the config's values, inference.limit=N runs only the first
N items, inference.seed=, and inference.sp_size=N (with a matching --nproc_per_node=N)
for sequence-parallel sampling.
Camera trajectories. Format is
key*Nsegments joined by commas — e.g.d*8,i*5,l*6= pan right 8, tilt up 5, pan left 6.w/s/a/dtranslate,i/k/j/lrotate. For 20 latent frames the segment counts sum to 19. The trajectory is per-sample, read from each benchmark item's"trajectory"field.
Renders the WASD/KIJL key presses onto each clip and concatenates them into one overview.
It reads manifest.json, so it works on any output directory produced above:
python demos/overlay_from_manifest.py \
--input-dir ./outputs/quickstart_wan_action2v \
--output final_with_keys.mp4
Needs
ffmpeg/ffprobeonPATH. Cluster images often ship without them — run this step locally on the (shared-filesystem) output directory instead.
Before starting any training stage, prepare the raw videos and camera trajectories. Choose one option; both produce the same ./dataset/ layout.
The videos are generated with HunyuanVideo (HY-WorldPlay); their use is subject to the upstream model's license terms.
hf download MIN-Lab/minWM-data --repo-type dataset \
--local-dir ./datas
暂无开放 Issues，或尚未同步最近议题。