qwen3_vl: no way to reset the KV cache, so a model instance is single-use
Summary
Qwen3VLModel provides no way to reset its KV cache. Each Attention creates kv_cache: Arc<Mutex<KvCache>> in Attention::new, and nothing else can reach it, so a second generation on the same instance decodes against the previous conversation's keys and values. Since the model is typically multiple GB, reconstructing it per generation isn't a practical workaround.
Where
candle-transformers/src/models/qwen3_vl/text.rs — kv_cache is created in Attention::new and used only in Attention::forward. grep -rn "clear_kv_cache\|fn reset" candle-transformers/src/models/qwen3_vl/ returns nothing.
For comparison, 33 model files in the crate expose pub fn clear_kv_cache, including qwen2.rs, qwen3.rs, qwen3_moe.rs, quantized_qwen3.rs, paligemma.rs and mixformer.rs.
Reproduction
Randomly initialised tiny model, CPU, no downloads, runs in well under a second.
Cargo.toml:
[dependencies]
candle-core = "0.10.2"
candle-nn = "0.10.2"
candle-transformers = "0.10.2"
serde_json = "1"src/main.rs:
use candle_core::{DType, Device, Tensor};
use candle_nn::{VarBuilder, VarMap};
use candle_transformers::models::qwen3_vl::{Config, Qwen3VLModel};
const TINY_CONFIG: &str = r#"{
"text_config": {
"head_dim": 8, "vocab_size": 64, "hidden_size": 32,
"intermediate_size": 64, "num_hidden_layers": 2,
"num_attention_heads": 4, "num_key_value_heads": 2,
"hidden_act": "silu", "max_position_embeddings": 128,
"rms_norm_eps": 0.000001, "tie_word_embeddings": false,
"rope_theta": 10000.0, "sliding_window": null
},
"vision_config": {
"depth": 1, "hidden_size": 32, "out_hidden_size": 32,
"intermediate_size": 32, "num_heads": 2
},
"image_token_id": 5, "video_token_id": 6,
"vision_start_token_id": 7, "vision_end_token_id": 8
}"#;
// Text-only forward: no images, so the vision tower is bypassed.
fn forward_text(model: &Qwen3VLModel, tokens: &[u32], offset: usize, device: &Device) -> Tensor {
let input = Tensor::new(tokens, device).unwrap().unsqueeze(0).unwrap();
model
.forward(&input, None, None, None, None,
vec![tokens.len()], vec![vec![]], vec![vec![]], &[offset])
.unwrap()
}
fn main() {
let device = Device::Cpu;
let config: Config = serde_json::from_str(TINY_CONFIG).unwrap();
let varmap = VarMap::new();
let vb = VarBuilder::from_varmap(&varmap, DType::F32, &device);
// Reference: a pristine model answering the second prompt.
let fresh = Qwen3VLModel::new(&config, vb.clone()).unwrap();
let reference = forward_text(&fresh, &[33, 44], 0, &device);
// Same prompt, on a model that already ran a different one.
let reused = Qwen3VLModel::new(&config, vb).unwrap();
let _ = forward_text(&reused, &[11, 22], 0, &device);
let after_reuse = forward_text(&reused, &[33, 44], 0, &device);
let a = reference.flatten_all().unwrap().to_vec1::<f32>().unwrap();
let b = after_reuse.flatten_all().unwrap().to_vec1::<f32>().unwrap();
let max_diff = a.iter().zip(b.iter()).map(|(x, y)| (x - y).abs()).fold(0.0f32, f32::max);
let scale = a.iter().map(|v| v.abs()).fold(0.0f32, f32::max);
println!("max |fresh - reused| = {max_diff:.6}");
println!("relative = {:.6}", max_diff / scale);
}The second conversation has to use different tokens from the first. Repeating the same tokens hides the bug: the stale entries are then bit-identical duplicates of the new keys and values, and attention over duplicated (key, value) pairs gives the same weighted average.
Expected: identical logits — same prompt, same weights.
Actual, four consecutive runs (weights are randomly initialised, so the magnitude varies, but it is never small):
max |fresh - reused| = 1.486256 relative = 0.342524
max |fresh - reused| = 3.680627 relative = 0.997730
max |fresh - reused| = 4.044358 relative = 0.773331
max |fresh - reused| = 1.621907 relative = 0.423533Suggested fix
Add pub fn clear_kv_cache(&mut self) on Qwen3VLTextModel forwarding to each layer's KvCache::reset(), and expose it on Qwen3VLModel, mirroring qwen2.rs / qwen3.rs. Happy to open a PR if that approach looks right.
Versions
candle-transformers 0.10.2 from crates.io; qwen3_vl/text.rs is unchanged on main as of 2026-08-17. Reproduced on macOS/aarch64, CPU.
Source: huggingface/candle