Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
T

tpu-inference

> 编程语言
Open source

TPU inference for vLLM, with unified JAX and PyTorch support.

397 stars0 likes0 views
WebsiteGitHub

About

TPU inference for vLLM, with unified JAX and PyTorch support.

| Documentation | Blog | User Forum | Developer Slack (#sig-tpu) |


Contribute to the Project

Looking to help? Click a badge below to find issues that need your attention.

Latest News

  • Announcing Gemma 4 on vLLM Byte for byte, the most capable open models - available on TPUs on Day 0!

Previous News

  • Pytorch Conference Learn how Spotify uses vLLM with both GPUs and TPUs to drive down costs and improve user experience.
  • Ray Summit, November 3-5 in San Francisco!
  • JAX DevLab on November 18th in Sunnyvale!
  • [2025/10] vLLM TPU: A New Unified Backend Supporting PyTorch and JAX on TPU

About

vLLM TPU is now powered by tpu-inference, an expressive and powerful new hardware plugin unifying JAX and PyTorch under a single lowering path within the vLLM project. The new backend now provides a framework for developers to:

  • Push the limits of TPU hardware performance in open source.
  • Provide more flexibility to JAX and PyTorch users by running PyTorch model definitions performantly on TPU without any additional code changes, while also extending native support to JAX.
  • Retain vLLM standardization: keep the same user experience, telemetry, and interface.

Recommended models and features

Although vLLM TPU’s new unified backend makes out-of-the-box high performance serving possible with any model supported in vLLM, the reality is that we're still in the process of implementing a few core components.

For this reason, we’ve provided Recommended Models and Recommended Features pages detailing the models and features that are validated through unit, integration, and performance testing.

Get started

Get started with vLLM on TPUs by following the quickstart guide.

Visit our documentation to learn more.

Compatible TPU Generations

  • Recommended: v7x, v5e, v6e
  • Experimental: v3, v4, v5p

Recipes

  • v7x (Ironwood) Recipes
  • v6e (Trillium) Recipes

TPU Support Matrix Dashboard

Below is the live status of our supported models, features, and kernels. Click on any category to expand the detailed support table. It is automatically updated from our detailed Support Matrices.

Last Updated: 2026-08-27 07:02 AM UTC

Status Legend

  • ✅ Passing: Tested and works as expected. Ready for use.

  • ❌ Failing: Known to be broken or not functional. Help is wanted to fix this!

  • Experimental: Works, but unoptimized or pending community validation.

  • Planned: Not yet implemented, but on the official roadmap.

  • ⛔️ Unplanned: There is no benefit to adding this.

  • ❓ Untested: The functionality exists but has not been recently or thoroughly verified.

    View Matrix Aggregation Rules (v6e/v7x & C+P)

  • ️ Correctness + Performance (C + P)

    • ❌ Failing: If either check fails.
    • ✅ Passing: If BOTH checks pass successfully.
    • ❓ Untested: If any check is untested (and neither fails).
  • ** Hardware Rollups (v6e + v7x)**

    • ❌ Failing: If the feature fails on either v6e or v7x.
    • ✅ Passing: If the feature passes on BOTH v6e and v7x.
    • ❓ Untested: If either generation is untested (and neither fails).

Release Support Matrices

Click to expand support matrices

Stable support status for official releases and production deployments.

✅ Tested Models

Model Type Unit Test Correctness Test Performance Test
google/gemma-4-26B-A4B-it Multimodal ✅ ✅ ✅
google/gemma-4-31B-it Multimodal ✅ ✅ ✅
google/gemma-4-E2B-it Multimodal ✅ ✅ ✅
google/gemma-4-E4B-it Multimodal ✅ ✅ ✅
google/gemma-3-27b-it Text ✅ ✅ ✅
meta-llama/Llama-3.1-8B-Instruct Text ✅ ✅ ✅
meta-llama/Llama-3.3-70B-Instruct Text ✅ ✅ ✅
Qwen/Qwen3-30B-A3B Text ✅ ✅ ✅
Qwen/Qwen3-32B Text ✅ ✅ ✅
Qwen/Qwen3-4B Text ✅ ✅ ✅
Qwen/Qwen3-Coder-480B-A35B-Instruct Text ✅ ✅ ✅
Qwen/Qwen3.5-397B-A17B Text ✅ ✅ ✅
Qwen/Qwen3-Embedding-8B Embedding ✅ ✅ ❓
Qwen/Qwen2.5-VL-7B-Instruct Multimodal ✅ ✅ ❓
deepseek-ai/DeepSeek-R1 Text ✅ ✅ ❓
moonshotai/Kimi-K2.6 Text ✅ ✅ ❓
openai/gpt-oss-120b Text ✅ ✅ ❓
deepseek-ai/DeepSeek-OCR Multimodal ❓ ❓ ❓
meta-llama/Llama-4-Maverick-17B-128E-Instruct/multimodal Multimodal ❓ ❓ ❓
Qwen/Qwen3-Omni-30B-A3B-Instruct Multimodal ❓ ❓ ❓
Qwen/Qwen3-VL-8B-Instruct Multimodal ❓ ❓ ❓
Qwen/Qwen3.5-9B Multimodal ❓ ❓ ❓
deepseek-ai/DeepSeek-Math-V2 Text ❓ ❓ ❓
deepseek-ai/DeepSeek-V3.1 Text ❓ ❓ ❓
deepseek-ai/DeepSeek-V3.2 Text ❓ ❓ ❓
deepseek-ai/DeepSeek-V3.2-Speciale Text ❓ ❓ ❓
MiniMaxAI/MiniMax-M2.5 Text ❓ ❓ ❓
mistralai/Mistral-Small-4-119B-2603/text-only Text ❓ ❓ ❓
moonshotai/Kimi-K2-Thinking Text ❓ ❓ ❓
openai/gpt-oss-20b Text ❓ ❓ ❓
zai-org/GLM-5 Text ❓ ❓ ❓

  Advanced Capabilities

Core Features

  Feature
  Flax
  Torchax
  Default






  async scheduler
  ✅
  ✅
  ✅




  Chunked Prefill
  ✅
  ✅
  ✅




  DCN-based P/D disaggregation
  ✅
  ✅
  ✅




  KV Cache Offload
  ✅
  ✅
  ✅




  LoRA_Torch
  ✅
  ✅
  ✅




  Multimodal Inputs
  ✅
  ✅
  ✅




  Out-of-tree model support
  ✅
  ✅
  ✅




  Prefix Caching
  ✅
  ✅
  ✅




  Single Program Multi Data
  ✅
  ✅
  ✅




  Speculative Decoding: Eagle3
  ✅
  ✅
  ✅




  Speculative Decoding: Ngram
  ✅
  ✅
  ✅




  Speculative Decoding: DFlash
  ✅
  ❓
  ✅




  hybrid kv cache
  ❓
  ❓
  ❓




  multi-host
  ❓
  ❓
  ❓




  runai_model_streamer_loader
  ❓
  ❓
  ❓




  sampling_params
  ❓
  ❓
  ❓




  Step Pooling (Embedding)
  ❓
  ❓
  ❓




  structured_decoding
  ❓
  ❓
  ❓

Parallelism Techniques

  Feature
  Flax
  Torchax




  Single-host
  Multi-host
  Single-host
  Multi-host






  PP
  ✅
  ✅
  ✅
  ✅




  DP
  ✅
  ❓
  ✅
  ❓




  EP
  ✅
  ❓
  ✅
  ❓




  TP
  ✅
  ❓
  ✅
  ❓




  CP
  ❓
  ❓
  ❓
  ❓




  SP&nbsp;(vote&nbsp;to&nbsp;prioritize)
  ❓
  ❓
  ❓
  ❓

Quantization Methods

  Checkpoint dtype
  Method
  Supported

Hardware Acceleration Flax Torchax

  FP4 W4A16
  mxfp4
  v7
  ❓
  ❓




  FP8 W8A16
  compressed-tensor
  v7
  ❓
  ❓




  FP8 W8A8
  compressed-tensor
  v7
  ❓
  ❓




  INT4 W4A16
  awq
  v5, v6
  ❓
  ❓




  INT8 W8A8
  compressed-tensor
  v5, v6
  ❓
  ❓




  NVFP4 W4A16
  modelopt_fp4
  v7
  ❓
  ❓

Note:

  • This table only tests checkpoint loading compatibility.

Microbenchmark Kernel Support

  Category
  Test
  W16A16
  W8A8
  W8A16
  W4A4
  W4A8
  W4A16






  Moe
  Fused&nbsp;MoE
  ❓
  ❓
  ❓
  ❓
  ❓
  ❓




  gmm
  ❓
  ❓
  ❓
  ❓
  ❓
  ❓






  Dense
  All&#8209;gather&nbsp;matmul
  ❓
  ❓
  ❓
  ❓
  ❓
  ❓






  Attention
  Generic&nbsp;Ragged&nbsp;Paged

Attention V3* ❓ ❓ ❓ ❓ ❓ ❓

  MLA
  ❓
  ❓
  ❓
  ❓
  ❓
  ❓




  Ragged&nbsp;Paged

Attention V3 Head_Dim 64* ❓ ❓ ❓ ❓ ❓ ❓

Note:

  • For attention kernels, W[x]A[y] denotes KV cache as W, A as compute, and x, y as bit precision.

Nightly Support Matrices

Click to expand support matrices

Support status for the latest nightly/main branch developments.

✅ Tested Models

Model Type Unit Test Correctness Test Performance Test
google/gemma-4-26B-A4B-it Multimodal ✅ ✅ ✅
google/gemma-4-31B-it Multimodal ✅ ✅ ✅
google/gemma-3-27b-it Text ✅ ✅ ✅
[meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co

Issues· 501 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Python

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 17, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言