X-VLA

X-VLA: Soft-Prompted Transformer as a Scalable Cross-Embodiment Vision-Language-Action Model

Paper	Project Page	Hugging Face
Read the Full Research	Explore the Demos	Access Models & Datasets

Highlights & News

Exciting News: X-VLA Accepted to ICLR 2026

We are thrilled to announce that X-VLA has been accepted to ICLR 2026.

Now Supported in LeRobot

X-VLA is now natively integrated into the LeRobot platform.
Give it a try! We sincerely appreciate the support and collaboration from the Hugging Face team.

Champion Winner at IROS 2025

X-VLA won 1st Place (Champion) at the AgiBot World Challenge, held at IROS 2025.

---

Overview

Successful generalist Vision–Language–Action (VLA) models depend on scalable, cross-platform training across diverse robotic embodiments.
To leverage the heterogeneity of large-scale robot datasets, X-VLA introduces a soft prompt mechanism — embodiment-specific learnable embeddings that guide a unified Transformer backbone toward effective multi-domain policy learning.

The resulting architecture — X-VLA-0.9B — achieves state-of-the-art generalization across six simulation platforms and three real-world robots, surpassing prior VLA approaches in dexterity, adaptability, and efficiency.

https://github.com/user-attachments/assets/c047bac4-17c3-4d66-8036-badfab2b8c41

---

Quick Start: Installation & Deployment

1️⃣ Installation

# Clone the repository
git clone https://github.com/2toinf/X-VLA.git
cd X-VLA

# Create and activate Conda environment
conda create -n XVLA python=3.10 -y
conda activate XVLA

# Install dependencies
pip install -r requirements.txt

or

conda env create -f environment.yml
conda activate xvla-stable

---

2️⃣ Deploying X-VLA for Inference

X-VLA adopts a Server–Client architecture to separate the model environment from simulation or robot-specific dependencies. This design avoids package conflicts and supports distributed inference across GPUs, SLURM clusters, or edge devices.

Available Pre-trained Models

• We observed a slight performance drop (around 1% across different datasets) after converting our models to the HF format, and we’re actively investigating the cause.

About Libero Setup and Evluation

• For questions about converting relative actions to absolute actions and our implementation, please first refer to issue #2 and #15. We have updated full preprocessing guidance here.

Update: We have released the LoRA fine-tuning code, along with checkpoints and the associated inference code.

Model ID	Embodiment	Description	Performance	Evaluation Guidance
2toINF/X-VLA-Pt	Foundation	Pretrained on large-scale heterogeneous robot–vision–language datasets for general transfer.	—	—
2toINF/X-VLA-AgiWorld-Challenge	Agibot-G1	Fine-tuned for AgiWorld Challenge.	Champion	-
2toINF/X-VLA-Calvin-ABC_D	Franka	Fine-tuned on CALVIN benchmark (ABC_D subset)	4.43	Calvin Eval
2toINF/X-VLA-Google-Robot	Google Robot	Fine-tuned on large-scale Google Robot dataset	83.5%(VM) 76.4%(VA)	Simpler Eval
2toINF/X-VLA-Libero	Franka	Fine-tuned on LIBERO benchmark	98.1%	LIBERO Eval
2toINF/X-VLA-VLABench	Franka	Fine-tuned on VLABench benchmark	51.1(score)	VLABench Eval
2toINF/X-VLA-RoboTwin2	Agilex	Trained on RoboTwin2 dataset for dual-arm coordinated manipulation(50 demos for each task).	70%	RoboTwin2.0 Eval
2toINF/X-VLA-WidowX	WidowX	Fine-tuned on BridgeDataV2 (Simpler benchmark).	95.8%	Simpler Eval
2toINF/X-VLA-SoftFold	Agilex	Fine-tuned on Soft-Fold Dataset. Specialized in deformable object manipulation (e.g., folding and cloth control).	cloth folding with a 100% success rate in 2 hours.	SoftFold-Agilex
LoRA Adapters
2toINF/X-VLA-libero-spatial-peft	Franka	Fine-tuned on LIBERO benchmark	96.2%	LIBERO Eval
2toINF/X-VLA-libero-object-peft	Franka	Fine-tuned on LIBERO benchmark	96%	LIBERO Eval
2toINF/X-VLA-libero-goal-peft	Franka	Fine-tuned on LIBERO benchmark	94.4%	LIBERO Eval
2toINF/X-VLA-libero-long-peft	Franka	Fine-tuned on LIBERO benchmark	83.2%	LIBERO Eval
2toINF/X-VLA-simpler-widowx-peft	WidowX	Fine-tuned on BridgeDataV2 (Simpler benchmark).	66.7%	Simpler Eval

---

Notes

• All models share a consistent architecture: configuration_xvla.py, modeling_xvla.py, and unified tokenizer (tokenizer.json).
• The X-VLA-Pt model is the foundation checkpoint, trained across multiple robot domains.
• Each embodiment is fine-tuned for its respective environment while retaining cross-embodiment alignment.
• Evaluation scripts (in evaluation/) follow a standardized format for reproducible benchmarking.

---

Performance metrics follow standard evaluation protocols detailed in the paper.

---

3️⃣ Launching the Inference Server

from transformers import AutoModel, AutoProcessor
import json_numpy

# Load model and processor
model = AutoModel.from_pretrained("2toINF/X-VLA-WidowX", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("2toINF/X-VLA-WidowX", trust_remote_code=True)

# Start the inference server
print(" Starting X-VLA inference server...")
model.run(processor, host="0.0.0.0", port=8000)

Once launched, the API endpoint is available at:

POST http://:8000/act

---

4️⃣ Client Interaction & Action Prediction

The client communicates via HTTP POST, sending multimodal data (vision + language + proprioception) as a JSON payload.

Payload Structure

Key	Type	Description
proprio	json_numpy.dumps(array)	Current proprioceptive state (e.g., joint positions).
language_instruction	str	Task instruction (e.g., "Pick up the red block").
image0	json_numpy.dumps(array)	Primary camera image (RGB).
image1, image2	optional	Additional camera views if applicable.
domain_id	int	Identifier for the current robotic embodiment/domain.
steps	int	denoising steps for flow-matching based generation (e.g., 10).

Example Client Code

…

Expected Output

[Server] Model loaded successfully on cuda:0
[Server] Listening on 0.0.0.0:8000
[Client] Sending observation to server...
✅ Received 30 predicted actions.

---

5️⃣ Standardized Control Interface: EE6D

To ensure consistency across embodiments, X-VLA adopts a unified EE6D (End-Effector 6D) control space.

Component	Specification	Notes
Proprio Input	Current EE6D pose (position + orientation)	Must align with training-space normalization.
Action Output	Predicted target delta/absolute pose (EE6D)	Executed by downstream controller.
Dimensionality	20-D vector = 3 (EE Pos) + 6 (Rotation in 6D) + 1 (Gripper) + 10 (Padding)
Single-arm Case	If only one arm exists, pad with zeros to maintain 20D vector.

⚙️ Reference Post-processing:

from datasets.utils import rotate6d_to_xyz
action_final = np.concatenate([
    action_pred[:3],
    rotate6d_to_xyz(action_pred[3:9]),
    np.array([1.0 if action_pred[9] > 0.5 else 0])
])

When feeding proprioception to the model, apply the inverse transformation accordingly.

---

6️⃣ Reference Client Implementations

Each released model includes a corresponding reference client under evaluation///client.py for reproducing exact deployment behaviors. We strongly recommend adapting from these clients when connecting to physical or simulated robots.

---

7️⃣ SLURM & Cluster Deployment

For large-scale or distributed training/deployment (e.g., HPC clusters, AgiBot nodes):

python -m deploy --model_path /path/to/your/model

This script automatically detects SLURM environment variables, launches distributed servers, and writes connection metadata to info.json.

---

⚙️ Training / Fine-tuning on Custom Data

X-VLA supports fine-tuning on new demonstrations via a m