LeRobot Community Roadmap 0.7.0 Release

Author: imstevenpmworkCreated Jun 17, 2026Updated Sep 14, 2026

Labelsbugdocumentationenhancementquestionpoliciesdatasetsimulationtests

LeRobot Community Roadmap 0.7.0

Welcome to the LeRobot Community Roadmap for v0.7.0! This document outlines what the core LeRobot team is actively working on and highlights the most impactful areas where the community can jump in to help.

How to read and use this roadmap:

Team vs. Community: Items prefixed with [Team] are actively being tackled by the internal team. Unmarked items are prime opportunities for community contributions.
High-Impact Contributions: The open community tasks are not just entry-level issues—they are significant, high-impact features. While the core team will eventually address them if needed, your contributions speed up our collective progress tremendously!
Reference Links & Existing PRs: Any linked PRs or issues are provided for context and do not necessarily represent the definitive final implementation. If a task already has an open PR, jumping in to review, test, and validate it provides immense value to the project.
Flexibility: The robotics AI field moves incredibly fast. We reserve the flexibility to pivot and change directions as the landscape evolves.
Priority: The items listed below are not in any specific order of priority.

1. Data Engine: Collect, Curate, and Stream at Scale

Make large, heterogeneous, continuously growing robot datasets reliable and efficient to create, inspect, combine, and train from.

DATA-04 — Resolve video timestamp drift: Converge on one reliable frame-addressing convention. Explicitly decide between integer frame indices and timestamps constrained to exact 1/fps multiples, including the dataset-format compatibility implications.
DATA-06 — Unify feature validation: Establish one source of truth for feature name, shape, data type, and semantic role.
DATA-05 — Fix zero-shaped feature handling: Make zero-shaped features behave consistently across recording, loading, processing, and training.
DATA-15 — Consolidate task representation: Align task handling across recording, task.parquet, and language annotations.
DATA-11 — Investigate synchronized camera concatenation: Determine whether placing synchronized camera streams in fewer video files materially reduces read amplification and whether the result belongs in a future dataset format.
DATA-19 — Refactor the dataset edition tools into a dedicated module and split operations into separate files. Add metadata checks and copy/in-place modifications. Related PRs: #3096, #3136, #3260, #3552, #3617, #3646, #3850, #400, #4244, #4148, #4246.

[Team] DATA-03 — Evaluate ABC/ABCDL: Assess the ABC/ABCDL design and document which ideas LeRobot should adopt. TLDR: views concatenation + faststart + larger GOP with no B frames (smaller files, longer decoding). Faststart is partially implemented in #4549.
[Team] DATA-01 — Design a streaming-first dataset model: Produce an RFC and prototype in which continuous streams are primitive, episodes are derived views, ingestion is append-friendly, initialization has near-zero cost, and metadata is lazy.
[Team] DATA-02 — Add efficient sliced and batched access: Support dataset[start:end] and an efficient __getitems__ path, including batched video decoding. WIP and ready to merge in #4549.
[Team] DATA-13 — Support additional data formats: Open the LeRobot data-loading layer to other formats. Complete the LanceDB exploration and assess MCAP interoperability. LanceDB is complete; MCAP is TBD. #4363
[Team] DATA-18 — Complete the TorchCodec migration: Use TorchCodec where codec support permits it and provide a documented fallback where it does not. Investigate GPU decoding, parallel decoding, full encoding support (streaming, CPU, GPU), and future 12/16-bit depth support.
[Team] DATA-17 — Surface recording corruption: Missing images, buffer overflows, and streaming failures must fail or warn loudly instead of silently contaminating episode video. WIP in #4533.
[Team] DATA-16 — Add task-centric dataset queries: Extend the public dataset API with task-centric querying.
[Team] DATA-14 — Publish dataset validation tools: Provide a standard suite or CLI of checks and repair or inspection scripts for validating dataset correctness. WIP reference.
[Team] DATA-08 — Convert community datasets to LeRobotDataset v3: Convert the selected collection, starting with datasets below 1 TB, and publish references for each conversion. WIP dataset collection.
[Team] DATA-12 — Extend MultiLeRobotDataset: Add weighted sampling, explicit per-dataset key mapping, padding for absent keys, per-robot normalization statistics, aggregate statistics, and per-dataset preprocessing. Allow a list of dataset repository IDs in the training configuration.
[Team] DATA-07 — Make egocentric data first-class: Define the representation for egocentric and human-centric data, add hand-tracking or HaWoR-style annotation support, and document co-training with robot data. Run a controlled comparison of pretraining with and without egocentric data before fine-tuning on a fixed amount of robot data.
[Team] DATA-09 — Build the FineRobotics corpus: Aggregate compatible open datasets into a reproducible large-scale training corpus with clear cleaning and conversion instructions. The candidate pool includes XDOF/ABC-130k, Kai0, Galaxea Open World, InternData-A1, Hy-Embodied, AgiBotWorld, FastUMI, HiFi-UMI, DROID, SO-100/SO-101, and MolmoAct collections. See Julien Blanchon’s work in #hf-community-datasets. Proposal in progress.
[Team] DATA-10 — Run the FineRobotics training experiment: Pretrain at scale, fine-tune on an accessible embodiment, evaluate zero-shot and fine-tuned behavior, and publish the checkpoints and findings.

2. Policy and Training Platform

Make policies easier to integrate, fine-tune, scale, inspect, and move between embodiments.

TRAIN-06 — Standardize encoder freezing: Provide one consistent API for freezing and unfreezing VLM and vision encoders across VLA policies.
TRAIN-07 — Refactor flow-matching policies: Reconcile the different flow-matching conventions and reduce duplicated infrastructure.
TRAIN-12 — Add T-Rex support: Integrate NVIDIA's tactile policy into the common policy and runtime interfaces.
TRAIN-09 — Document ONNX and TensorRT export: Publish and verify a guide for exporting policies to edge deployment targets.
TRAIN-15 — Make our code compatible with other inference-for-robotics libraries, similar to how Transformers is compatible with vLLM and SGLang. In Transformers, layers can be attached and replaced, so ownership for integration sits with the other libraries. First, identify inference engines or core robotics libraries and explore how LeRobot can be made compatible with them (related: RL-09). FlashRT · LoongForge

[Team] TRAIN-01 — Define pretrained-policy contracts: Specify state and image ordering, action-space and embodiment conventions, normalization, and required processors for every pretrained policy.
[Team] TRAIN-02 — Rework feature mapping and pre/post-processing: Make policies depend on semantic feature types rather than repository-specific names. Preserve policy defaults instead of unintentionally overriding rename maps.
[Team] TRAIN-03 — Simplify cross-embodiment fine-tuning: Make it straightforward to fine-tune a pretrained model on a new embodiment or action space through clearer processor contracts.
[Team] TRAIN-08 — Make attention backends configurable: Support consistent selection of attention implementations across relevant models, including Flash Attention, context parallelism, and Hugging Face Kernels optimization where appropriate.
[Team] TRAIN-04 — Expand policy performance features: Extend gradient checkpointing and torch.compile support consistently across policies.
[Team] TRAIN-05 — Expand distributed-training strategies: Add the remaining strategies, including HSDP and context parallelism. Retain gradient accumulation support and verify it across strategies. Gradient accumulation was completed in #4010.
[Team] TRAIN-14 — Define third-party policy integration: Decide and document how external policies are accepted and loaded. Evaluate Hub- or package-based discovery so integrations do not permanently expand the core maintenance surface.
[Team] TRAIN-15 — Maintain a policy candidate list: Track high-impact integrations, including major-lab releases such as Xiaomi Robotics and NVIDIA RoboTTT, while treating well-maintained community contributions fairly.
[Team] TRAIN-13 — Complete Cosmos 3 support: Finish the policy integration and verify its training and rollout workflows. WIP in #3745.
[Team] TRAIN-10 — Improve the model harness: Give model families, including world-action models, clearer shared interfaces and CLI handling. Add tools to visualize predictions and inspect or prompt compatible models.
[Team] TRAIN-11 — Validate zero-shot and fine-tuning workflows: Use the FineRobotics experiment to validate reusable zero-shot and fine-tuning workflows on SO-100/SO-101 or a similarly accessible platform. Waiting for other items.

3. Runtime, Inference, and Autonomy

Provide one safe, understandable runtime for interactive, local, asynchronous, and remote policy execution.

RUN-05 — Make RTC snapshots temporally atomic: Ensure observation and action-queue snapshots describe one coherent instant.
RUN-09 — Establish a minimal runtime safety baseline: Investigate and publish the minimum safety expectations for interactive and remote policy execution.
RUN-02 — Connect high-level planners: Connect the rollout runtime to configurable LLM/VLM planners with documented prompts, permissions, and boundaries.
RUN-08 — Build a navigation and mobile-manipulation reference project: Combine semantic maps, navigation tool calls, a high-level planner, and VLAs in one documented project.
RUN-10 — Define robot disconnection behavior: Decide whether robot hardware should disconnect before the final dataset upload and document the preferred lifecycle.

[Team] RUN-04 — Secure and converge remote inference: Replace unsafe pickle-based transport with SafeTensors plus JSON. Remove behavioral divergence between standard and remote inference, including in simulation.
[Team] RUN-03 — Unify inference modes: Place synchronous, local asynchronous, GPU asynchronous, and remote inference behind the rollout contract. Cover remote teleoperation and larger controllers such as Unitree G1. WIP branch.
[Team] RUN-06 — Integrate training-time RTC and cloud inference: Add RTC to training and document cloud inference for models that cannot run on the robot's local compute for multiple policies. Integrate PiR2 as well. Partly done for π0; ensure the flow-matching strategy is unified across other policies. #4427
[Team] RUN-01 — Expand the interactive rollout CLI: Turn lerobot-rollout --interactive into a polished language runtime with discoverable commands, color, policy loading, questions about the current observation, and natural-language task requests. The language contract PR is also in progress. Demo · #3528
[Team] RUN-07 — Make reward models available at runtime: Score rollouts automatically for checkpoint comparison, data collection, or DAgger-style intervention heuristics.

4. Evaluation, Simulation, and Benchmarks

Make model quality and regressions visible through repeatable, statistically sound evaluation.

EVAL-01 — Define the benchmark leaderboard: Specify supported tasks, default training and evaluation commands, checkpoint rules, statistical methodology, and reporting format. Use Wilson confidence intervals and Fisher's exact test where appropriate.
EVAL-03 — Run nightly policy smoke tests: Evaluate existing checkpoints against an appropriate benchmark, without retraining, to detect regressions in policy code.
EVAL-02 — Build a Hub-native evaluation loop: Upload a policy, run remote evaluation, and publish comparable results to the leaderboard.

[Team] EVAL-04 — Align simulation and real-world execution: Add a simulator. Minimize divergent code paths between simulation and real-world experimentation.
[Team] EVAL-05 — Maintain useful simulation coverage: Continue expanding simulation support, including benchmarks such as RoboCasa and SimplerEnv, subject to a clear maintenance and adoption bar.

5. Reinforcement Learning and Reward Models

Establish a coherent foundation for offline and online learning, with reward models usable throughout the data and runtime loops.

[Team] RL-01 — Split offline and online trainers: Separate imitation/offline training and online RL responsibilities into explicit OfflineTrainer and OnlineTrainer abstractions.
[Team] RL-02 — Complete the second RL architecture pass: Align the IL and RL user experience and define lerobot-rl-actor, lerobot-rl-learner, and lerobot-gym-manipulator surfaces.
[Team] RL-09 — Evaluate reward-model serving engines: Assess vLLM and SGLang for serving reward and world models.
[Team] RL-03 — Integrate additional reward models: Candidates include SOLE-R1, VITA, RoboReward, ReWiND, SARM2, ARM, and WARP-RM.
[Team] RL-04 — Provide a lerobot-reward workflow: Isolate reward components and make them reusable by dataset tools and lerobot-rollout/reward.
[Team] RL-07 — Use reward models for dataset quality: Support annotation, filtering, and quality scoring while retaining failure data when useful for advantage-weighted or preference-based learning. TODO: add the Discord thread.
[Team] RL-08 — Use reward models for online RL: Connect common reward-model interfaces to online-learning workflows.
[Team] RL-05 — Integrate SONIC: Complete the implementation and align it with the common RL interfaces. A blog post is also planned. WIP in #3827, #4267, #4262, and #4264.
[Team] RL-11 — Complete the π0.6 RECAP RL integration: Finish the community contribution, including its distributional value function.
[Team] RL-12 — Prioritize additional RL algorithms: Evaluate RLT, PLD, SPIRAL, DICE-RL, and SARL; integrate only the algorithms that strengthen the common stack.
[Team] RL-10 — Demonstrate distributed online RL: Train with two or three agents, environment-reset integration, DSRL, and at least one reward model. Compare the design with systems such as RLinf.
[Team] RL-06 — Validate the stack on real gripper tasks: Publish a reproducible training and rollout example. WIP in #3984.

6. Hardware, Embodiments, and Sensing

Support larger and more varied robots and modalities without fragmenting collection, training, or runtime APIs.

HW-02 — Add audio recording: Add microphone recording and storage to the dataset pipeline.
HW-07 — Support RealSense JSON configurations: Allow RealSense cameras to load their native JSON configuration files.
Add support for MicroDuck in LeRobot.

[Team] HW-01 — Assess multi-frequency modality recording: Publish a written assessment covering synchronization, storage, and training implications.
[Team] HW-03 — Add tactile-sensor support: Provide a unified interface for image-based tactile signals or 6D point-cloud-style output and connect it to relevant policy experiments.
[Team] HW-04 — Consolidate DM and RS CAN motors behind a MotorBridge adapter, and Feetech and Dynamixel serial servos behind a Rustypot adapter, ensuring both remain interchangeable backends behind a stable LeRobot MotorsBus interface. Rustypot
[Team] HW-06 — Send depth frames over the network: Support depth transport for LeKiwi and ZMQ camera setups, including Unitree G1 scenarios.
[Team] HW-05 — Expand hardware support selectively: Add high-value devices through an explicit integration bar rather than an unbounded core registry.
[Team] HW-09 — Support larger widely used robots: Initially evaluate Franka and WidowX 250s integrations.
[Team] HW-10 — Make LeRobot humanoid-ready: Document the humanoid stack and close the gaps required for humanoid training, including egocentric-data workflows.
[Team] HW-11 — Add support for reBot B601 RS.

7. Platform Architecture, Reliability, and Extensibility

Keep LeRobot understandable and dependable as its datasets, policies, robots, and users grow.

ARCH-05 — Refactor relative actions: Allow relative actions to be enabled independently of policy choice, with correct and understandable statistics.
ARCH-06 — Increase end-to-end test coverage: Shift critical coverage toward end-to-end tests, mock Hub calls in local tests, and restore useful coverage reporting.
ARCH-11 — Define a scalable extension mechanism: Support external robots, cameras, environments, policies, and tools through discoverable third-party packages or Hub assets rather than indefinite maintenance in core.
ARCH-03 — Improve processor documentation: Publish a focused guide and examples, including a large-scale training example.
ARCH-12 — Support agents controlling robots and open models running on Hugging Face infrastructure.

[Team] ARCH-07 — Expand strict typing: Move from 8 to 14 strictly checked modules and replace broad dict[str, Any] interfaces with documented feature and observation types.
[Team] ARCH-08 — Tighten robot and teleoperator typing: Introduce generics or TypedDict-style structures and document feature shapes. get_observation → LeRobotObservation (dict[str, LeRobotFeature]).
[Team] ARCH-04 — Clarify the robot observation API: Give state and image acquisition explicit boundaries such as get_state() and get_images().
[Team] ARCH-01 — Embed pipelines into robots and teleoperators: Let robot and teleoperator objects own the relevant processing, remove aggregate features from policy pipelines, and simplify the ProcessorStep hierarchy.
[Team] ARCH-02 — Specialize processors: Split processor steps into clearer types and move model-adjacent preprocessing, especially tokenizer logic, out of model implementations. WIP in #4487.
[Team] ARCH-09 — Standardize logging: Replace inconsistent print, root-logger, and ad hoc initialization patterns with module-level logging.
[Team] ARCH-12 — Implement and document bring-your-own integrations: Cover external components, training strategies, and experiment-logging integrations, including bring-your-own strategy and logging library.
[Team] ARCH-10 — Support long-running projects in LeLab: Cover registration, collection, filtering, multi-dataset assembly, experiments, training, results, and team workflows.
[Team] ARCH-11 — Migrate to Trackio: Replace wandb.

8. Documentation, Hub Experience, and Community

Make the complete workflow discoverable and keep project growth sustainable.

DOC-03 — Improve LeRobot × Hub workflows: Explore better training with Jobs, native dataset visualization, online inspection or editing tools, and a clearer LeRobot Hub landing page.
DOC-05 — Support documentation translation: Enable community-driven internationalization of the documentation.
DOC-07 — Explore team automation workflows: Create a shared place for team-built automation skills and cautiously evaluate scheduled maintenance workflows such as PR triage or review assistance.
DOC-10 — Support AI documentation: Create a page for AI-driven context.

[Team] DOC-06 — Publish component-inclusion criteria: Balance Hub adoption, project growth, user value, and long-term maintenance when deciding what belongs in LeRobot core.
[Team] DOC-01 — Build public API documentation: Add Google-style docstrings to public APIs, generate reference pages, automatically test documentation examples, publish documentation on the Hugging F

Source: huggingface/lerobot

View original on GitHub View discussion on GitHub