Architectures for complex task orchestration, sub-agents, and tool use with Moshi

Due diligence

• I have done my due diligence in trying to find the answer myself.

Topic

The paper

Question

Hi Kyutai team,

Moshi is trained end-to-end over interleaved text/audio streams and is particularly strong at low-latency full-duplex interaction. I am interested in extending this paradigm from conversational response generation to more complex agentic tasks—multi-step planning, tool/function calls, and delegating subtasks to specialized sub-agents—without losing streaming interactivity.

I found #84 (function calling) and #76 (custom prompts/RAG). Those discussions mainly focus on prompting and basic function calling; my question is about the recommended system architecture and training path for more complex orchestration.

Which direction do you consider most promising?

1. Native end-to-end tool use: represent tool calls and tool results as structured tokens/events, then fine-tune them jointly with the text/audio streams.
2. Dual-model architecture: keep Moshi as the real-time speech interface, while a separate reasoning/orchestrator LLM performs planning and tool calls, feeding actions or results back through an asynchronous side channel (similar in spirit to MoshiRAG).
3. Hierarchical delegation: let Moshi decide when to delegate to specialized sub-agents, while it continues handling acknowledgements, backchannels, pauses, and user interruptions during long-running tasks.
4. Post-training: use SFT, LoRA, or RL to learn when to call tools, wait, speak, or cancel/replan.

A few concrete questions:

• Does the current Moshi architecture/codebase expose a stable text or control channel through which planner/tool results can be injected during streaming decoding?
• How would you train latency-aware conversational behavior when a tool or sub-agent may take several seconds to return?
• Do you think complex planning should live inside the speech-native model, or should it remain decoupled from the real-time speech model?
• Are tool use or agent orchestration capabilities on the Moshi roadmap?

Thanks for any guidance or pointers to related work.