[Feature] Unified audio input and transcription workflow for Chat/Agent
Author: DracieVajkoCreated Sep 16, 2026Updated Sep 16, 2026
Labelsenhancementready for review
Prerequisites
- I searched open issues and this has not already been proposed.
- I searched discussions and this is not already being debated there.
- This is a concrete, actionable proposal — not a vague "it would be nice if..." request.
Area
Chat / Agent
Problem or Motivation
Odysseus already contains several pieces of speech-to-text and audio functionality, including local faster-whisper support, microphone recording, an STT API endpoint, and audio file uploads.
However, these features are currently disconnected.
The existing microphone path can record audio and insert a transcription into the chat composer, but the microphone control is tied to the existing Send button behaviour and is not a clearly discoverable standalone audio workflow.
Audio files can already be uploaded as chat attachments, but uploaded audio is currently passed through the multimodal attachment path rather than through the existing STT service. This means a recording from a phone, for example a meeting, lecture, interview, or personal note, does not have a unified "transcribe and continue working with this" workflow for text-oriented Chat/Agent models.
There is also no visible STT settings workflow for selecting local Whisper models or configuring transcription language, even though the backend already has stt_model and stt_language settings.
A practical self-hosted use case is:
phone recording -> upload recording to Odysseus -> local Whisper transcription -> preserve the raw transcript -> optionally send it to the current Chat/Agent model -> optionally generate a cleaned transcript, summary, or structured notes
This would make existing STT and attachment infrastructure significantly more useful without requiring users to manually convert recordings into text outside Odysseus.
Proposed Solution
Add a unified audio input and transcription workflow built on top of the existing STT and attachment infrastructure.
1. Microphone input in the chat composer
Add a dedicated microphone / speech-to-text button to the chat composer instead of reusing or morphing the Send button.
The composer should keep the existing Send action as a separate control.
Conceptually:
+----------------------------------------------------------------------------+
| [expand] [search] [terminal] [attach] [microphone] [Agent/Chat] [↑] |
+----------------------------------------------------------------------------+
The existing microphone recording implementation should be reused rather than creating a second recording system.
Expected flow:
Microphone
|
v
Record audio
|
v
Local / configured STT provider
|
v
Transcript
|
+--> Insert into composer
|
+--> Send to current Chat/Agent
The user should be able to review the transcription before sending it, while also having an option to send it directly.
2. Audio file -> transcription
Allow existing audio recordings to use the same STT pipeline.
Initial formats to investigate/support:
WAV
MP3
M4A
WEBM
OGG
The exact supported formats should be determined by the actual decoding/STT stack rather than by filename extension alone.
Example:
audio.m4a
|
v
Upload
|
v
STTService
|
v
Transcript
|
+--> Insert into composer
|
+--> Send to Chat/Agent
|
+--> Process with LLM
This is particularly useful for recordings created on phones and processed later on a self-hosted Odysseus instance.
3. Whisper/STT model selection
Expose local STT model configuration through a visible Settings UI.
The current backend already supports an stt_model setting, but there is currently no dedicated STT model catalog/download/status workflow.
The UI should make it possible to:
see available/recommended local STT models;
see which models are installed or cached;
select the active model;
download models when required;
show model/download status where practical.
The design should support different model sizes and make their resource requirements clear.
It should be possible to use larger Whisper variants where the hardware can reasonably support them, but the UI should not assume that every model is suitable for every device.
4. Language selection and multilingual transcription
Expose the existing STT language configuration in Settings.
Desired behaviour:
Auto-detect language;
explicitly select a language when known;
support multilingual transcription;
include Slovak, Czech and English as normal use cases.
The detected language should be retained where the STT backend provides that information.
5. Preserve the raw transcript
The original STT output should remain available as a raw transcript.
The raw transcription should not be silently replaced by an LLM rewrite.
This provides a source representation of what the transcription engine actually produced and allows later processing without losing the original result.
6. Optional LLM post-processing
After transcription, provide an optional second processing stage using an LLM.
Possible operations include:
Clean transcript
Readable transcript
Summary
Structured notes
Key points
Action items
Dates
Decisions
People / entities
User-defined processing instructions
The normal Odysseus model-routing/task/utility infrastructure should be reused where appropriate.
The user should also be able to send the transcript into the currently selected Chat/Agent flow so the transcript can be handled using the user's normal model, instructions and tools.
A possible high-level workflow:
+----------------+
| Audio input |
| mic / file |
+-------+--------+
|
v
+----------------+
| STT / Whisper |
+-------+--------+
|
v
+----------------+
| Raw transcript |
+---+--------+---+
| |
+----------+ +--------------------+
| |
v v
+---------------------+ +------------------------+
| Current Chat/Agent | | Optional LLM processing|
+---------------------+ +-----------+------------+
|
+-----------+-----------+
| | |
v v v
Clean text Summary Structured data
7. Audio attachment actions
When an audio file is attached, provide an explicit transcription action rather than treating the audio only as opaque multimodal content.
Possible actions:
Transcribe
Transcribe and insert
Transcribe and send
Transcribe and process
For audio-capable models, the existing direct-audio attachment path may remain available as an alternative.
8. Long recordings and limits
The existing STT endpoint has a dedicated audio size limit.
The new workflow should account for recordings substantially longer than normal chat messages, such as meetings and lectures.
The normal chat attachment limit and STT-specific processing limit should not be coupled accidentally.
Where necessary, the implementation should define how larger recordings are handled.
9. Privacy and security
When local STT is selected, the original audio should remain on the Odysseus host.
When a remote STT endpoint is selected, the user should be able to understand that the audio is sent to that configured external service.
Existing authentication, upload ownership, attachment security and untrusted-input rules should remain in effect.
10. Reuse existing infrastructure
The implementation should build on existing components where possible, including:
services/stt/stt_service.py
routes/stt_routes.py
static/js/voiceRecorder.js
existing composer/microphone handling in static/app.js
existing upload and attachment handling
existing audio attachment handling
existing model download/status patterns
existing task/model routing
existing prompt and summarization infrastructure
The goal is to connect and extend existing capabilities rather than introduce a parallel STT stack.
A **toto by som ešte mierne upravil v Alternatives**, aby to neznelo, že externé STT je zlé — iba že má iné trade-offs:
```md
### Alternatives Considered
#### Keep the existing microphone-only workflow
This works for short voice input, but it does not address existing recordings created outside the browser, which is an important use case for meetings, lectures, interviews and mobile recordings.
#### Upload audio as a normal attachment and rely on the LLM
Odysseus can already pass audio attachments through the multimodal path, but this is not a general solution for text-oriented models and does not provide a consistent local transcription workflow.
#### Use an external transcription service
This could simplify model management, but introduces external data transfer, provider dependency and potentially additional cost. Since Odysseus already supports local `faster-whisper`, local transcription should remain a first-class option rather than the only option.
#### Build a separate audio application/workflow
This would duplicate existing recorder, upload, STT, attachment and model-routing infrastructure. Extending the current components keeps the feature closer to the existing architecture.
#### Make this only a hands-free voice mode
Hands-free voice interaction is already covered by related work such as #4118. This proposal focuses on the broader audio-input and transcription workflow, especially existing audio files and transcript processing, while leaving continuous voice conversation as a separate concern.
### Alternatives Considered
_No response_
### Prior Art / Related Issues
Related work already exists in the repository/community and should be considered rather than duplicated:
- #4118 - Hands-free voice mode — continuous STT -> chat -> TTS loop
- #3554 - Discussion about configuring local `faster-whisper` STT
- #2238 - Discussion about Whisper/STT discoverability
- #3680 - Discussion about missing Speech-to-Text settings
- #2801 - Discussion about alternative local STT engines such as Moonshine
- Existing Ideas discussion: "Voice-to-Action Pipeline & Adjustable Timeouts for Local Hardware"
The proposed feature is intentionally broader than the hands-free voice-mode issue. It combines microphone input, existing audio-file transcription, multilingual/local STT configuration, raw transcript preservation and optional LLM post-processing into one user-facing audio workflow.
The current codebase already contains much of the underlying infrastructure, including `STTService`, `/api/stt/transcribe`, microphone recording, audio attachments, model/task routing and existing model download patterns.
### Are you willing to implement this?
Yes — I can open a PR
Source: odysseus-dev/odysseus