From a Simple Voice Conversation to a Multi-Capability Healthcare Voice Agent Over the past 10 days, I had the opportunity to participate in 10 Days of Voice Agents — VoiceForBharat Edition, a challenge focused on learning how to build practical, real-world voice AI agents.
Instead of treating the challenge as just a series of coding tasks, I wanted to build something around a problem that genuinely matters: making healthcare access more conversational and accessible through voice.
That idea became AarogyaMitra — a voice-first healthcare access assistant designed to interact with users naturally, provide useful assistance, use tools when required, remember relevant user context, and involve humans or specialist agents when the situation requires it.
This article documents my journey, the architecture behind the project, the important features I built, the challenges I faced, and what I learned while developing a real-time voice AI system.
What is AarogyaMitra?
AarogyaMitra is a voice AI assistant focused on the Health Access track of the VoiceForBharat challenge.
The goal is simple: Make healthcare assistance more accessible through natural voice conversations.
Many digital healthcare experiences assume that users are comfortable reading, typing, navigating menus, and interacting with conventional applications.
Voice can provide a more natural alternative.
Instead of searching through menus or typing a question, a user can simply speak to the assistant and have a conversation.
AarogyaMitra is designed around this idea.
The core objectives are: Make healthcare-related interactions more conversational Provide a simple voice-first interface Use AI tools when additional information or actions are required Maintain useful context during conversations Follow safety-oriented guardrails Escalate situations that require human assistance Route specialized requests to a specialist agent AarogyaMitra is intended to assist users, not replace qualified healthcare professionals.
Safety and appropriate escalation are therefore important parts of the system.
Why Voice AI for Healthcare Access?
Voice changes the way users interact with technology.
A traditional application might require: Open app → Find feature → Type query → Read result A voice assistant can reduce that interaction to: Speak → Listen → Respond This can be particularly useful when users prefer speaking over typing or when navigating a conventional interface is inconvenient.
For healthcare access, conversational interaction can also make the experience feel more approachable.
However, healthcare is a sensitive domain.
A voice agent should not simply answer everything confidently.
That is why AarogyaMitra was designed with guardrails, escalation paths, and specialist handoffs rather than treating the AI as an unrestricted chatbot.
What I Built During the 10-Day Challenge The biggest lesson from this challenge was that a useful voice agent is much more than speech input and speech output.
Over the 10 days, I gradually added capabilities that transformed the initial agent into a more complete voice AI system.
1.
Natural Voice Conversations The foundation was creating an agent capable of having a real-time conversation with the user.
The basic interaction follows: This creates a conversational loop rather than a traditional request-response interface.
2.
Indian Voice with Murf Falcon One of the important parts of the project was giving the agent a natural voice.
For text-to-speech, I used Murf Falcon, the fastest TTS API highlighted by the challenge.
The goal was not simply to convert text into audio, but to make the interaction feel more natural and suitable for a voice-first application.
The voice layer is particularly important because even a technically powerful AI system can feel frustrating if responses are slow, unnatural, or difficult to understand.
3.
Personality and Safety Guardrails An AI agent needs more than a model.
It needs clear instructions defining: Its role What it should help with How it should communicate What it should avoid When it should ask for clarification When it should involve a human For a healthcare-oriented agent, these boundaries are especially important.
AarogyaMitra is designed to provide assistance and information while avoiding the assumption that an AI system should independently make high-stakes medical decisions.
This was one of the important lessons from the challenge: A good AI agent is not only defined by what it can do, but also by what it knows it should not do.
4.
Memory for Better Conversations Another major step was adding memory.
Without memory, every interaction can feel disconnected.
With memory, the system can maintain relevant information about returning users and provide a more continuous experience.
The concept can be represented as: Memory makes the agent feel less like a collection of isolated questions and more like an ongoing assistant.
At the same time, memory must be handled carefully, especially for sensitive information.
5.
Tools: Giving the Agent the Ability to Do More A language model alone is limited to generating responses.
Tools allow an agent to interact with external functionality.
Instead of asking the model to invent an answer, the agent can use a tool when the required information or action belongs outside the model itself.
Conceptually: This was an important transition in my project because the agent became capable of doing more than simply having a conversation.
6.
Human Escalation One of the most important lessons I learned was that AI should not always try to handle everything itself.
Some situations require human assistance.
AarogyaMitra therefore includes a path for escalating conversations when the AI should not continue independently.
The basic flow is: This is particularly important in sensitive domains such as healthcare.
A strong AI system should know when to say: "This is something where human assistance would be more appropriate."
7.
Outbound Voice Calls The challenge also introduced outbound calling.
This expanded the project beyond a user simply opening a webpage and talking to an agent.
The system can be used as part of an outbound voice workflow, creating another way for an AI agent to interact with users.
This introduced a completely different set of considerations around: Call initiation Voice interaction User experience Call outcomes Reliability Privacy Building this made me realize that voice AI becomes significantly more complex when it moves from a controlled browser environment into real-world communication workflows.
8.
Call Outcome Tracking and Analytics A production-oriented voice system should not only make calls.
It should also help us understand what happened during those calls.
Tracking outcomes provides useful information about: Successful interactions Failed calls Escalations Conversation outcomes Overall system behavior This creates a feedback loop: This was another important shift in my thinking: building an AI agent is only one part of the problem; understanding how it performs is equally important.
9.
Specialist Agent Handoff One of my favorite features from the final stages of the challenge was introducing a specialist agent.
The idea is simple: One agent does not need to be an expert at everything.
Instead of making the main agent responsible for every possible healthcare-related task, a specialized agent can take over when a request requires a narrower area of expertise.
The flow becomes: The important part is that the user should not have to repeat their entire problem.
The main agent can announce the transition, and the specialist can continue using the relevant conversation context.
This introduced me to an important concept in agentic AI: Specialization can make an AI system more manageable, rather than trying to make one agent responsible for everything.
How the Complete System Fits Together After combining the different capabilities, AarogyaMitra can be viewed as a collection of intercon