Over the last 10 days, I built FinSaathi, a voice-first AI assistant for the Financial Services track of the VoiceForBharat challenge.
The goal was simple: build an assistant that can talk naturally with users, understand financial and government-scheme related queries, remember relevant information, use tools, and know when a human or specialist should take over.
What started as a basic voice agent gradually became a complete system with memory, tools, outbound calling, human escalation, call analytics, and specialist-agent handoffs. ๐ก The Problem Financial and government-scheme processes can involve eligibility requirements, documents, deadlines, and complicated terminology.
For users who are more comfortable speaking than typing, voice can make these interactions much more natural.
For example, a user can simply ask: "PMJJBY ke liye main eligible hoon?" Instead of navigating through multiple forms, FinSaathi can understand the request, collect the required information, perform an eligibility check, and explain the result conversationally.
The goal is not to replace banks or human support, but to provide a conversational first layer of assistance and escalate situations when human help is required. ๐๏ธ Architecture USER โ โผ LiveKit โ โผ Speech-to-Text โ โผ LLM / Agent โ โโโโโโโโโโโโโโผโโโโโโโโโโโโโ โผ โผ โผ Memory Tools Escalation โ โ โ โโโโโโโโโโโโโโผโโโโโโโโโโโโโ โผ SQLite DB โ โโโโโโโโดโโโโโโโ โผ โผ Human Support Analytics Dashboard Dashboard โ โผ Murf Falcon โ โผ USER Technology Stack Component Technology Frontend Next.js / React AI Agent LiveKit Agents Real-time Transport LiveKit Text-to-Speech Murf Falcon Backend Python API FastAPI Database SQLite Calling SIP / LiveKit ๐๏ธ Key Features Indian Voice & Natural Conversations FinSaathi uses Murf Falcon for text-to-speech and supports natural Hindi/Hinglish conversations.
The goal was to make the interaction feel more like talking to an assistant rather than interacting with a traditional chatbot.
Safety Guardrails Financial conversations require strong safety boundaries.
For example, when a user reports an unauthorized transaction, FinSaathi warns them not to share: OTP PIN Password CVV Card details The agent also clearly explains when it does not have direct access to a user's bank account or transaction system.
User Memory FinSaathi stores relevant user information in a SQLite database so it can be reused in future conversations.
The database logic is separated into its own layer: Agent โ database.py โ SQLite โ users table This made it easier to extend the project without mixing database operations throughout the agent code.
Government Scheme Eligibility FinSaathi can use tools to perform government-scheme eligibility checks.
User asks about scheme โ Collect required information โ Eligibility tool โ Eligible / Not Eligible โ Database โ User receives result The eligibility result can also be used to determine whether the user's intended task was successfully completed.
Outbound Calling FinSaathi can proactively contact eligible users using outbound calling.
Database โ Check deadlines / eligible users โ Outbound calling logic โ SIP / LiveKit โ AI Agent โ User Setting up and debugging the SIP and LiveKit calling flow was one of the most challenging parts of the project.
Human Escalation FinSaathi can recognize situations where human support is more appropriate.
For example, for an unauthorized transaction, the agent can ask: "Agar aap chahein, toh main aapki situation ke liye ek human support request create kar sakti hoon." After the user gives consent, the system creates an escalation and generates a unique reference ID.
Example: FS-A5323F The request is stored with information such as: Reason: possible_fraud Urgency: HIGH Status: OPEN Preferred Follow-up: phone The human-support dashboard then allows the request to move through: OPEN โ IN_PROGRESS โ RESOLVED Call Analytics FinSaathi also records the outcome of conversations.
For this project, a successful interaction means that the u