Loan Advisory Agent using Pipecat

Overview

This guide demonstrates how to build a voice-based loan advisory agent that helps customers understand loan products, eligibility, and application processes using Pipecat for real-time communication and Sarvam AI for speech processing. Perfect for banks, NBFCs, fintech companies, and lending platforms serving Indian customers.

For the broader architecture pattern this agent fits into, compliance guardrails, model/param choices, latency targets, see the BFSI Voice Bots use-case guide.

What You’ll Build

A loan advisory agent that can:

Explain different types of loans (personal, home, vehicle, business, education)
Help customers understand eligibility criteria and required documents
Provide information about interest rates, EMIs, and loan tenure
Guide customers through the application process
Answer questions in multiple Indian languages

Quick Overview

Get API keys (Sarvam)
Install packages
Create .env file with your API keys
Write the agent code
Run with appropriate transport

Quick Start

1. Prerequisites

Python 3.9 or higher
API keys from:
- Sarvam AI (get API key from dashboard)

2. Install Dependencies

macOS/Linux

Windows

pip install "pipecat-ai[daily,sarvam]" python-dotenv loguru

3. Create Environment File

Create a file named .env in your project folder and add your API keys:

SARVAM_API_KEY=sk_xxxxxxxxxxxxxxxxxxxxxxxx

Replace the values with your actual API keys.

4. Write Your Agent

Create loan_advisor.py:

import os
from dotenv import load_dotenv
from loguru import logger
from pipecat.frames.frames import LLMRunFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.sarvam.stt import SarvamSTTService
from pipecat.services.sarvam.tts import SarvamTTSService
from pipecat.services.sarvam.llm import SarvamLLMService
from pipecat.transports.base_transport import TransportParams
from pipecat.transports.daily.transport import DailyParams
load_dotenv(override=True)
async def bot(runner_args: RunnerArguments):
    """Main bot entry point."""
    
    # Create transport (supports both Daily and WebRTC)
    transport = await create_transport(
        runner_args,
        {
            "daily": lambda: DailyParams(audio_in_enabled=True, audio_out_enabled=True),
            "webrtc": lambda: TransportParams(
                audio_in_enabled=True, audio_out_enabled=True
            ),
        },
    )
    # Initialize AI services
    stt = SarvamSTTService(
        api_key=os.getenv("SARVAM_API_KEY"),
        language="unknown",  # Auto-detect for diverse customer base
        model="saaras:v4",
        mode="transcribe"
    )
    
    tts = SarvamTTSService(
        api_key=os.getenv("SARVAM_API_KEY"),
        language_code="en-IN",
        model="bulbul:v3",
        speaker="aditya"  # Professional and trustworthy voice
    )
    
    llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)
    # Set up conversation context with loan advisor personality
    messages = [
        {
            "role": "system",
            "content": """You are a professional and helpful loan advisory agent for a leading financial institution in India.
Your expertise covers:
**Loan Products:**
- Personal Loans: Unsecured loans for various personal needs
- Home Loans: For purchasing, constructing, or renovating homes
- Vehicle Loans: Car loans, two-wheeler loans
- Business Loans: For SMEs, startups, and working capital
- Education Loans: For domestic and international education
- Gold Loans: Secured loans against gold jewelry
- Loan Against Property (LAP): Secured loans using property as collateral
**Key Information to Provide:**
- Interest rates (mention that actual rates depend on profile and market conditions)
- Typical loan amounts and tenure options
- Eligibility criteria (age, income, credit score, employment type)
- Required documents (ID proof, address proof, income documents, etc.)
- Processing fees and other charges
- EMI calculation basics
- Prepayment and foreclosure options
**Communication Guidelines:**
- Be professional, trustworthy, and helpful
- Explain financial terms in simple language
- Never guarantee loan approval - mention that final approval depends on detailed assessment
- Always recommend customers to read terms and conditions carefully
- If asked about specific interest rates, provide indicative ranges and mention they vary
- Encourage customers to visit the branch or website for exact current rates
- Be transparent about fees and charges
- Help customers understand their EMI burden before recommending loan amounts
- Suggest customers maintain a good credit score
**Compliance Reminders:**
- Never make false promises about loan approval
- Always mention that loans are subject to eligibility and documentation
- Recommend customers to compare offers before deciding
- Remind about the importance of timely repayments
Start by greeting the customer professionally and asking how you can help them with their loan requirements.""",
        },
    ]
    context = LLMContext(messages)
    context_aggregator = LLMContextAggregatorPair(context)
    # Build pipeline
    pipeline = Pipeline(
        [
            transport.input(),
            stt,
            context_aggregator.user(),
            llm,
            tts,
            transport.output(),
            context_aggregator.assistant(),
        ]
    )
    task = PipelineTask(pipeline)
    @transport.event_handler("on_client_connected")
    async def on_client_connected(transport, client):
        logger.info("Customer connected")
        messages.append(
            {"role": "system", "content": "Greet the customer professionally and ask how you can help them with their loan requirements."}
        )
        await task.queue_frames([LLMRunFrame()])
    @transport.event_handler("on_client_disconnected")
    async def on_client_disconnected(transport, client):
        logger.info("Customer disconnected")
        await task.cancel()
    runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
    await runner.run(task)
if __name__ == "__main__":
    from pipecat.runner.run import main
    main()

5. Run Your Agent

python loan_advisor.py

The agent will create a Daily room and provide you with a URL to join.

6. Test Your Agent

Open the provided Daily room URL in your browser and start speaking. Your loan advisor will listen and respond!

Customization Examples

Example 1: Hindi Loan Advisor

For Hindi-speaking customers:

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language="hi-IN",  # Hindi
    model="saaras:v4",
    mode="transcribe"
)
tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language_code="hi-IN",
    model="bulbul:v3",
    speaker="anand"  # Professional male voice
)
llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Example 2: Tamil Loan Advisor

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language="ta-IN",
    model="saaras:v4",
    mode="transcribe"
)
tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language_code="ta-IN",
    model="bulbul:v3",
    speaker="priya"
)
llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Example 3: Multilingual Advisor (Auto-detect)

For diverse customer bases:

stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language="unknown",  # Auto-detects language
    model="saaras:v4",
    mode="transcribe"
)
tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language_code="en-IN",
    model="bulbul:v3",
    speaker="aditya"
)
llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Example 4: Speech-to-English Advisor (Saaras)

When customers speak regional languages but you need English processing:

# Customer speaks Hindi/Tamil/etc. → Saaras converts to English → LLM processes
stt = SarvamSTTService(
    api_key=os.getenv("SARVAM_API_KEY"),
    model="saaras:v4",  # Speech-to-English translation
    mode="translate"
)
tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language_code="en-IN",
    model="bulbul:v3",
    speaker="aditya"
)
llm = SarvamLLMService(
    api_key=os.getenv("SARVAM_API_KEY"),
    settings=SarvamLLMService.Settings(model="sarvam-105b"),
)

Available Options

Language Codes

Language	Code
English (India)	`en-IN`
Hindi	`hi-IN`
Bengali	`bn-IN`
Tamil	`ta-IN`
Telugu	`te-IN`
Gujarati	`gu-IN`
Kannada	`kn-IN`
Malayalam	`ml-IN`
Marathi	`mr-IN`
Punjabi	`pa-IN`
Odia	`od-IN`
Auto-detect	`unknown`

Speaker Voices (Bulbul v3)

Male (23): Shubh (default), Aditya, Rahul, Rohan, Amit, Dev, Ratan, Varun, Manan, Sumit, Kabir, Aayan, Ashutosh, Advait, Anand, Tarun, Sunny, Mani, Gokul, Vijay, Mohit, Rehan, Soham

Female (14): Ritu, Priya, Neha, Pooja, Simran, Kavya, Ishita, Shreya, Roopa, Tanya, Shruti, Suhani, Kavitha, Rupali

TTS Additional Parameters

Customize the voice for professional financial advisory:

tts = SarvamTTSService(
    api_key=os.getenv("SARVAM_API_KEY"),
    language_code="en-IN",
    model="bulbul:v3",
    speaker="aditya",
    pace=1.0,            # Normal pace for clear communication
    speech_sample_rate=24000  # 8000, 16000, 22050, 24000 Hz (default). v3 REST API also supports 32000, 44100, 48000 Hz
)

Understanding the Pipeline

Pipecat uses a pipeline architecture where data flows through a series of processors:

Customer Audio → STT → Context Aggregator → LLM → TTS → Audio Output

Transport Input: Receives audio from the customer
STT (Speech-to-Text): Converts audio to text using Sarvam’s Saaras v4 (transcription via mode="transcribe", or translation to English via mode="translate")
Context Aggregator (User): Adds customer’s query to conversation context
LLM: Generates advisory response using Sarvam
TTS (Text-to-Speech): Converts response to audio using Sarvam’s Bulbul
Transport Output: Sends audio back to the customer
Context Aggregator (Assistant): Saves advisor’s response to context

Pro Tips

Use language="unknown" to support customers who code-mix (Hinglish, etc.)
Use professional voices like aditya for financial services
Sarvam’s models handle code-mixing naturally
Always maintain compliance - never guarantee loan approvals
Consider integrating with your loan management system for real-time information

Troubleshooting

API key errors: Check that all keys are in your .env file and the file is in the same directory as your script.

Module not found: Run the installation command again based on your operating system.

Poor transcription: Try language="unknown" for auto-detection, or specify the correct language code.

Connection issues: Ensure you have a stable internet connection and the transport is properly configured.

Additional Resources

Need Help?

Sarvam Support: support@sarvam.ai
Community: Join the Discord Community

Happy Building!