Support Voice Notes

Overview

This cookbook builds a WhatsApp support bot whose spoken replies sound like a specific agent, not a stock speaker. It clones that agent once from a short reference clip, then on every inbound message it transcribes with Saaras, drafts a reply with a Sarvam chat model, and speaks the reply with Instant Voice Cloning.

The WhatsApp Bot guide is the same channel with a Bulbul speaker. Use this page when the reply has to be a named person’s voice.

What you’ll build:

• A saved cloned voice from a 10-15 second support-agent clip
• A Flask webhook that receives WhatsApp text and voice notes via Twilio
• Automatic transcription of inbound voice notes
• A short support reply from a Sarvam chat model
• A spoken reply in the cloned agent voice, attached as WhatsApp audio

Business Value

• Keep one familiar agent voice on a channel customers already have open
• Reply in Hindi, Tamil, or any supported language from a single English (or other) reference clip
• Avoid hiring per-language voice talent for routine status and FAQ replies

How It Works

1. Prerequisites

• Python 3.9 or higher
• sarvamai ≥ 0.1.37
• A Sarvam API key from Key Management
• A 10-15 second, single-speaker clip of the support agent, saved as reference.wav. See Prepare Reference Audio.
• A Twilio account with the WhatsApp Sandbox (or a WhatsApp-enabled Sender)
• ngrok to expose the local server

2. Install the SDK and Dependencies

pip install "sarvamai>=0.1.37" flask twilio python-dotenv requests

3. Set Environment Variables

export SARVAM_API_KEY="your-sarvam-api-key"
export TWILIO_ACCOUNT_SID="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export TWILIO_AUTH_TOKEN="your-twilio-auth-token"
# Optional: reuse a voice created earlier instead of uploading the clip on every start
# export VOICE_ID="svc-your-voice-id"

4. The Full Bot Script

Copy this into support_voice_notes.py. It creates (or reuses) the agent voice, then handles every inbound WhatsApp message.

import base64
import logging
import os
import uuid
from pathlib import Path
import requests
from dotenv import load_dotenv
from flask import Flask, Response, request, send_from_directory
from sarvamai import SarvamAI
from twilio.twiml.messaging_response import MessagingResponse
load_dotenv(override=True)
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
app = Flask(__name__)
sarvam = SarvamAI(api_subscription_key=os.environ["SARVAM_API_KEY"])
AUDIO_DIR = Path("audio_replies")
AUDIO_DIR.mkdir(exist_ok=True)
SYSTEM_PROMPT = (
    "You are a support agent for a consumer app, chatting over WhatsApp. "
    "Reply in the customer's language. Keep replies to 2-3 sentences."
)
def resolve_voice_id() -> str:
    existing = os.getenv("VOICE_ID")
    if existing:
        return existing
    with open("reference.wav", "rb") as f:
        created = sarvam.voice_cloning.create_voice(
            file=f,
            name="support-agent",
            language="en-IN",
        )
    voice_id = created.data.voice_id
    logger.info("Created cloned voice %s — set VOICE_ID to reuse it", voice_id)
    return voice_id
VOICE_ID = resolve_voice_id()
@app.route("/audio/<filename>")
def serve_audio(filename):
    return send_from_directory(AUDIO_DIR, filename, mimetype="audio/wav")
@app.route("/whatsapp", methods=["POST"])
def whatsapp_webhook():
    num_media = int(request.form.get("NumMedia", 0))
    twiml = MessagingResponse()
    try:
        if num_media > 0 and "audio" in request.form.get("MediaContentType0", ""):
            media_url = request.form["MediaUrl0"]
            media_response = requests.get(
                media_url,
                auth=(os.getenv("TWILIO_ACCOUNT_SID"), os.getenv("TWILIO_AUTH_TOKEN")),
                timeout=10,
            )
            media_response.raise_for_status()
            audio_file = Path("/tmp/voice_note.ogg")
            audio_file.write_bytes(media_response.content)
            transcript = sarvam.speech_to_text.transcribe(
                file=open(audio_file, "rb"),
                model="saaras:v4",
                language_code="unknown",
            )
            user_text = transcript.transcript
        else:
            user_text = request.form.get("Body", "")
        completion = sarvam.chat.completions(
            model="sarvam-105b",
            messages=[
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": user_text},
            ],
        )
        reply_text = completion.choices[0].message.content
    except Exception:
        logger.exception("Failed to process incoming WhatsApp message")
        reply_text = "Sorry, I couldn't process that. Please try again in a moment."
    message = twiml.message(reply_text)
    try:
        spoken = sarvam.voice_cloning.convert(
            voice_id=VOICE_ID,
            text=reply_text,
        )
        filename = f"{uuid.uuid4().hex}.wav"
        (AUDIO_DIR / filename).write_bytes(
            base64.b64decode(spoken.audio_b64 or spoken.audio)
        )
        message.media(f"{request.host_url}audio/{filename}")
    except Exception:
        logger.exception("Cloned-voice synthesis failed, sending text-only reply")
    return Response(str(twiml), mimetype="application/xml")
if __name__ == "__main__":
    app.run(port=7861)

How it works

• resolve_voice_id clones the agent from reference.wav once, unless VOICE_ID is already set. Create is not on the per-message path.
• Inbound voice notes download from Twilio and go to Saaras with language_code="unknown", so a Hindi note and a Tamil note work without a code change.
• sarvam-105b drafts a short reply in the customer’s language.
• voice_cloning.convert speaks that reply with the saved agent voice. Synthesis has its own try/except, so a cloning failure still leaves the customer with the text reply.
• language_code is omitted on convert so the service detects the language from the reply. Pass the code when pronunciation has to be exact — see Supported languages.

5. Run It

Put reference.wav next to the script, point the Twilio Sandbox webhook at https://<ngrok-host>/whatsapp, then:

python support_voice_notes.py

ngrok http 7861

Send a text or a voice note to the Sandbox number. You should get a text reply and an audio attachment in the cloned agent’s voice.

6. Tips & Best Practices

• Reference clip: 10-15 seconds, one speaker, no music. The clone copies pace and emotion, not only timbre. See Prepare Reference Audio.
• Create once: Reuse VOICE_ID. Re-uploading the clip on every process start repeats transcription and can hit the free-plan voice cap.
• Reply length: Keep the system prompt explicit. WhatsApp bodies are capped at 1600 characters, and short replies clone more cleanly.
• One-shot vs stream: WhatsApp needs a finished file to attach, so this guide uses convert. For a live voice agent, use HTTP streaming or WebSocket.
• Sandbox opt-in: Recipients must send the Sandbox join code first. See WhatsApp Bot.

7. Error Handling

Retry 429, 500, 502, 503, and 504 with exponential backoff. Do not retry 400, 401, 402, 403, 404, or 422. Log request_id from failed Sarvam responses.

You may also see:

• 400 on clone — both voice_id and ref_audio were sent, the clip is too long, or the reply is romanized Indic / Urdu. Keep native script in the chat prompt.
• 402 — the subscription cannot generate this request.
• 403 — invalid API key.
• Twilio 63015 — the sender has not joined the Sandbox.

For the full table, see Errors & Troubleshooting and the Instant Voice Cloning FAQ.

8. Additional Resources

• API overview: Instant Voice Cloning
• Same channel, stock voice: WhatsApp Bot
• How-tos: Prepare Reference Audio · Clone Across Languages · HTTP Streaming
• Related APIs: Speech-to-Text · Chat Completion
• Twilio: Sandbox for WhatsApp
• Community: Join the Discord Community

9. Final Notes

• Keep API keys in environment variables.
• Only clone a voice you have the right to use.
• Create the agent voice once, then reuse voice_id on every reply.

Keep Building! 🚀