WhatsApp Bot

Overview

This cookbook builds a WhatsApp bot that replies to inbound text messages and voice notes on a WhatsApp number. It chains together three Sarvam AI APIs, Saaras STT to transcribe voice notes, a Sarvam chat model to generate a reply, and Bulbul TTS to optionally speak that reply back, behind a small Flask webhook.

What you’ll build:

• A Flask webhook that receives inbound WhatsApp messages, text or voice notes
• Automatic transcription of voice notes, with auto-detection of whichever language the sender uses
• A conversational reply generated by a Sarvam chat model
• An optional spoken voice-note reply, synthesized with Bulbul TTS
• Graceful fallback replies if a download or API call fails, so the sender never gets silence

Business Value

• Offer support on a channel most users already have open, no app install required
• Let users speak instead of type, useful for anyone more comfortable in a regional language than in written English
• Handle common questions in 11 languages without hiring per-language support staff
• Log every transcript for later analytics or QA review

How It Works

1. Prerequisites

• Python 3.9 or higher
• A Sarvam API key, sign up on the Sarvam AI Dashboard to get one
• A Twilio account, with an Account SID, an Auth Token, and access to the WhatsApp Sandbox (or a WhatsApp-enabled Sender for production)
• ngrok, to expose your local server during development

2. Install the SDK and Dependencies

$
pip install flask twilio sarvamai python-dotenv requests

3. Configure Twilio for WhatsApp

Set your keys as environment variables:

$
export SARVAM_API_KEY="your-sarvam-api-key"
$
export TWILIO_ACCOUNT_SID="ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$
export TWILIO_AUTH_TOKEN="your-twilio-auth-token"

Twilio’s Sandbox is a shared number (+1 XXX XXX XXXX, shown on the console page below), used by every developer testing WhatsApp. Twilio only forwards messages from phone numbers that have explicitly opted in to your sandbox, so before writing any code, connect your own number:

1. Open the Twilio Console and go to Messaging → Try it out → Send a WhatsApp message (or search for “WhatsApp sandbox” in the console search bar). This page shows the sandbox number and your account’s unique join code, for example join happy-elephant.
2. From the WhatsApp app on your phone, send that exact join <your-code> message to the sandbox number shown on that page (or scan the QR code Twilio shows, which pre-fills this message for you).
3. Twilio replies confirming you’re connected.

Once your webhook is running (Step 4 below) and exposed via ngrok, come back to the Sandbox settings page, find When a message comes in, and set it to your ngrok URL plus /whatsapp (for example, https://xxxx.ngrok-free.app/whatsapp), method HTTP POST.

4. The Full Bot Script

Everything below, imports, the Flask webhook, and the transcribe → chat → reply flow, lives in one script. Copy it into whatsapp_bot.py:

1
import logging
2
import os
3
from pathlib import Path
4
5
import requests
6
from dotenv import load_dotenv
7
from flask import Flask, Response, request
8
from sarvamai import SarvamAI
9
from twilio.twiml.messaging_response import MessagingResponse
10
11
load_dotenv(override=True)
12
13
logging.basicConfig(level=logging.INFO)
14
logger = logging.getLogger(__name__)
15
16
app = Flask(__name__)
17
sarvam = SarvamAI(api_subscription_key=os.getenv("SARVAM_API_KEY"))
18
19
SYSTEM_PROMPT = (
20
    "You are Sarvam AI Agent, a multilingual assistant powered by Sarvam AI, "
21
    "chatting over WhatsApp. You can understand and respond fluently in "
22
    "English and 10+ Indian languages. Keep replies short (2-3 sentences) "
23
    "and conversational."
24
)
25
26
27
@app.route("/whatsapp", methods=["POST"])
28
def whatsapp_webhook():
29
    num_media = int(request.form.get("NumMedia", 0))
30
    twiml = MessagingResponse()
31
32
    try:
33
        if num_media > 0 and "audio" in request.form.get("MediaContentType0", ""):
34
            # Sender sent a voice note - download and transcribe it.
35
            # Twilio media URLs require your account credentials to fetch.
36
            media_url = request.form["MediaUrl0"]
37
            media_response = requests.get(
38
                media_url,
39
                auth=(os.getenv("TWILIO_ACCOUNT_SID"), os.getenv("TWILIO_AUTH_TOKEN")),
40
                timeout=10,
41
            )
42
            media_response.raise_for_status()
43
44
            audio_file = Path("/tmp/voice_note.ogg")
45
            audio_file.write_bytes(media_response.content)
46
47
            transcript = sarvam.speech_to_text.transcribe(
48
                file=open(audio_file, "rb"),
49
                model="saaras:v3",
50
                language_code="unknown",  # auto-detects the sender's language
51
            )
52
            user_text = transcript.transcript
53
        else:
54
            user_text = request.form.get("Body", "")
55
56
        completion = sarvam.chat.completions(
57
            model="sarvam-105b",
58
            messages=[
59
                {"role": "system", "content": SYSTEM_PROMPT},
60
                {"role": "user", "content": user_text},
61
            ],
62
        )
63
        reply_text = completion.choices[0].message.content
64
    except Exception:
65
        # Network hiccups, a failed download, or an API error shouldn't crash
66
        # the webhook - Twilio would just get a 500 and the sender gets silence.
67
        logger.exception("Failed to process incoming WhatsApp message")
68
        reply_text = "Sorry, I couldn't process that. Please try again in a moment."
69
70
    twiml.message(reply_text)
71
    return Response(str(twiml), mimetype="application/xml")
72
73
74
if __name__ == "__main__":
75
    app.run(port=7861)

How it works

• whatsapp_webhook is the single entry point Twilio calls for every inbound message. NumMedia and MediaContentType0 tell you whether the message carries a voice note; Body holds typed text.
• If a voice note is present, it’s downloaded with HTTP Basic Auth (Twilio media URLs require your Account SID and Auth Token) and handed straight to Saaras v3 with language_code="unknown", so the bot works for a Hindi caller followed by a Tamil caller without any code changes.
• The (transcribed or typed) text is sent to sarvam-105b along with a system prompt that sets the bot’s persona and instructs it to keep replies short, since WhatsApp is a chat surface, not a document.
• The try/except around the whole body exists because this endpoint is a system boundary, it calls two external services (Twilio’s media download, Sarvam’s API) over the network, either of which can fail independently of your code. Replying with a short apology keeps the conversation alive instead of leaving the sender with no response at all.

Run it, then expose it with ngrok:

$
python whatsapp_bot.py

$
ngrok http 7861

Send a text message, or a voice note, to your Twilio WhatsApp number from any phone. Your bot will transcribe and reply within a few seconds.

5. Reply with a Voice Note (Optional)

To have your bot speak its reply back instead of (or alongside) text, synthesize it with Bulbul TTS, serve the audio file from your own Flask app, and attach it to the TwiML response as media:

1
import uuid
2
3
from flask import send_from_directory
4
from sarvamai.play import save
5
6
AUDIO_DIR = Path("audio_replies")
7
AUDIO_DIR.mkdir(exist_ok=True)
8
9
10
@app.route("/audio/<filename>")
11
def serve_audio(filename):
12
    return send_from_directory(AUDIO_DIR, filename, mimetype="audio/wav")
13
14
15
# Inside whatsapp_webhook(), replace the final `twiml.message(reply_text)` with:
16
message = twiml.message(reply_text)
17
18
try:
19
    audio = sarvam.text_to_speech.convert(
20
        text=reply_text,
21
        target_language_code="en-IN",
22
        model="bulbul:v3",
23
        speaker="shubh",
24
    )
25
    filename = f"{uuid.uuid4().hex}.wav"
26
    save(audio, AUDIO_DIR / filename)
27
    message.media(f"{request.host_url}audio/{filename}")
28
except Exception:
29
    # If TTS fails, the sender still gets the text reply above.
30
    logger.exception("Text-to-speech failed, sending text-only reply")
31
32
return Response(str(twiml), mimetype="application/xml")

6. Example Conversation

Here’s what a typical exchange looks like once the bot is running, a Hindi voice note in, a text reply out:

User (voice note, Hindi): "मुझे अपने ऑर्डर का स्टेटस चेक करना है"
Bot: "बिल्कुल! कृपया अपना ऑर्डर नंबर बताएं, मैं अभी स्टेटस चेक करता हूं।"
User (text): "Can you also reply in English?"
Bot: "Of course! I can switch to English anytime, just let me know what you need."

Available Options

Supported Languages

Speaker Voices (Bulbul v3)

Male: Shubh (default), Aditya, Rahul, Rohan, Amit, Anand, Karun, and more

Female: Ritu, Priya, Neha, Pooja, Simran, Kavya, Ishita, and more

See Text-to-Speech for the full voice list.

Customization

Change the bot’s persona

SYSTEM_PROMPT in the script controls tone and behavior. Swap it for something domain-specific, for example a support bot that only answers questions about order status, and stays on-topic otherwise.

Reply in the sender’s own language

target_language_code in the TTS call is hardcoded to en-IN in the example above. Pair it with the language Saaras detected (transcript.language_code) so a Hindi voice note gets a Hindi voice-note reply, not an English one.

Move off the Sandbox

Before real users message your bot, register a WhatsApp Sender under Messaging → Senders → WhatsApp senders in the Twilio Console. This requires Meta Business verification and removes both the shared-number and 3-day opt-in limitations of the Sandbox.

Tips & Best Practices

• Language auto-detection: Use language_code="unknown" in STT to auto-detect whichever language the sender types or speaks in, rather than hardcoding one.
• Reply length: Keep system prompts explicit about reply length. WhatsApp message bodies are capped at 1600 characters by Twilio, and short replies also read better in a chat window.
• The 24-hour session window: WhatsApp only allows free-form replies within 24 hours of the user’s last message. Outside that window, you need a pre-approved template message, this is a WhatsApp Business Platform policy that neither Sarvam nor Twilio can bypass.
• Sandbox opt-in: If messages never reach your webhook, check that the sending number actually joined the Sandbox (Step 3), Twilio silently drops messages from numbers that haven’t opted in, and this is the most common reason nothing arrives.
• API key security: Load SARVAM_API_KEY, TWILIO_ACCOUNT_SID, and TWILIO_AUTH_TOKEN from environment variables rather than hardcoding them, especially outside local experimentation.

Error Handling

You may encounter these errors while using the underlying APIs:

• 403 Forbidden (invalid_api_key_error), invalid Sarvam API key. Use a valid key from the Sarvam AI Dashboard.
• 429 Too Many Requests (insufficient_quota_error / rate_limit_exceeded_error), Sarvam credits exhausted or rate limit hit. Retry with exponential backoff.
• 500 Internal Server Error (internal_server_error), issue on Sarvam’s servers. Try again later; contact support if persistent.
• Twilio error 63015, “recipient has not joined the Sandbox.” The sending number needs to send the join <code> message first (or its 3-day opt-in has expired), see Step 3 above.

For the full Sarvam error-code table and SDK exception reference, see Errors & Troubleshooting.

Additional Resources

• Documentation: docs.sarvam.ai
• Related APIs: Speech-to-Text · Chat Completion · Text-to-Speech
• Twilio docs: Sandbox for WhatsApp · WhatsApp API Overview · Python Helper Library
• Community: Join the Discord Community

Final Notes

• Keep your API keys secure, prefer environment variables over hardcoding.
• The Sandbox is for development only, each recipient must opt in, and the connection expires after 3 days of inactivity.
• The webhook degrades gracefully: a failed transcription or chat call still gets the sender a reply, and a failed voice-note synthesis still gets them the text.

Keep Building! 🚀