Skip to navigation

Dictate Live

View as Markdown

The live socket sends partial text while the person speaks, then one finished text after each pause. Open it from your server, not from a browser: the socket needs your key in a header, and a long-lived key must not go in a browser or app.

1

Connect

Call connect with the language and the audio format. The first message is session.begin.

2

Configure (optional)

Send config.update with output_rules, style, vocabulary or context.

3

Send audio

Base64 audio: 16 kHz (or 8 kHz), mono, 16-bit, at most 1 s per message (0.5 s with stream_type=fast), at about real-time speed.

4

Show the text

Show each transcript.partial as a live preview. Replace the turn with dictation.final after each pause.

5

End the session

Send end and read until session.end. output.final comes before it.

A live session

import asyncio, base64, os, wave
from sarvamai import AsyncSarvamAI
from sarvamai.types import DictateAudioInput, DictateConfigUpdate, DictateEnd
client = AsyncSarvamAI(api_subscription_key=os.environ["SARVAM_API_KEY"])
CHUNK = 3200 # 100 ms of 16 kHz, 16-bit mono audio
RULES = (
'List the action items from this note as bullets that start with "- ". '
'For each, write the owner, the task and the due date, as said. '
'Write "None" if there are none.'
)
async def live(path):
with wave.open(path) as w: # 16 kHz, mono, 16-bit WAV
pcm = w.readframes(w.getnframes())
async with client.dictate_streaming.connect(
language_code="en-IN", encoding="linear16", sample_rate="16000"
) as ws:
async def send_audio():
await ws.update_config(DictateConfigUpdate(output_rules=RULES))
for i in range(0, len(pcm), CHUNK):
chunk = base64.b64encode(pcm[i:i + CHUNK]).decode()
await ws.send_audio(DictateAudioInput(audio=chunk))
await asyncio.sleep(CHUNK / 32000) # real time
await ws.end(DictateEnd())
sender = asyncio.create_task(send_audio())
async for m in ws:
if m.event == "transcript.partial":
pass # show m.text as a live preview
elif m.event == "dictation.final":
print(f"turn {m.utterance_idx}: {m.text}")
elif m.event == "output.final":
print("output:\n" + (m.text or "(none)"))
elif m.event == "error":
print("error:", m.code, m.message)
elif m.event == "session.end":
print(f"billed {m.usage.audio_duration_s} s")
break
await sender
asyncio.run(live("live_note.wav"))
Output
turn 0: Note for the team, our order from Nilga Exports is stuck at the Chennai port. Neha will call the customs agent tomorrow at 10. Rohan, please send the updated budget to Kriya by Monday.
output:
- Neha, call the customs agent, tomorrow at 10
- Rohan, send the updated budget to Kriya, by Monday
billed 14.362 s

JavaScript on Node.js 22 and later: the built-in WebSocket of these versions cannot send the key header, so the connection is refused with a 403. Run the script with node --no-experimental-websocket live.mjs. Node.js 18 and 20 work without the flag.

The messages you get

EventWhenWhat to do
session.beginRight after you connectNothing: the session is ready
transcript.partialWhile the person speaksShow it as a live preview
transcript.finalThe raw text of a turnOptional: keep it for search
dictation.finalAfter each pauseReplace the turn with text. Match turns by utterance_idx
output.finalAfter the last turn, when you sent output_rulesUse text
errorA problem in the sessionRead code and message
session.endAfter you send endRead the billed usage, then close

Good to know

Connect

  • Key in a header. Send api-subscription-key as a header. A key in the URL query string is refused with 403.
  • Match sample_rate. Use 16000 or 8000, the real rate of your audio. Dictate does not check it, and it measures the audio at the rate you declare: 16 kHz audio sent as 8000 raises no error, but the text is wrong and the billed seconds double.
  • Send raw samples. Leave out the WAV header. A header is not refused, but it is read as audio.

While you stream

  • Push-to-talk. Connect with endpointing="manual". Send speech_start when the button goes down and speech_end when it comes up. Each press is one turn, and only the pressed audio is billed.
  • Add a name in the middle. Send config.update with vocabulary. It applies from the next turn that ends. An update that changes nothing is refused with invalid_config.
  • Keep reading. A client that stops reading also stops answering the server’s pings, and Dictate closes it with 1011 (keepalive ping timeout) in under a minute. Read messages in their own task, and do not block it on slow work.
  • Quiet sockets. The socket closes after 60 s with no message from you. Send ping during long silences.
  • Long speech. About 18 to 20 s of speech with no pause ends a turn, even inside a word.
  • A repeated key. If a JSON message repeats a key, Dictate uses the last one. A repeated form field on a clip or a job is refused with 400.

Finish

  • Always send end and read until session.end, or you lose the last turn and the output. Dictate then closes the socket (1000). After end, only read the remaining messages until session.end.
  • Every close code and message is in the WebSocket reference.