WebSocket API

​

End-to-end flow

Client ──► Connect /ws
          └─► send {"type":"config", ...}
Sayna  ──► validate + spin up providers + optional LiveKit join
Sayna  ──► send {"type":"ready", ...}
Client ◄── start streaming audio / send commands
Client & Sayna ── keep exchanging JSON control frames + binary audio
Either side closes socket → Sayna tears down providers + LiveKit room

​

Message taxonomy

​

Configuration message

{
  "type": "config",
  "stream_id": "support-call-123",
  "audio": true,
  "stt_config": {
    "provider": "deepgram",
    "language": "en-US",
    "sample_rate": 16000,
    "channels": 1,
    "punctuation": true,
    "encoding": "linear16",
    "model": "nova-3",
    "auth": {
      "api_key": "your-deepgram-key"
    }
  },
  "tts_config": {
    "provider": "elevenlabs",
    "model": "eleven_flash_v2_5",
    "voice_id": "voice_id_here",
    "speaking_rate": 1.0,
    "audio_format": "linear16",
    "sample_rate": 24000,
    "auth": {
      "api_key": "your-elevenlabs-key"
    }
  },
  "livekit": {
    "room_name": "conversation-room-123",
    "enable_recording": true,
    "sayna_participant_identity": "sayna-ai",
    "sayna_participant_name": "Sayna AI",
    "listen_participants": []
  },
  "loading_audio": {
    "data": "UklGRiQAAABXQVZFZm10IBAAAAABAAEA...",
    "format": "wav",
    "volume": 0.3
  }
}

• audio=false skips STT/TTS initialization but still allows LiveKit control + data channel relays.
• stream_id is optional; provide one to control recording paths and correlate sessions, or let Sayna generate a UUID v4 and return it in the ready message.
• stt_config.sample_rate/channels/encoding must exactly match the binary frames you send—Sayna will not resample.
• tts_config values become session defaults; individual speak messages can override most fields or add id for correlation.
• stt_config.auth and tts_config.auth are optional objects that supply provider credentials for this session only. When provided, they fully replace server-configured credentials (no partial merge). Omit or send {} to use server defaults. See Provider auth overrides below for supported shapes per provider.
• The livekit block is optional but required for send_message and mirrored playback/recording.
• loading_audio is optional and additive; it is only processed when audio=true and the livekit block is present, and a decode failure is non-fatal (the session continues without loading capability). See the dedicated Loading indicator section below.

​

Streaming audio best practices

• Chunk size: ~100–200 ms of audio per binary frame (e.g., 3 200 bytes for 16 kHz mono linear16).
• Maintain consistent cadence; bursty uploads increase STT latency.
• Enable punctuation + turn-detect (default features) when you need is_speech_final cues to know when a speaker stopped.
• If the client detects a barge-in (human interrupts AI), send {"type":"clear"} before queueing new speak text.

​

Client commands in detail

​

speak

{
  "type": "speak",
  "text": "Hello! How can I help you today?",
  "flush": true,
  "allow_interruption": true,
  "tts_config": {"voice_id": "aura-luna-en"},
  "id": "greeting-1"
}

• flush=true clears queued audio before speaking; set false to build playlists.
• allow_interruption=false protects critical prompts even if the client sends clear.
• tts_config may override any session default, including provider credentials via the auth field.
• Finish signal arrives via tts_playback_complete with the same id.

​

loading_start

{ "type": "loading_start" }

​

loading_stop

{ "type": "loading_stop" }

​

send_message

{
  "type": "send_message",
  "message": "{\"action\":\"typing\"}",
  "role": "assistant",
  "topic": "status",
  "debug": {"source": "sayna-ai", "version": "1.0"}
}

​

Server events in detail

​

ready

​

stt_result

{
  "type": "stt_result",
  "transcript": "Hello, how are you today?",
  "is_final": true,
  "is_speech_final": true,
  "confidence": 0.94
}

• Interim updates set is_final=false and keep streaming as long as audio flows.
• The final chunk for an utterance flips is_final=true.
• is_speech_final=true indicates the turn detector heard silence—ideal for triggering a response.

​

message and participant_disconnected

{"type":"message","label":"livekit.joined","details":{"identity":"user-123"}}
{"type":"participant_disconnected","participant":{"identity":"user-123","name":"Alex"}}

​

Binary TTS payloads

socket.onmessage = (event) => {
  if (event.data instanceof Blob) playAudioChunk(event.data);
  else handleControl(JSON.parse(event.data));
};

​

Provider auth overrides

​

Fallback behavior

• If auth is omitted, the server uses its configured provider credentials.
• If auth is sent as an empty object {}, the server also falls back to its configured credentials.

​

No partial merge

​

Supported auth shapes

​

Validation

• Required auth fields for the selected provider are missing.
• Auth field types are invalid.
• Unsupported auth keys are sent for that provider.
• The provider name is unsupported.

​

Loading indicator

​

loading_audio configuration

• Only 16-bit PCM is supported (16-bit integer WAV, or raw 16-bit little-endian PCM). Other bit depths are rejected.
• Decoded duration: ~250 ms to ~10 s, 1–4 s recommended.
• Decoded size is capped at ~1.9 MB.
• Author the clip so its start and end amplitudes match — otherwise the loop seam will click.
• Author the clip quiet enough to sit under speech rather than compete with it.

• loading_audio is processed only when audio=true and a livekit block is present in the same config message. Supplied without those, the server emits an error and continues the session without loading capability.
• Decode or validation failures are non-fatal — the session continues, STT/TTS are unaffected, and only the loading feature is unavailable. A later loading_start re-reports the original decode error so the failure is never silent.

​

Controlling the loop

​

Typical turn

client → stt_result (final, is_speech_final=true)
client → {"type": "loading_start"}      # background work begins
…application calls its LLM/tools…
client → {"type": "loading_stop"}        # background work ends — stop BEFORE speak
client → {"type": "speak", "text": "Here is what I found…"}

​

LiveKit interplay

• Sayna manages the agent-side participant automatically; humans still fetch tokens via POST /livekit/token.
• listen_participants limits which identities can feed audio back into Sayna.
• Recordings respect your LiveKit/S3 configuration when enable_recording=true; files are stored at {server_prefix}/{stream_id}/audio.ogg using the stream_id from your config or the ready message.
• send_message and clear affect both the WebSocket client and LiveKit room so playback remains in sync.

​

Troubleshooting