Note: when a voice bot responds with a 2-second pause, the user leaves. In production, we encountered projects where 300 ms latency determined conversion fate. For example, in one call center, reducing TTFA from 1.2s to 250ms improved retention by 35%. In another project, optimizing from 1.2s to 200ms saved the company $45,000 annually in operational costs. Streaming speech synthesis (Streaming TTS) is not an optimization—it's a basic necessity for real-time voice interfaces.
We implement Streaming TTS with time-to-first-audio (TTFA) from 100 ms. Below is the technical side: how chunking, buffering, and parallel synthesis work. Our experience: 5+ years in speech technologies and over 30 projects with TTS.
How streaming speech synthesis works?
Text is split into logical chunks—usually by sentences or phrases of 10–20 words. The first chunk goes directly to synthesis, while the rest are prepared in parallel. The client receives the audio stream via WebSocket or HTTP chunked encoding and starts playback immediately.
Implementation with OpenAI TTS Streaming:
from openai import AsyncOpenAI import asyncio client = AsyncOpenAI() async def stream_tts(text: str): async with client.audio.speech.with_streaming_response.create( model="tts-1", voice="alloy", input=text, response_format="pcm", ) as response: async for chunk in response.iter_bytes(chunk_size=4096): yield chunk WebSocket server for real-time TTS
For self-hosted solutions (e.g., Coqui XTTS), we use WebSocket:
from fastapi import FastAPI, WebSocket from TTS.api import TTS import numpy as np import asyncio app = FastAPI() tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to("cuda") def split_into_sentences(text: str) -> list[str]: import re sentences = re.split(r'(?<=[.!?])\s+', text) return [s.strip() for s in sentences if s.strip()] @app.websocket("/tts-stream") async def tts_websocket(websocket: WebSocket): await websocket.accept() try: while True: text = await websocket.receive_text() sentences = split_into_sentences(text) for sentence in sentences: wav = await asyncio.get_event_loop().run_in_executor( None, lambda s=sentence: tts.tts(text=s, language="en", speaker_wav="default.wav") ) audio_bytes = (np.array(wav) * 32767).astype(np.int16).tobytes() await websocket.send_bytes(audio_bytes) await websocket.send_json({"type": "done"}) except Exception: await websocket.close() Why minimal latency matters?
Microsoft Research shows that latency >1s reduces user retention by 20%. For voice assistants, the critical threshold is 400 ms—beyond that, the dialogue feels unnatural. In our projects, we achieve p95 TTFA <300 ms even on self-hosted solutions.
TTFA comparison of popular engines
| TTS | TTFA |
|---|---|
| ElevenLabs Turbo | ~100 ms |
| OpenAI TTS-1 streaming | ~200 ms |
| Azure Neural TTS streaming | ~150 ms |
| Coqui XTTS (self-hosted, GPU) | ~300–500 ms |
| Yandex SpeechKit | ~200–300 ms |
ElevenLabs Turbo is twice as fast in TTFA compared to Coqui XTTS on GPU.
TTFA optimization steps
| Step | Action | TTFA reduction |
|---|---|---|
| 1 | Chunking | 10–20% |
| 2 | Caching templates | 5–15% |
| 3 | Parallel chunk synthesis | 20–30% |
| 4 | Streaming playback | 10–20% |
How we do it
We use the optimal approach for each task:
- Cloud APIs: OpenAI, ElevenLabs, Azure—for fast integration (2–3 days).
- Self-hosted: Coqui XTTS, Silero on GPU—for full control and offline capabilities.
- Hybrid: cache template phrases (greetings, hold messages), stream dynamic content.
- Monitoring: log TTFA, latency p99, inference FLOPS—for proactive alerting.
More about monitoring
We use Prometheus + Grafana to collect TTFA and p99 metrics. When a threshold is exceeded (e.g., 300 ms), an alert fires, and we automatically switch to a backup TTS engine. This guarantees stability under load up to 1000 concurrent sessions.Choosing a TTS engine
The choice depends on latency requirements, quality, and infrastructure. If minimal TTFA is needed—ElevenLabs Turbo. For custom voices and offline—Coqui XTTS. Cloud APIs (OpenAI, Azure) suit typical scenarios. We help you select the stack and optimize it for your use case.
Measuring TTFA
TTFA (Time To First Audio) is the time from text submission to the first audio packet arriving at the client. Measured as the difference between timestamps. We use built-in engine metrics and monitoring tools (Prometheus). For some projects, TTFA is critical; for others, overall dialogue latency matters.
Want to implement streaming TTS? Contact us for a preliminary assessment of your project.
What is included in the implementation
- Server-client architecture (WebSocket or HTTP Streaming)
- Integration with chosen TTS (OpenAI, ElevenLabs, self-hosted)
- Chunk and buffer optimization
- Testing on your scenario (N+ hours of conversations)
- Integration and monitoring documentation
- Stability guarantee under load (up to 1000 concurrent sessions)
- Monitoring and alerting (TTFA, p99, GPU utilization)
Estimated timelines
- Cloud TTS integration: from 2 to 5 days
- Self-hosted server with GPU: from 1 to 2 weeks
- Full implementation with monitoring and optimization: from 3 weeks
Cost is calculated individually—depends on volume and chosen stack.
Our team has 5+ years of experience in speech technologies and has implemented over 30 TTS projects. Order streaming TTS implementation—get a consultation and project assessment. Contact us—we will evaluate your project and propose the optimal solution.