LLM Streaming Responses Implementation – Turnkey

You launched an LLM-powered chatbot, and users are complaining about long response times. A 5–10 second delay kills conversion — each token appears almost instantly on the client, creating a live typing effect. This is critical for chatbots, assistants, and any interface where the user waits for a response. Once we implemented streaming for a fintech service with 10,000 concurrent users: after switching to streaming output, latency p99 dropped from 12 s to 400 ms, and conversion increased by 30%. Streaming output speeds up time to first response by 10–20 times compared to batch delivery. The solution is streaming responses. We implement LLM streaming using Server-Sent Events (SSE) and WebSocket, reducing visible latency to 200–500 ms. Our experience: 50+ LLM integrations for fintech, e-commerce, and SaaS. We guarantee quality: latency p99 < 500 ms.

According to MDN documentation, SSE uses a standard HTTP connection and automatically reconnects on drop. This makes it optimal for most chatbot scenarios.

How LLM Streaming Works

Instead of waiting for the full response, the server sends tokens to the client as they are generated. The protocol used is SSE (Server-Sent Events) — a text stream over HTTP for streaming data. The client receives data: chunks with JSON field text. When the model completes, it sends done: true. SSE is 2–3 times simpler to implement than WebSocket and requires no additional backend libraries. Without streaming, the user waits 5–10 seconds — long enough to leave. With streaming, the first character appears within 200 ms. This way, the user sees streaming output in real time.

Why SSE Over WebSocket?

SSE is simpler: no handshake, automatic reconnection, works over HTTP/2. WebSocket enables bidirectionality but is overkill for a simple chat. Comparison:

For 90% of cases, SSE is the optimal choice. WebSocket is justified for bidirectional transfer (e.g., voice commands).

Supported Models

Virtually all modern LLMs support streaming: Claude (Anthropic), GPT-4o (OpenAI), LLaMA 3 (via vLLM), Gemini (Google), Mistral, Qwen. For each model, the corresponding SDK with streaming mode must be used. For example, AsyncAnthropic and AsyncOpenAI provide async token generators. We choose the model for your use case: for chatbots — Claude Sonnet or GPT-4o, for real-time assistants — LLaMA 3 via vLLM for minimal latency.

Handling Interruptions and Backpressure

The client can cancel the request at any time. If not handled, the model continues generation — wasting tokens and money. We implement a cancel_event:

import asyncio

async def stream_with_cancellation(
    messages: list[dict],
    cancel_event: asyncio.Event,
) -> AsyncGenerator[str, None]:
    """Streaming with cancellation support"""
    async with anthropic_client.messages.stream(
        model="claude-sonnet-4-5",
        max_tokens=2048,
        messages=messages,
    ) as stream:
        async for text in stream.text_stream:
            if cancel_event.is_set():
                stream.close()
                yield f"data: {json.dumps({'cancelled': True})}\n\n"
                return
            yield f"data: {json.dumps({'text': text})}\n\n"

Additionally, we use backpressure: when the client buffer is full, we pause the generator iteration. This prevents connection drops.

Backend: FastAPI + SSE

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from anthropic import AsyncAnthropic
from openai import AsyncOpenAI
import asyncio
import json

app = FastAPI()
anthropic_client = AsyncAnthropic()
openai_client = AsyncOpenAI()

async def stream_anthropic(messages: list[dict], system: str = "") -> AsyncGenerator[str, None]:
    """Generator for Claude streaming"""
    async with anthropic_client.messages.stream(
        model="claude-sonnet-4-5",
        max_tokens=2048,
        system=system,
        messages=messages,
    ) as stream:
        async for text in stream.text_stream:
            yield f"data: {json.dumps({'text': text})}\n\n"
        yield f"data: {json.dumps({'done': True})}\n\n"

async def stream_openai(messages: list[dict]) -> AsyncGenerator[str, None]:
    """Generator for OpenAI streaming"""
    async with await openai_client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        stream=True,
    ) as stream:
        async for chunk in stream:
            delta = chunk.choices[0].delta
            if delta.content:
                yield f"data: {json.dumps({'text': delta.content})}\n\n"
        yield f"data: {json.dumps({'done': True})}\n\n"

@app.post("/chat/stream")
async def chat_stream(request: dict):
    messages = request.get("messages", [])
    provider = request.get("provider", "anthropic")

    generator = (
        stream_anthropic(messages)
        if provider == "anthropic"
        else stream_openai(messages)
    )

    return StreamingResponse(
        generator,
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "X-Accel-Buffering": "no",
        }
    )

We choose AsyncAnthropic and AsyncOpenAI for async streaming. Important: set X-Accel-Buffering: no for nginx — otherwise nginx will buffer SSE.

Frontend: React with Streaming Read

import { useState, useCallback } from 'react';

function useStreamingChat() {
  const [response, setResponse] = useState('');
  const [isStreaming, setIsStreaming] = useState(false);

  const sendMessage = useCallback(async (messages: Message[]) => {
    setIsStreaming(true);
    setResponse('');

    const res = await fetch('/chat/stream', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ messages }),
    });

    const reader = res.body!.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;

      const chunk = decoder.decode(value);
      const lines = chunk.split('\n\n');

      for (const line of lines) {
        if (line.startsWith('data: ')) {
          const data = JSON.parse(line.slice(6));
          if (data.done) {
            setIsStreaming(false);
            return;
          }
          setResponse(prev => prev + data.text);
        }
      }
    }
    setIsStreaming(false);
  }, []);

  return { response, isStreaming, sendMessage };
}

For POST requests we use ReadableStream — more flexible than EventSource, and allows sending a request body.

Process

1. Analytics: review your architecture, select model and protocol (SSE/WebSocket).
2. Design: draw flow diagram, define interruption points.
3. Implementation: write backend endpoint, frontend component, error handling.
4. Testing: load test with simulated N users, measure p99 latency.
5. Deployment & monitoring: configure nginx, CDN, monitoring (Grafana + Prometheus) to track streaming connections.

What's Included

• Backend endpoint with SSE or WebSocket (FastAPI).
• React component (TypeScript) with interruption handling, loading indicator, auto-scroll.
• Documentation: protocol description, nginx configs, request examples.
• Post-launch support: 1 month incident management.

Estimated Timelines

Cost is calculated individually. Contact us for a project estimate. Order streaming output implementation for your LLM service — improve UX and conversion.

Streaming output is not just a feature, but a requirement for modern UX of streaming chatbots. We implement it turnkey in 5–10 days. Our experience: 50+ LLM projects. Get a consultation right now.