Streaming transforms perceived latency even when total generation time is unchanged. This guide covers a React client consuming a FastAPI SSE endpoint — our default pattern for copilots, support chat, and inline drafting tools. Proxy misconfiguration is the silent killer — nginx, Cloudflare, and corporate gateways buffer SSE unless explicitly told not to.
First-token time drives satisfaction more than total generation time. Optimize retrieval and time-to-first-byte before model size.
Backend: FastAPI EventSource
Return StreamingResponse with media type text/event-stream. Yield data chunks and flush per token or per small batch. Disable proxy buffering in nginx or Cloudflare — otherwise users see nothing until the full response completes. Set Cache-Control: no-cache and X-Accel-Buffering: no on SSE responses. Test streaming through your production CDN config, not only localhost — buffering bugs appear only behind reverse proxies.
Send heartbeat comments during long tool-call gaps so clients know the connection is alive. Include a final event with usage metadata for client-side analytics. Include a [DONE] sentinel or event type so clients distinguish complete streams from dropped connections mid-generation. Send periodic comment heartbeats during tool-call gaps longer than five seconds.
Frontend: consume the stream safely
Use fetch with ReadableStream or EventSource depending on auth needs. Batch React state updates via refs to avoid re-render storms on every token. The Vercel AI SDK abstracts much of this if you prefer conventions over custom plumbing. Debounce UI updates to every 50–100ms rather than every token — users cannot read faster than that and React reconciliation cost adds up on long answers. Use useRef for accumulated text and flush to state on animation frames.
Cancel in-flight streams when users navigate away or submit a new message. Orphaned streams waste tokens and can apply stale output to the wrong conversation thread. Pass AbortSignal from React cleanup to fetch so navigation cancels the server stream and stops token billing for abandoned requests. Handle 401 mid-stream by pausing output and prompting re-auth without losing partial content.
UX details that matter
Show a typing indicator until the first token arrives. Render markdown incrementally but defer syntax highlighting until the block completes. On mid-stream error, keep partial content visible and offer Continue or Retry. Lock scroll position during streaming unless the user scrolls up — auto-scroll jitter frustrates readers of long answers. Offer a Stop button that aborts client-side and calls a server cancel endpoint if supported.
Streaming without error states feels broken the first time the model times out. Design for failure as carefully as you design for the happy path. Log stream duration and token count client-side for product analytics — correlate drop-off with first-token latency to prioritize backend optimizations. Test streaming on Safari iOS — EventSource limitations may require fetch-based SSE polyfills.
What to do next
Start with one measurable outcome — latency, accuracy, adoption, or revenue — and design your stack around that constraint. GenOrbit ships production AI systems across healthcare, astrology, fintech, and SaaS; if you want a second pair of eyes on architecture or delivery, we are happy to talk.
Related: /services/llm-rag-agent-development · /blog/building-ai-copilot-saas