A demo LLM endpoint is ten lines. Production is FastAPI async HTTP, provider abstraction, and hard boundaries around cost, latency, and data handling — the layer between your product and OpenAI, Anthropic, or self-hosted models. This API surface is where billing, security, and observability attach — treat it as core infrastructure, not a thin wrapper.
API design that scales
Separate routes for chat, completion, embeddings, and tool-augmented flows. Whitelist models server-side; never trust client model strings. Return usage metadata — prompt tokens, completion tokens, model ID — in every response for billing and debugging. Define Pydantic request models with max token limits, allowed tool lists, and tenant context so malformed client payloads fail validation before they reach the provider. OpenAPI docs generated from FastAPI become the contract your frontend team codes against.
Version your prompt templates in git and expose prompt_version in logs. Treat prompt changes like API schema changes: diff, review, eval, deploy. Abstract providers behind a common interface — OpenAI, Anthropic, Azure OpenAI, and self-hosted vLLM should swap without route handler changes. Normalize token usage fields across providers for consistent billing dashboards.
Resilience patterns
Set aggressive timeouts — users abandon chat after 8–12 seconds of silence. Retry 429 and 5xx with exponential backoff and jitter. Maintain fallback model tiers for degraded mode when primary provider is down. Stream partial responses so users see progress even when total generation exceeds ten seconds — perceived latency matters as much as wall-clock time. Use asyncio.gather for parallel retrieval and generation prep where routes allow.
Circuit breakers prevent cascade failures when a provider degrades. Queue long-running jobs instead of blocking HTTP workers on multi-minute inference. Implement request-level budgets: if a tenant exceeds daily token quota, return 429 with upgrade messaging rather than silently truncating responses mid-stream. Queue overflow requests with position feedback rather than failing hard during traffic spikes.
Security and compliance
Authenticate per tenant, rate-limit by plan tier, and scrub logs for PHI in healthcare deployments. Align OpenAI or Anthropic data retention settings with your DPA or BAA before launch. Disable provider-side training on customer data explicitly in API configuration — defaults vary by provider and contract tier. Review subprocessors list when adding a new provider for GDPR and BAA coverage.
Hash prompts when full content logging is prohibited. Store enough metadata to reproduce incidents without storing sensitive user text indefinitely. Implement input sanitization for prompt injection via user-supplied content that gets embedded in system prompts — treat user text as untrusted data, not instructions. Strip or escape HTML in user inputs before they enter prompt templates.
"A demo endpoint proves the model works. Production proves your team can operate it."
— — GenOrbit engineering practiceWhat to do next
Start with one measurable outcome — latency, accuracy, adoption, or revenue — and design your stack around that constraint. GenOrbit ships production AI systems across healthcare, astrology, fintech, and SaaS; if you want a second pair of eyes on architecture or delivery, we are happy to talk.
Related: /services/llm-rag-agent-development · /blog/ai-observability-from-day-one · /industries/healthcare-ai