Retrieval-augmented generation (RAG) is the default pattern for grounding LLMs on private data. At GenOrbit we pair LangChain for orchestration with FastAPI for a typed, async API — a stack that scales from MVP to production without a rewrite. This guide covers the ingestion, indexing, and query layers we deploy for client SaaS products, including failure modes we learned fixing production regressions.
Treat RAG as three services: ingestion, index, and query. Coupling them works for demos; production needs independent deploy and rollback.
Ingestion and chunking
The first failure mode in RAG is not the model — it is messy source material. Normalize PDFs, HTML exports, Notion pages, and database rows into clean text with provenance metadata: source URL, document ID, last-updated timestamp, and access tier. Without that metadata, you cannot filter by tenant, debug wrong answers, or expire stale content. Schedule ingestion as incremental sync jobs that detect document hash changes rather than full re-ingests nightly, and log parse failures per source so ops teams can fix upstream exports before users notice stale answers.
Chunk with intentional overlap — typically 10–20% — and preserve structural signals in metadata. Headings, section titles, and page numbers dramatically improve retrieval precision. For Indic scripts and mixed-language corpora, test tokenization early; naive byte splits break words and degrade embedding quality. Treat ingestion as a batch pipeline with idempotent jobs, not a one-off script you run before demo day. For tables and invoices, preserve row boundaries in metadata so retrieval returns complete records, not orphaned cells — LangChain splitters with custom separators for markdown and legal numbering reduce orphan chunks that confuse retrieval.
Embeddings and vector store selection
Embedding model choice should follow your latency budget and language mix, not leaderboard hype. Open-source models like bge-m3 work well for multilingual Indian SaaS; hosted embeddings reduce ops at the cost of vendor lock-in. Benchmark on your own queries, not public benchmark averages alone. Measure recall@k on held-out questions before switching models — a cheaper embedding that misses one critical policy doc costs more than premium API fees. Cache embedding vectors keyed by content hash to avoid re-embedding unchanged chunks during incremental sync.
Store vectors with tenant-scoped metadata filters for multi-tenant SaaS. Keep the vector client injectable behind an interface — you will swap providers as cost, hybrid search needs, and compliance mature. Start with a namespace per customer or workspace; never rely on post-hoc filtering as your only isolation layer. Plan for hybrid retrieval early if your corpus mixes exact identifiers with prose — SKU codes, drug names, and statute numbers often need BM25 alongside vectors. Abstract the retriever interface so you can add rerankers without rewriting the FastAPI route.
FastAPI serving layer and quality gates
Expose POST /query with Pydantic request and response models, async retrieval, and optional SSE streaming for chat UX. Log retrieval document IDs, prompt version, model name, and token usage on every request. Return citations in structured form so the frontend can link to sources without parsing markdown. Authenticate tenants at the API gateway and propagate tenant_id into every retrieval filter — post-hoc redaction is not isolation. Return partial results with explicit empty-state messaging when retrieval confidence falls below threshold instead of letting the model speculate.
Gate deploys on golden-set evaluations — faithfulness and context recall regressions should block release. RAG in production is three services: ingestion, index, and query. Coupling them works for demos; mature teams deploy and roll back each independently. Version your index alongside API deploys so rollbacks restore both code and retrieval behavior. Run shadow queries against staging indexes before promoting embedding model or chunking changes — compare answer diffs on your golden set automatically.
What to do next
Start with one measurable outcome — latency, accuracy, adoption, or revenue — and design your stack around that constraint. GenOrbit ships production AI systems across healthcare, astrology, fintech, and SaaS; if you want a second pair of eyes on architecture or delivery, we are happy to talk.
Related: /services/llm-rag-agent-development · /blog/pinecone-vs-chromadb-vs-weaviate · /blog/evaluating-llm-output-ragas-trulens