We treat prompts, retrieval configs, and model routes like API schema — versioned, tested, and rolled back. Here is the pipeline we run on client and internal AI systems before traffic hits new behavior. Prompt changes that skip eval are production incidents waiting for peak traffic — we treat them with the same rigor as database migrations. Staging environments must mirror production index versions — testing against stale indexes passes CI and fails users.
Testing pyramid for AI
Unit tests on deterministic code: parsers, chart engines, tool validators. Golden-set eval on end-to-end LLM flows — faithfulness, citation, latency budgets. Deterministic unit tests cover parsers, validators, chart math — anything without LLM calls. Golden-set eval runs on every PR touching prompts, chunk size, embedding model, or retrieval top-k. Synthetic load tests on embedding endpoints reveal batch size limits before launch traffic arrives.
Regression suite runs in CI on every PR touching prompts or chunking. Human review sample weekly for domains where metrics lie. Human review samples scale with domain risk — healthcare gets fifty reviews weekly; internal tools may get ten. Flaky eval tests get quarantined and fixed, not ignored. Contract tests between ingestion and query services validate metadata schema compatibility on every deploy.
Deploy mechanics
Feature flags for prompts and models. Canary five to ten percent traffic on retrieval index changes. Separate deploy trains for ingestion, index, and query services. Feature flags default off for new prompts — enable per tenant for design partners before global rollout. Canary compares cost and faithfulness between cohorts — rollback if treatment exceeds error budget. Eval dataset versioning tags results — compare v2.3 versus v2.4 faithfulness, not against moving targets.
Rollback must be one command — not a redeploy meeting. Database migrations for vector metadata get backward-compatible plans. Ingestion deploys should not block query deploys — separate pipelines prevent re-index jobs from taking chat offline. Backward-compatible metadata migrations avoid downtime during embedding dimension changes. Security regression tests include prompt injection cases in CI — not only OWASP on REST routes.
Post-deploy discipline
Review dashboards at forty-eight hours and two weeks: cost, errors, feedback, support tickets. Incident runbooks for provider outage and index corruption. Forty-eight-hour reviews catch provider rate limits; two-week reviews catch usage patterns that stress retrieval. Incident runbooks live in repo docs linked from PagerDuty. Post-deploy automated smoke queries run every fifteen minutes — catch index corruption before users flood support.
Pair with prototype-to-production checklist for client handoffs. Client handoffs include eval suite ownership — teams without CI eval regress within one month. Cross-link prototype checklist for security items this pipeline assumes. Client training includes how to read eval diffs — handoff without literacy repeats failure mode.
What to do next
Start with one measurable outcome — latency, accuracy, adoption, or revenue — and design your stack around that constraint. GenOrbit ships production AI systems across healthcare, astrology, fintech, and SaaS; if you want a second pair of eyes on architecture or delivery, we are happy to talk.
Related: /services/custom-software-development · /blog/prototype-to-production-checklist · /blog/evaluating-llm-output-ragas-trulens