Load Testing and Cost per Query in RAG

ShopBot was serving 900 queries a day. Krishna's stated next-quarter target was 10,000 queries a day. The marketing team was already talking about a campaign that would put the chatbot link in front of 100,000 customers in the second week of February.

Arjun did not know whether the system would survive 10,000 queries a day. He did not want to find out at 3 in the afternoon on the day a campaign launched.

The Valid Failure: 500 Queries per Minute

The load-test harness drew queries from the labeled RAGAS dataset and the failed-query log mixed together — the full range of production-style queries, not cherry-picked ones.

At 50 queries/minute (current peak): untroubled. At 200 queries/minute: Qdrant CPU up, latency held at 380ms. At 500 queries/minute (≈30,000 queries/day):

• Median latency: 1.2 seconds (up from 380ms).
• Qdrant cluster: 65% CPU.
• LLM provider: occasional rate-limit warnings.
• Cross-encoder rerank: holding one CPU core fully busy per query.

The bottleneck was the cross-encoder. It ran on the same machine as the FastAPI server. FastAPI was waiting for rerank to finish before sending the prompt to the LLM. The two operations were sequentially coupled on a shared CPU. At 500 qpm, the cross-encoder was the single hottest component in the stack.

async def run(qpm: int, duration_s: int) -> None:
    interval = 60.0 / qpm
    latencies: list[float] = []
    async with httpx.AsyncClient() as client:
        deadline = time.time() + duration_s
        while time.time() < deadline:
            query = random.choice(QUERIES)
            asyncio.create_task(_record(client, query, latencies))
            await asyncio.sleep(interval)
    p50 = statistics.median(latencies)
    p95 = statistics.quantiles(latencies, n=20)[18]
    print(f"qpm={qpm}  p50={p50*1000:.0f}ms  p95={p95*1000:.0f}ms")

The Valid Source: Cost per Query at Each Architecture Layer

The cost of an answered query on the Book 2 architecture (before caching):

With the 22% semantic cache hit rate: ₹0.0156 blended cost per query.

At 10,000 queries/day without cache: ₹200/day. With cache: ₹156/day. At 50,000 queries/day: ₹1,000/day without cache, ₹780/day with cache.

The embedding cost is a separate line item — every cache miss produces a fresh embedding API call. The path to eliminating it is a fine-tuned in-house embedding model (a bge-small variant, exported to ONNX INT8, running on the same CPU box as the reranker). That removes the per-query API call entirely. Book 3's work.

The Valid Knowledge: What the Fix Required

Cross-encoder split. The reranker was pulled into its own FastAPI service, containerised, deployed on a separate machine. The main ShopBot service called the rerank service over HTTP. Network overhead added: 8ms per query. CPU-contention problem: gone. The rerank service could now be scaled independently — two replicas behind a load balancer, as many as traffic required.

Qdrant cluster upgrade. One node → two nodes with replication in asia-south1. Monthly cost: ₹3,200 → ₹6,800. At 10,000 queries/day with 22% cache hit rate: the cache savings of ₹44/day pay for the ₹120/day cluster cost within the quarter. At 50,000 queries/day, the cache pays for the cluster outright.

After both changes. Re-ran load test at 1,000 queries/minute (≈60,000 queries/day): 580ms median latency. Three hours with no failures the labeled evaluation would have called a regression.

Campaign Friday. 7,000 queries in a single day. Cache hit rate climbed to 28% (repeated queries from new users hitting the same popular questions). Average latency held at 420ms. RAGAS Monday that morning: F: 0.84 / AR: 0.83 / CP: 0.89 / CR: 0.81 — the strongest scores the system had ever recorded, under three times its previous peak load.

The 22% from Chapter 1 had become 9% by the end of Book 2. The remaining 9% sorted into known categories:

• 68 queries: multi-product styling questions (CRAG, Book 3)
• 41 queries: vague queries — something nice, what's new (HyDE, Book 3)
• 63 queries: genuine out-of-scope (correct routing, not a failure)
• 238 queries: long-tail conversational edge cases (routed pipeline, LangGraph, Book 3)

A system that survives load is not the same as a system that works. Book 2 ended with ShopBot being run, not built — which is a different kind of engineering and a different kind of attention.

This article covers concepts from Book 2, Chapter 9 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to cloud-native deployment.