How to Load Test a RAG API with Locust

A RAG API that works at 10 requests per minute will fail at 500. The failure mode is rarely a crash — it is latency degradation: the cross-encoder backs up, Qdrant CPU saturates, and p99 climbs past the customer's patience threshold. Load testing finds the breaking points before the campaign does.

The Valid Failure

ShopBot was performing well in daily usage: median latency 280ms, RAGAS scores stable. Then a Friday fashion sale drove 600 queries per minute — 40× the normal load. No one had tested above 50 qpm.

At 600 qpm: median latency 1,240ms. p99: over 3 seconds. Customers were waiting 3 seconds for each chatbot response. The sale conversion rate dropped 18% compared to the previous campaign. The system did not crash. It degraded gracefully and expensively.

A Locust load test run before the campaign would have found the breaking point at ~180 qpm and prompted the three fixes that held the system at 280ms median under 600 qpm.

What Locust Is

Locust is a Python load testing tool. You define user behaviour in Python, specify how many virtual users to simulate and how fast to ramp up, and Locust reports latency percentiles, error rates, and throughput.

pip install locust

A Minimal ShopBot Locust File

# locustfile.py
from locust import HttpUser, task, between
import random

TEST_QUERIES = [
    "Do you have cotton kurtas under ₹2,000?",
    "What sizes does the silk saree come in?",
    "Is the linen co-ord available in XL?",
    "How do I wash the embroidered lehenga?",
    "Show me something for a beach wedding",
    "What is SKU K-1299?",
    "Do you have anything in red under ₹3,000?",
]

class ShopBotUser(HttpUser):
    wait_time = between(1, 3)  # seconds between requests per user

    @task
    def ask_question(self):
        query = random.choice(TEST_QUERIES)
        with self.client.post(
            "/chat",
            json={"question": query},
            catch_response=True,
        ) as response:
            if response.status_code != 200:
                response.failure(f"Status {response.status_code}")
            elif response.elapsed.total_seconds() > 2.0:
                response.failure(f"Too slow: {response.elapsed.total_seconds():.2f}s")

Running the Test

locust -f locustfile.py --host https://your-api.railway.app

# Or headless (no browser UI)
locust -f locustfile.py \
  --host https://your-api.railway.app \
  --headless \
  --users 100 \
  --spawn-rate 10 \
  --run-time 2m \
  --csv results

Open http://localhost:8089 to see the live dashboard: requests/second, median latency, 95th percentile, error rate.

What to Measure

Three numbers matter:

Run at increasing user counts: 10, 25, 50, 100, 200. Plot each metric against user count. The inflection point — where latency starts growing faster than user count — is your breaking point.

ShopBot's Breaking Points

The cross-encoder was the first failure. It shared CPU with FastAPI. At 180 qpm it saturated a single core — latency doubled. Fix: move reranking to a separate service with 3 replicas.

Qdrant was the second failure. The managed cluster was on the free tier (1 CPU, 1GB RAM). Fix: upgrade to a dedicated cluster.

LLM rate limit was third. A single OpenAI API key has a tokens-per-minute limit. At 320 qpm × ~800 tokens per request, the limit was hit. Fix: add a second API key and round-robin between them.

Diagnosing the Breaking Point

When p99 climbs, add timing instrumentation to identify which step is responsible:

import time

@app.post("/chat")
async def chat(query: Query):
    t0 = time.perf_counter()
    chunks = retrieve(query.question)
    t_retrieve = time.perf_counter()
    answer = generate(query.question, chunks)
    t_generate = time.perf_counter()

    return {
        "answer": answer,
        "timing": {
            "retrieve_ms": round((t_retrieve - t0) * 1000),
            "generate_ms": round((t_generate - t_retrieve) * 1000),
            "total_ms": round((t_generate - t0) * 1000),
        }
    }

Export timing data from Locust and correlate with user count. If retrieve_ms grows with load but generate_ms stays flat, the vector database is the bottleneck. If generate_ms grows, the LLM or cross-encoder is the bottleneck.

After the Fixes

With cross-encoder replicas (3×), Qdrant tier upgrade, and second API key:

Campaign Friday: 38,000 queries over 6 hours at campaign load. RAGAS scores during load: F 0.88, CP 0.91 — no quality degradation under stress.

The Valid Knowledge

• Load test before every major traffic event: a campaign is not the right time to discover a breaking point. Run Locust at 2× expected peak load before the event.
• Identify the bottleneck before adding capacity: "add more servers" is not a fix if the bottleneck is a single-threaded cross-encoder. Instrument to find which step breaks.
• p99 matters more than median: median latency looks fine even when 1 in 100 users waits 5 seconds. Track p95 and p99 explicitly.
• Test with realistic queries: random words are not real queries. Use a mix of your actual production query types — SKU lookups, vague semantic queries, multi-turn context. The query type affects which component is stressed.
• Cost per query is a load test output: measure LLM token consumption under load. At campaign scale, a 30% cost overrun is a budget line item.