Redis Rate Limiting, SQS Per-Tenant Queues, and Locust Load Testing in RAG

Eight days before the national retailer's Diwali sale, Krishna put a number on the whiteboard.

Forty times the national retailer's average weekly query volume, concentrated across a four-hour window on the opening day.

"What happens to Vastralaya's queries during that window?"

Arjun opened the monitoring dashboard, looked at the numbers, did the arithmetic, and realised he did not know.

The Valid Failure: A Shared Container Cannot Isolate by Policy

The arithmetic was simple and precise.

The bge-small model cache served all tenants from a single FastAPI container running four worker processes. Each worker handled one embedding request at a time — a CPU-bound operation running 15–25 milliseconds per query. At four workers and 25ms per request, maximum sustained throughput was approximately 160 embedding requests per second.

The national retailer's projected peak, at 40× its average of 8 queries per second, was 320 queries per second. At 320 qps against a 160 qps capacity, the queue would grow at 160 requests per second. After 10 seconds the queue would hold 1,600 requests. After 30 seconds, the FastAPI request timeout would begin firing. Vastralaya's queries, sitting in the same queue, would wait behind the national retailer's backlog before timing out.

This was not a speculation. It was queue depth arithmetic.

First approach: scale up the shared container. A container with sixteen workers could sustain 640 requests per second, comfortably above the 320 qps peak. The economics were wrong: a container provisioned for 640 qps consumed the CPU and memory of four standard containers. The national retailer's Diwali spike lasted four hours. The overhead would be paid for 716 hours to cover 4.

More critically: a larger shared container was still shared. A slow query from the national retailer — a complex catalog comparison query routing through the detail_lookup branch — occupied a worker for 60 milliseconds. During those 60 milliseconds the worker was unavailable to Vastralaya. Scaling up reduced contention by probability. It did not eliminate it by architecture.

The Valid Source: Redis Sliding-Window Rate Limiter + SQS Per-Tenant Queues

The correct architecture had two components: per-tenant rate limiting and per-tenant queue isolation.

Redis sliding-window rate limiter — each request incremented a tenant-specific sorted-set counter with a one-second window. Requests within the rate limit passed through; requests beyond the limit were held in a per-tenant queue up to a maximum depth, then returned a 429 response:

TENANT_RPS_LIMITS: dict[str, int] = {
    "national_retailer": 50,
    "vastralaya":        20,
    "zudyog_fashion":    15,
    "surat_saree_house": 10,
    "jaipur_boutique":   10,
    "kolkata_silks":     10,
}
# Total allocated: 115 RPS — leaves headroom on the container

def check_rate_limit(tenant_id: str) -> bool:
    limit  = TENANT_RPS_LIMITS.get(tenant_id, 5)
    key    = f"rate:{tenant_id}"
    now    = time.time()
    window = 1.0

    pipe = redis_client.pipeline()
    pipe.zremrangebyscore(key, 0, now - window)   # drop old entries
    pipe.zadd(key, {str(now): now})               # add this request
    pipe.zcard(key)                               # count in window
    pipe.expire(key, 2)                           # TTL for cleanup
    results = pipe.execute()

    count = results[2]
    return count <= limit

The pipeline executes the four commands atomically — the zremrangebyscore, zadd, zcard, and expire run as a single Redis transaction. This is the pipeline atomicity that ensures no race condition between the old-entry removal and the new-entry count.

The rate limit was not a punishment. It was a reservation: each tenant was guaranteed exactly its allocated RPS, no more and no less, regardless of what was happening in other tenants' queues.

SQS per-tenant queues — one queue per tenant, named by tenant slug. When a request passed the rate check it was enqueued to the tenant's SQS queue. A pool of worker processes pulled from all six queues in weighted round-robin: the national retailer's queue was polled twice per cycle given its higher allocation; boutique tenants were polled once per cycle. A worker processing a national retailer query was not blocking a Vastralaya query from a different worker:

TENANT_QUEUES: dict[str, str] = {
    "national_retailer": os.environ["SQS_NATIONAL_RETAILER"],
    "vastralaya":        os.environ["SQS_VASTRALAYA"],
    "zudyog_fashion":    os.environ["SQS_ZUDYOG"],
    "surat_saree_house": os.environ["SQS_SURAT"],
    "jaipur_boutique":   os.environ["SQS_JAIPUR"],
    "kolkata_silks":     os.environ["SQS_KOLKATA"],
}

def enqueue_query(tenant_id: str, payload: dict) -> str:
    queue_url = TENANT_QUEUES[tenant_id]
    response  = sqs.send_message(QueueUrl=queue_url,
                                  MessageBody=json.dumps(payload))
    return response["MessageId"]

POLL_WEIGHTS: dict[str, int] = {
    "national_retailer": 2,   # polled twice per cycle
    "vastralaya":        1,
    "zudyog_fashion":    1,
    "surat_saree_house": 1,
    "jaipur_boutique":   1,
    "kolkata_silks":     1,
}

Gateway layer — checks rate limit before routing to queue:

@app.post("/query")
async def query_endpoint(request: Request):
    body      = await request.json()
    tenant_id = body.get("tenant_id")

    if not tenant_id or tenant_id not in TENANT_RPS_LIMITS:
        raise HTTPException(status_code=400, detail="Unknown tenant")

    if not check_rate_limit(tenant_id):
        return JSONResponse(status_code=429, content={
            "error": "Rate limit exceeded", "tenant": tenant_id,
            "limit": TENANT_RPS_LIMITS[tenant_id], "retry_after": 1
        })

    message_id = enqueue_query(tenant_id, body)
    return {"queued": True, "message_id": message_id}

The Valid Knowledge: What the Locust Load Test Found

The Locust load-test harness simulated the Diwali traffic pattern: 40 national retailer users and 5 Vastralaya users running concurrently, the national retailer count ramping from 10 to 40 over 90 seconds:

class NationalRetailerUser(HttpUser):
    weight    = 40
    wait_time = between(0.5, 1.5)
    @task
    def text_query(self):
        self.client.post("/query", json={
            "tenant_id":   "national_retailer",
            "query":       random.choice(NATIONAL_RETAILER_QUERIES),
            "customer_id": f"nr_test_{random.randint(1, 5000)}",
            "session_id":  f"sess_{random.randint(1, 5000)}",
        })

class VastralayaUser(HttpUser):
    weight    = 5
    wait_time = between(2, 5)
    @task
    def text_query(self):
        self.client.post("/query", json={
            "tenant_id":   "vastralaya",
            "query":       random.choice(VASTRALAYA_QUERIES),
            "customer_id": f"vas_test_{random.randint(1, 200)}",
            "session_id":  f"sess_{random.randint(1, 200)}",
        })

Baseline (no isolation):

Both tenants degraded. The national retailer's spike had saturated the shared container. Every Vastralaya timeout was a customer who sent a query and received nothing.

Isolated (rate limiting + per-tenant queues):

Vastralaya's numbers had not moved. The p50 went from 45ms at nominal to 51ms — six milliseconds of overhead from the SQS hop. The SLA was 200ms. The wall had held.

The national retailer's 1.2% error rate was 429 responses to requests that exceeded the rate limit — requests that would have timed out silently in the baseline now receiving an honest rejection the client could retry. The national retailer's performance was degraded. It was bounded.

The RAGAS scores for a sample of national retailer queries pulled from the audit log after the load test: Faithfulness 0.88, Context Precision 0.91 — identical to nominal traffic. Latency and correctness were independent variables. A query that waited 182 milliseconds in the queue received the same grounded answer as a query served immediately. The isolation protected latency. The grounding protected accuracy. Both held.

When the national retailer's simulated user count hit forty and its p50 crossed 180ms, Krishna asked the question he had opened the chapter with.

"What is Vastralaya's p99 right now?"

"One fifteen."

He capped the marker he had been holding since the session began — he had not written anything, only watched — and set it on the tray.

This article covers concepts from Book 4, Chapter 7 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to a fleet of six isolated, compliant, auditable tenants.