The Immutable Audit Log: DynamoDB, 3 GSIs, IAM Deny, and the Query Event Schema

Sanjaya's message arrived on a Monday morning at nine forty-three.

"I ran a test session through the Vastralaya chatbot last Thursday between 14:00 and 14:30. I asked four queries — one text, one image, two text follow-ups. Can you pull the audit record for that session? I want to see which chunks were retrieved, which model version served each query, which path the LangGraph router took. If the audit trail is what the whiteboard says it is, this should take you less than five minutes."

The last sentence was not a provocation. It was Sanjaya being precise about what the whiteboard line had promised.

Arjun opened a terminal. He did not have five minutes.

The Valid Failure: Three Layers an Audit Trail Must Address

The audit trail had three requirements that had not yet been fully addressed.

Completeness. The deletion receipt from Chapter 3 wrote one event — the deletion itself — to DynamoDB. Every query needed its own event: customer hash, query fingerprint, chunks retrieved, model version, LangGraph path, query modality, timestamp. These fields were what Sanjaya's whiteboard line specified. They were not yet being written consistently for every query across every tenant.

Queryability. DynamoDB is fast at looking up a single item by primary key. It is not fast, without Global Secondary Indexes, at answering "show me all QUERY events for this customer hash between Thursday 14:00 and 14:30." That query required a GSI. The GSI needed to be defined before the first record was written — adding a GSI to a populated table is possible, but querying it before backfill completes returns partial results.

Immutability. DynamoDB does not prevent deletion by default. Any developer with dynamodb:DeleteItem permission on the audit table could erase a record. The table was append-only by convention, not enforcement. If the convention were violated — a cleanup script against the wrong table, an incident response that treated the audit log as application data — the record was gone. An audit trail is only auditable if its contents are guaranteed intact.

Why CloudWatch logs failed: CloudWatch log groups have a configurable retention period. The default is thirty days. GDPR Article 30 — Records of Processing Activities — calls for records to be maintained for the duration of the processing activity, years not days. The default could be overridden, but the override was a configuration value that could be changed back. CloudWatch logs can also be deleted — DeleteLogGroup and DeleteLogStream exist. Retention was configurable; mutability was built in. Appropriate for operational debugging, not for the regulator who arrives six months later.

Why S3 with Object Lock failed: S3 Object Lock prevents deletion and overwriting — genuine immutability. The queryability was not genuine. A regulator asking "show me every query this customer made between these dates" required either Athena (thirty to ninety second cold start) or a custom indexing layer on top of S3. That was two systems where one was needed, and the index could drift from the source.

The Valid Source: DynamoDB with Three GSIs and IAM Deny

Table definition with three Global Secondary Indexes:

dynamodb.create_table(
    TableName="shopbot_audit_log",
    AttributeDefinitions=[
        {"AttributeName": "event_id",        "AttributeType": "S"},
        {"AttributeName": "timestamp",        "AttributeType": "S"},
        {"AttributeName": "tenant_id",        "AttributeType": "S"},
        {"AttributeName": "customer_id_hash", "AttributeType": "S"},
        {"AttributeName": "event_type",       "AttributeType": "S"},
    ],
    KeySchema=[
        {"AttributeName": "event_id",   "KeyType": "HASH"},
        {"AttributeName": "timestamp",  "KeyType": "RANGE"},
    ],
    GlobalSecondaryIndexes=[
        {
            "IndexName": "tenant-time-index",
            "KeySchema": [
                {"AttributeName": "tenant_id",  "KeyType": "HASH"},
                {"AttributeName": "timestamp",  "KeyType": "RANGE"},
            ],
            "Projection": {"ProjectionType": "ALL"},
        },
        {
            "IndexName": "customer-hash-index",
            "KeySchema": [
                {"AttributeName": "customer_id_hash", "KeyType": "HASH"},
                {"AttributeName": "timestamp",         "KeyType": "RANGE"},
            ],
            "Projection": {"ProjectionType": "ALL"},
        },
        {
            "IndexName": "event-type-index",
            "KeySchema": [
                {"AttributeName": "event_type", "KeyType": "HASH"},
                {"AttributeName": "timestamp",  "KeyType": "RANGE"},
            ],
            "Projection": {"ProjectionType": "ALL"},
        },
    ],
    BillingMode="PAY_PER_REQUEST",
)

tenant-time-index answered show me all events for this tenant in this time window — the operational monitoring query. customer-hash-index answered show me all events for this customer hash — the regulatory inspection query. event-type-index answered show me all GDPR deletions in the last month — the compliance reporting query.

IAM Deny on DeleteItem — attached to every role with table access. Effect: Deny overrides any Allow elsewhere — including AdministratorAccess:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AuditLogImmutability",
      "Effect": "Deny",
      "Action": [
        "dynamodb:DeleteItem",
        "dynamodb:UpdateItem"
      ],
      "Resource": "arn:aws:dynamodb:ap-south-1:*:table/shopbot_audit_log"
    }
  ]
}

A developer with full AWS admin rights could not delete an audit record unless the IAM policy itself was first modified. Modifying the IAM policy was itself an auditable event in AWS CloudTrail. The immutability was not a convention. It was an enforcement chain.

Query event schema — written at the end of every query path, after the LLM responded, regardless of whether the query had succeeded or failed:

def write_query_event(
    tenant_id: str, customer_id: str,
    query_text: str | None, session_id: str,
    retrieved_chunk_ids: list[str],
    langgraph_path: str, model_version: str,
    query_modality: str = "text",   # "text" | "image"
) -> str:
    event_id = f"evt_{uuid.uuid4().hex}"
    ts       = datetime.utcnow().isoformat() + "Z"

    item = {
        "event_id":          {"S": event_id},
        "event_type":        {"S": "QUERY"},
        "tenant_id":         {"S": tenant_id},
        "customer_id_hash":  {"S": audit_customer_id(tenant_id, customer_id)},
        "query_fingerprint": {"S": audit_query_fingerprint(
                                        tenant_id, customer_id,
                                        query_text or "", session_id)},
        "retrieved_chunks":  {"SS": retrieved_chunk_ids},
        "langgraph_path":    {"S": langgraph_path},
        "model_version":     {"S": model_version},
        "query_modality":    {"S": query_modality},
        "timestamp":         {"S": ts},
    }
    dynamodb.put_item(TableName="shopbot_audit_log", Item=item)
    return event_id

For image queries, query_text was None; the fingerprint was computed over the session ID and a constant "IMAGE_QUERY" marker — preserving uniqueness without storing the image. The writer added fewer than two milliseconds to query latency on average.

The Valid Knowledge: Four Events, Seven Minutes

Arjun computed the customer ID hash for Sanjaya's test account and queried customer-hash-index. The report:

{
  "report_generated": "2024-12-09T09:51:17Z",
  "tenant": "vastralaya",
  "query_count": 4,
  "events": [
    { "timestamp": "2024-12-05T14:03:22Z", "modality": "text",
      "chunks": ["VAS-2847-desc", "VAS-1923-desc", "VAS-3301-desc"],
      "model_version": "bge-small-vastralaya-v1", "path": "catalog_lookup" },
    { "timestamp": "2024-12-05T14:07:45Z", "modality": "image",
      "chunks": ["VAS-2847-img", "VAS-0934-img", "VAS-1102-img"],
      "model_version": "clip-vit-base-patch32", "path": "image_lookup" },
    { "timestamp": "2024-12-05T14:11:03Z", "modality": "text",
      "chunks": ["VAS-2847-desc", "VAS-2847-care", "VAS-2847-size"],
      "model_version": "bge-small-vastralaya-v1", "path": "catalog_lookup → detail_lookup" },
    { "timestamp": "2024-12-05T14:18:29Z", "modality": "text",
      "chunks": ["VAS-2847-size", "VAS-2847-care"],
      "model_version": "bge-small-vastralaya-v1", "path": "catalog_lookup → detail_lookup" }
  ]
}

Four queries. Four events. Every field the whiteboard line had specified. The report took seven minutes — two minutes over Sanjaya's five-minute benchmark, most of it waiting for the GSI to confirm it had finished backfilling the previous week's records.

Sanjaya's reply arrived forty minutes later:

"The four events are correct. The timestamps match. The chunk IDs match what I retrieved manually from the Vastralaya collection."

"The model version for the image query shows clip-vit-base-patch32, not the Vastralaya fine-tune. This is accurate — we haven't yet fine-tuned CLIP for Vastralaya. The record shows what actually happened, not what we would have preferred."

"This is sufficient."

A system that records what actually happened rather than what would have been preferable is the only kind of system whose audit trail is worth inspecting.

This article covers concepts from Book 4, Chapter 6 of the RAG Mastery Series. The series builds ShopBot — a production RAG system — across four books, from first embeddings to a fleet of six isolated, compliant, auditable tenants.