ONNX INT8 Quantisation for Embedding Models: A Practical Guide

The fine-tuned PyTorch model is correct but not deployable. A 1.8GB container image and 41-second cold start are not acceptable for a chatbot API. ONNX export and INT8 quantisation reduce this to 33MB and 2ms per inference — without losing a meaningful point of retrieval quality.

The Valid Failure

After fine-tuning bge-small-en-v1.5 in Book 3, the team tried to deploy the PyTorch model directly to Railway.

Container image size: 1.8GB (PyTorch + CUDA dependencies). Railway free tier memory limit: 512MB. The container would not start.

On a paid Railway instance with 2GB RAM: cold start time 41 seconds. After 30 minutes of inactivity, Railway shut down the container. Every 30 minutes, the first customer to use ShopBot waited 41 seconds. Two customers filed support tickets thinking the site was broken.

PyTorch is the training environment. ONNX is the inference environment.

What ONNX Is

ONNX (Open Neural Network Exchange) is a format that represents a trained model as a computation graph, independent of the framework that trained it. Once exported to ONNX, a model can be run with the ONNX Runtime — a C++ inference engine that is significantly lighter than PyTorch.

PyTorch model (1.8GB, Python runtime required)
         ↓ export
ONNX FP32 model (90MB, PyTorch not required)
         ↓ quantise
ONNX INT8 model (33MB, pure C++ inference)

Step 1: Export to ONNX FP32

from sentence_transformers import SentenceTransformer
from optimum.onnxruntime import ORTModelForFeatureExtraction
from transformers import AutoTokenizer

model = SentenceTransformer("shopbot-bge-small-finetuned")

# Export using optimum (Hugging Face ONNX export library)
ort_model = ORTModelForFeatureExtraction.from_pretrained(
    "shopbot-bge-small-finetuned",
    export=True,
)
tokenizer = AutoTokenizer.from_pretrained("shopbot-bge-small-finetuned")
ort_model.save_pretrained("shopbot-bge-small-onnx-fp32")
tokenizer.save_pretrained("shopbot-bge-small-onnx-fp32")

After this step: model directory is 90MB. Inference does not require PyTorch.

RAGAS scores after FP32 export: identical to PyTorch. FP32 export is lossless — it is a format conversion, not a precision reduction.

Step 2: INT8 Dynamic Quantisation

INT8 quantisation converts 32-bit floating-point weights to 8-bit integers. The model is 4× smaller. Inference is faster (integer arithmetic is cheaper on CPU than floating-point). Some precision is lost.

from optimum.onnxruntime import ORTQuantizer
from optimum.onnxruntime.configuration import AutoQuantizationConfig

quantizer = ORTQuantizer.from_pretrained("shopbot-bge-small-onnx-fp32")

# Dynamic quantisation — no calibration data required
dqconfig = AutoQuantizationConfig.avx512_vnni(
    is_static=False,   # dynamic = no calibration dataset needed
    per_channel=False,
)

quantizer.quantize(
    save_dir="shopbot-bge-small-onnx-int8",
    quantization_config=dqconfig,
)

After this step: model is 33MB. Zero PyTorch dependencies in production.

Step 3: Run Inference with ONNX Runtime

import numpy as np
from transformers import AutoTokenizer
from onnxruntime import InferenceSession

tokenizer = AutoTokenizer.from_pretrained("shopbot-bge-small-onnx-int8")
session = InferenceSession("shopbot-bge-small-onnx-int8/model.onnx")

def mean_pooling(token_embeddings: np.ndarray, attention_mask: np.ndarray) -> np.ndarray:
    mask_expanded = attention_mask[:, :, np.newaxis].astype(float)
    sum_embeddings = np.sum(token_embeddings * mask_expanded, axis=1)
    sum_mask = np.clip(mask_expanded.sum(axis=1), a_min=1e-9, a_max=None)
    return sum_embeddings / sum_mask

def embed_onnx(text: str) -> np.ndarray:
    inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True, max_length=512)
    outputs = session.run(None, dict(inputs))
    embeddings = mean_pooling(outputs[0], inputs["attention_mask"])
    # L2-normalise
    norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
    return embeddings / norms

The Numbers

The INT8 quantisation cost: 1 point on Faithfulness, 1 point on Context Precision. In absolute terms: F 0.87→0.86, CP 0.91→0.90.

Is a 1-point drop acceptable? In Book 3's evaluation, this drop sat inside weekly RAGAS score noise — the score fluctuated by ±1–2 points week to week due to query variation. A 1-point reduction that was indistinguishable from evaluation noise, in exchange for 19× smaller deployment and 19× faster inference, was accepted.

Deployment Container Comparison

Before quantisation (PyTorch):

FROM python:3.11
RUN pip install torch sentence-transformers  # ~1.6GB
COPY model/ /app/model/                       # +200MB
# Total: ~1.8GB

After quantisation (ONNX INT8):

FROM python:3.11-slim
RUN pip install onnxruntime transformers      # ~150MB
COPY shopbot-bge-small-onnx-int8/ /app/model/ # +33MB
# Total: ~183MB — 90% smaller

Cold start: 0.8 seconds. Railway's inactivity restart is no longer a user-visible event.

The Valid Knowledge

• FP32 export is lossless: convert to ONNX FP32 first. Verify RAGAS scores are identical. Then quantise.
• Dynamic INT8 requires no calibration data: is_static=False means you do not need to collect representative inputs before quantisation. This makes the process fast and repeatable.
• Mean pooling + L2 normalisation must be reproduced: sentence-transformers handles this internally. When using ONNX Runtime directly, you must implement it yourself. Without L2 normalisation, cosine similarity scores will be incorrect.
• The 1-point drop is context-dependent: for a retrieval system where weekly RAGAS score noise is ±2 points, a 1-point drop from quantisation is within noise. For a high-stakes medical or legal system, evaluate on your domain before accepting the trade-off.
• ₹0 per embedding at any volume: the local ONNX model costs nothing to run. At ShopBot's 38,000 queries/day, this replaced ₹570/day in OpenAI embedding API calls.