How to Fine-Tune bge-small-en for a Custom Domain

A general-purpose embedding model was trained on web text, not on fashion product catalogs. It does not know that "mirrorwork" and "mirror-work" are the same thing, or that "voile" clusters near "breathable summer fabric" in the Indian fashion market. Fine-tuning teaches it your domain.

The Valid Failure

After Book 2, ShopBot's embedding model was sentence-transformers/all-MiniLM-L6-v2. It was good at general semantic similarity. It was not good at the specific vocabulary of Indian fashion e-commerce:

• "Supplex performance fabric" → low similarity to "quick-dry sportswear" (it had no training exposure to that brand name)
• "mirrorwork lehenga" → low similarity to "mirror embroidered lehenga" (hyphenation variation)
• "chikankari kurta" → low similarity to "Lucknowi embroidery kurta" (regional terminology)

These were not retrieval threshold problems. Lowering the threshold returned unrelated chunks. These were embedding model problems: the model had not learned the vocabulary patterns of this domain.

The team considered switching to text-embedding-3-large (OpenAI, 3072 dimensions, API cost). Cost at ShopBot's production volume: ₹0.015 per query × 38,000 queries/day = ₹570/day. The fine-tuned bge-small-en-v1.5 at 384 dimensions on ONNX: ₹0 per query, 2ms latency.

Why bge-small-en-v1.5

BAAI/bge-small-en-v1.5 is the fine-tuning target for three reasons:

1. 384 dimensions — small enough for fast inference, large enough for domain discrimination
2. Architecture designed for fine-tuning — trained with a contrastive objective that makes triplet fine-tuning straightforward
3. Strong baseline — top-tier on MTEB benchmarks before any fine-tuning; the fine-tuning adds margin, not capability

Step 1: Build Training Triplets

Fine-tuning with triplet loss requires triplets: (anchor, positive, negative).

• Anchor: a query phrased as a user would ask it
• Positive: the correct chunk that answers the anchor
• Negative: a chunk that seems relevant but is not the right answer

# Example triplets for a fashion catalog
triplets = [
    {
        "anchor": "lightweight fabric for hot weather",
        "positive": "Cotton voile — featherweight and breathable, ideal for summer wear. Skin-friendly and quick to dry.",
        "negative": "Warm woollen shawl in winter hues — perfect for cold evenings and festive occasions.",
    },
    {
        "anchor": "mirrorwork lehenga",
        "positive": "Mirror embroidered lehenga in ivory — intricate shisha mirror work on georgette. Festive occasion wear.",
        "negative": "Cotton printed anarkali — floral block print, suitable for casual daywear.",
    },
]

The team generated 3,480 triplets from 100 manually labeled queries by:

1. Taking each labeled query and its correct chunk (100 anchor-positive pairs)
2. Generating 34–35 paraphrase variants per query using GPT-4o-mini
3. Hard-mining negatives: finding chunks that scored 0.65–0.80 (near-misses, not obvious negatives)

Step 2: Fine-Tune with sentence-transformers

from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

train_examples = [
    InputExample(texts=[t["anchor"], t["positive"], t["negative"]])
    for t in triplets
]

train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=32)

train_loss = losses.TripletLoss(
    model=model,
    distance_metric=losses.TripletDistanceMetric.COSINE,
    triplet_margin=0.5,
)

model.fit(
    train_objectives=[(train_dataloader, train_loss)],
    epochs=3,
    warmup_steps=100,
    output_path="shopbot-bge-small-finetuned",
)

Training time on an M2 MacBook Pro (CPU): 47 minutes for 3,480 triplets × 3 epochs. On a free Google Colab GPU: 8 minutes.

Step 3: Evaluate Before Committing

Before re-ingesting the catalog with the new model, compare retrieval scores on a held-out evaluation set:

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

base_model = SentenceTransformer("BAAI/bge-small-en-v1.5")
finetuned_model = SentenceTransformer("shopbot-bge-small-finetuned")

eval_pairs = [
    ("lightweight fabric for hot weather", "Cotton voile — featherweight and breathable"),
    ("mirrorwork lehenga", "Mirror embroidered lehenga in ivory"),
    ("chikankari kurta", "Lucknowi embroidery kurta — hand-crafted white threadwork"),
]

for query, target in eval_pairs:
    base_score = cosine_similarity(
        [base_model.encode(query)], [base_model.encode(target)]
    )[0][0]
    ft_score = cosine_similarity(
        [finetuned_model.encode(query)], [finetuned_model.encode(target)]
    )[0][0]
    print(f"Base: {base_score:.3f} | Fine-tuned: {ft_score:.3f} | {query}")

Results from Book 3:

Every query that was failing or borderline on the base model is above threshold on the fine-tuned model.

Step 4: Re-Ingest the Catalog

The fine-tuned model produces different vectors than the base model. You must re-ingest all chunks:

# Re-ingest with fine-tuned model
finetuned_model = SentenceTransformer("shopbot-bge-small-finetuned")

client.recreate_collection(
    collection_name=COLLECTION,
    vectors_config=VectorParams(size=384, distance=Distance.COSINE),
)

for chunk in all_chunks:
    vector = finetuned_model.encode(chunk["text"]).tolist()
    client.upsert(collection_name=COLLECTION, points=[...])

Re-ingestion of 850 products × ~4 chunks each = 3,400 chunks. Time: 94 seconds.

RAGAS After Fine-Tuning

Context Recall improved most significantly — the fine-tuned model retrieved relevant chunks that the base model had missed on domain-specific vocabulary.

The Valid Knowledge

• 3,480 triplets from 100 labeled queries is sufficient: you do not need thousands of labeled queries. You need enough to cover the vocabulary patterns specific to your domain.
• Hard negatives are more valuable than random negatives: triplets where the negative scores 0.65–0.80 (near-miss) teach the model finer discrimination than triplets where the negative scores 0.20 (obvious wrong answer).
• Re-ingest after fine-tuning: the fine-tuned model and base model produce incomparable vectors. Mixing them in the same collection produces broken retrieval.
• Evaluate before re-ingesting: if the fine-tuned model does not improve scores on the held-out eval pairs, do not re-ingest. Check the training data quality.
• ₹0 per query vs ₹0.015 per query: the fine-tuned local model costs nothing to run after training. At production volume, this is the most significant operational cost reduction in the entire Book 3 arc.