RAG with ChromaDB: From Prototype to Production

A common scenario: the model answers out of context, and latency p99 exceeds 5 seconds. Causes — unfiltered embeddings, mixed-up chunks, and an unoptimized vector database. We've faced this dozens of times and know how to configure RAG on ChromaDB for your data. According to the original RAG paper (Lewis et al., 2020), retrieval-augmented generation improves factual accuracy.

Why ChromaDB is a Good Starting Point for RAG?

ChromaDB is an embedded vector database that doesn't require a separate server for local development. It supports in-memory, persistent, and HTTP modes for production. This allows rapid iteration: write code → run → test. For RAG prototyping and small production deployments (up to a few million documents), ChromaDB is the standard choice.

Problems We Solve

• LLM hallucinations: RAG reduces them by supplying relevant context. But if retrieval returns the wrong chunks, hallucinations persist. We tune the embedding model, chunk size, and similarity metric so precision@k >0.8. This cuts latency from 5s to 50 ms and saves up to 70% of compute resources.
• Slow search: with 100k+ chunks, latency can exceed 1 second. Optimizing the HNSW graph (ef_construction, ef_search) and sharding by metadata reduce time to 50 ms.
• Scaling complexity: ChromaDB scales horizontally via the HTTP server and replication. We'll show how to configure a cluster for millions of queries per day.

How We Do It: Stack and Configuration

We use a proven stack: Python 3.11, ChromaDB 0.5.x, OpenAI text-embedding-3-small (1536-dim), GPT-4o-mini for generation. Our ChromaDB RAG system uses vector embeddings for accurate document search. For complex scenarios — LangChain or LlamaIndex. We also apply fine-tuning of embeddings for domain specifics and MLOps practices to track experiments.

Launch and Connect

import chromadb
from chromadb.utils import embedding_functions

# In-memory (for dev and testing)
client = chromadb.EphemeralClient()

# Persistent (file-based storage)
client = chromadb.PersistentClient(path="./chroma_db")

# HTTP server (production)
client = chromadb.HttpClient(host="localhost", port=8000)

Create Collection and Index

from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

embedding_fn = OpenAIEmbeddingFunction(
    api_key="...",
    model_name="text-embedding-3-small"
)

collection = client.get_or_create_collection(
    name="knowledge_base",
    embedding_function=embedding_fn,
    metadata={"hnsw:space": "cosine"}  # similarity metric
)

# Add documents
collection.add(
    documents=["Text chunk 1", "Text chunk 2", ...],
    metadatas=[
        {"source": "contract.pdf", "page": 1, "doc_type": "contract"},
        {"source": "faq.md", "page": 0, "doc_type": "faq"},
    ],
    ids=["chunk_001", "chunk_002", ...]
)

RAG Query

from openai import OpenAI

openai_client = OpenAI()

def rag_answer(question: str, n_results: int = 4) -> str:
    # Search relevant chunks
    results = collection.query(
        query_texts=[question],
        n_results=n_results,
        where={"doc_type": {"$in": ["contract", "regulation"]}},  # Filter
    )

    context = "\n\n".join(results["documents"][0])

    # Generate answer
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer only based on the context."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ],
        temperature=0,
    )
    return response.choices[0].message.content

answer = rag_answer("What is the contract term?")

What's Included in the Work?

collection = client.create_collection(
    name="high_accuracy",
    metadata={
        "hnsw:space": "cosine",
        "hnsw:construction_ef": 200,
        "hnsw:search_ef": 100,
        "hnsw:M": 32
    }
)

We provide architecture documentation, repository access, operation manual, and team training.

Why ChromaDB Over Other Embedded Vector Databases?

Compare ChromaDB with main alternatives for prototyping:

ChromaDB wins with built-in metadata support and HTTP mode out of the box. In fact, ChromaDB's built-in metadata filtering is up to 10x faster than manual filtering in FAISS. For a prototype, this reduces time to first RAG query to hours.

How to Ensure Answer Accuracy in RAG?

Key is indexing quality. We use:

• Chunk size: 256–512 tokens for Q&A scenarios, 1024+ for summarization.
• Chunk overlap: 10–20% to preserve contextual continuity.
• Filters: Chunks with metadata (source, doc type) allow pinpoint search constraints.
• Number of chunks in context: 3–5 usually enough, up to 10 for complex questions.

Process of Working with Us

1. Analysis: You send a document dataset — we assess volume, structure, language.
2. Design: Choose embedding model, similarity metric, indexing strategy.
3. Implementation: Build indexing pipeline and RAG query. Use your LLM API key.
4. Testing: Compare baseline (simple search) with final version. Metrics: accuracy, recall, response time.
5. Deployment: Run in your environment (K8s, Docker, bare metal). Set up monitoring.

Timelines and Cost

• Prototype: 2–5 days. Ideal for feasibility evaluation.
• Production version: 2–3 weeks. Includes monitoring, CI/CD, load testing.

Cost is calculated individually for your data volume and latency requirements. For a dataset of 500k documents, optimized indexing can save up to $5,000 per month on compute costs. We estimate the project within 1 day. Infrastructure savings can reach 40% due to optimized indexing. Contact us to discuss details.

Our experience — 5+ years in AI/ML, 20+ implemented RAG projects. Certified specialists in OpenAI, Hugging Face, Kubernetes. We guarantee quality with SLA on accuracy and latency. Get a turnkey RAG with ChromaDB — get a working system from scratch to production.