Vector Databases in Production: Selection and Optimization Guide

Vector databases power similarity search at the heart of RAG systems. Choosing the wrong one - or misconfiguring the right one - directly impacts LLM application quality and cost.

How Vector Search Works

Traditional databases compare values with equality or range operators. Vector search finds K-Nearest Neighbors by computing distance in high-dimensional embedding space:

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Exact KNN is O(n*d) per query
# 1M vectors * 1536 dims = 1.5B operations per query
# ANN algorithms trade small accuracy loss for massive speed

ANN Index Types

HNSW (Hierarchical Navigable Small World): Best for most use cases

• Query: O(log n), high recall (>95%)
• Memory: ~O(n * d * 4 bytes) - expensive

IVF (Inverted File Index): For huge datasets with memory constraints

• Quantizes vectors into clusters
• Query: O(sqrt(n)), lower recall (70-90%)
• Memory: 8-16x less than HNSW

PQ (Product Quantization): Compression

• Reduces 1536 float32 (6KB) to 96 bytes (64x compression)
• Combined IVF+PQ is the sweet spot for billion-scale

Pinecone (Managed Cloud)

Best for: Teams wanting zero infrastructure overhead.

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key=os.environ['PINECONE_API_KEY'])
pc.create_index(
    name='documents',
    dimension=1536,
    metric='cosine',
    spec=ServerlessSpec(cloud='aws', region='us-east-1')
)
index = pc.Index('documents')

# Upsert with metadata
index.upsert(vectors=[
    ('doc-001', embedding_vector, {
        'text': 'chunk text here',
        'source': 'handbook.pdf',
        'category': 'hr-policy'
    })
], namespace='production')

# Query with metadata filter
results = index.query(
    vector=query_embedding,
    top_k=10,
    filter={'category': {'$in': ['hr-policy', 'benefits']}},
    include_metadata=True,
    namespace='production'
)

Qdrant (Self-Hosted / Managed)

Best for: Cost efficiency at scale, advanced filtering.

from qdrant_client import QdrantClient
from qdrant_client.models import (
    Distance, VectorParams, PointStruct, HnswConfigDiff,
    Filter, FieldCondition, MatchValue, Range
)

client = QdrantClient(url='http://localhost:6333')

client.create_collection(
    collection_name='documents',
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
    hnsw_config=HnswConfigDiff(
        m=16,           # Connections per node
        ef_construct=200  # Build quality
    )
)

# Batch upsert
client.upsert(
    collection_name='documents',
    points=[PointStruct(id=i, vector=emb, payload={'text': txt})
            for i, (emb, txt) in enumerate(data)],
    wait=True
)

# Complex filtering
results = client.search(
    collection_name='documents',
    query_vector=query_embedding,
    limit=10,
    query_filter=Filter(
        must=[FieldCondition(key='category', match=MatchValue(value='technical'))],
        must_not=[FieldCondition(key='archived', match=MatchValue(value=True))]
    ),
    score_threshold=0.75
)

Chroma (Local / Simple RAG)

Best for: Development, prototyping, small-scale (<1M vectors).

import chromadb
from chromadb.utils import embedding_functions

client = chromadb.PersistentClient(path='./chroma_db')
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.environ['OPENAI_API_KEY'],
    model_name='text-embedding-3-small'
)
collection = client.get_or_create_collection(
    name='documents', embedding_function=openai_ef
)

# Add documents (auto-embeds)
collection.add(
    documents=['The refund policy allows returns within 30 days...'],
    metadatas=[{'source': 'policy.pdf', 'category': 'support'}],
    ids=['doc-001']
)

# Natural language query (auto-embeds)
results = collection.query(
    query_texts=['What is the return policy?'],
    n_results=5,
    where={'category': 'support'}
)

Hybrid Search: Dense + Sparse

Combine vector search with BM25 for better recall:

def hybrid_search(query, collection, bm25_index, alpha=0.7):
    dense = collection.query(query_texts=[query], n_results=20)
    sparse = bm25_index.get_scores(query.split())
    combined = {}
    for rank, doc_id in enumerate(dense['ids'][0]):
        combined[doc_id] = alpha / (rank + 60)
    for rank, (doc_id, _) in enumerate(
        sorted(sparse.items(), key=lambda x: x[1], reverse=True)[:20]
    ):
        combined[doc_id] = combined.get(doc_id, 0) + (1-alpha) / (rank + 60)
    return sorted(combined.items(), key=lambda x: x[1], reverse=True)[:10]

Selection Guide

Choose Pinecone for fast production deployment. Choose Qdrant for cost control and advanced filtering. Use Chroma for development.