ভেক্টর ডাটাবেস

১ million document এর মধ্যে ৫ টা most similar বের করতে হবে — naive cosine similarity তে ১ million calculation। 100 million এ? Impossible। Vector Database এর জন্ম এই সমস্যা থেকে — ANN (Approximate Nearest Neighbor) algorithm দিয়ে millisecond এ billion-scale similarity search। Pinecone, Chroma, FAISS, Qdrant, pgvector — RAG এর storage backbone।

Vector Database = vectors store + ANN index দিয়ে fast similarity search এর জন্য optimized database। Key algorithm: HNSW (Hierarchical Navigable Small World), IVF (Inverted File), Annoy। Trade-off: 100% accuracy বনাম speed (ANN ~99% accuracy দিয়ে 1000x faster)। Modern vector DB hybrid search (vector + keyword), metadata filter, multi-tenancy support।

ভাবুন একটা library — naive search মানে প্রতিটা বই pick করে compare। ANN মানে আগে থেকেই বই গুলোকে similar group এ সাজানো — query এলে শুধু সঠিক group এ search। HNSW graph structure বানায় যেখানে similar vector একসাথে connected — query থেকে graph traverse করে quickly nearest neighbor পৌঁছায়।

• Pinecone — managed, easy, expensive।
• Chroma — open source, embedded, dev-friendly।
• Qdrant — Rust-based, high performance।
• FAISS — Meta এর library, on-device।
• pgvector — Postgres extension, RAG এর easy choice।
• Weaviate — schema + hybrid search strong।

import chromadb
import os
from openai import OpenAI

client_ai = OpenAI(
    base_url="https://ai.gateway.lovable.dev/v1",
    api_key=os.environ["LOVABLE_API_KEY"],
)

def embed(texts):
    res = client_ai.embeddings.create(model="google/gemini-embedding-001", input=texts)
    return [d.embedding for d in res.data]

chroma = chromadb.Client()
collection = chroma.create_collection(name="articles")

docs = [
    "Python is a popular programming language for AI and data science.",
    "FastAPI is a modern Python web framework with automatic OpenAPI docs.",
    "Machine learning models predict outcomes from labeled training data.",
    "Bangladesh cricket team won an exciting match against India yesterday.",
    "The Tokyo Olympics featured swimming, athletics, and gymnastics events.",
]
ids = [f"doc_{i}" for i in range(len(docs))]
metadatas = [{"topic": "tech" if i < 3 else "sports"} for i in range(len(docs))]

collection.add(documents=docs, embeddings=embed(docs), ids=ids, metadatas=metadatas)

query = "Tell me about AI frameworks"
results = collection.query(
    query_embeddings=embed([query]),
    n_results=3,
    where={"topic": "tech"},
)

for doc, dist in zip(results["documents"][0], results["distances"][0]):
    print(f"[{dist:.3f}] {doc}")

• Embed model mismatch — query এ আলাদা model।
• Metric mix — cosine vector কে L2 দিয়ে search।
• Index parameter (ef, M) tune না করা — slow query।
• Backup না নেওয়া — embedding রি-compute expensive।
• Single-tenant DB কে multi-tenant এর জন্য use।

1. Chroma, FAISS, pgvector একই dataset এ benchmark।
2. Metadata filter (date range, category) সহ search।
3. HNSW vs flat index — speed/accuracy tradeoff measure।
4. Hybrid search (BM25 + vector) implement করুন।

• Vector DB = ANN-optimized vector storage + search।
• HNSW, IVF — popular algorithm।
• Choices: pgvector, Chroma, Pinecone, Qdrant, FAISS।
• RAG, semantic search, recommendation — সবার storage backbone।
• Hybrid search (vector + keyword) modern standard।