Vector Databases & Chunking

Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding LLMs in custom database knowledge. To implement RAG, documents must be processed, embedded into high-dimensional vectors, and stored in dedicated databases like Pinecone, Milvus, Qdrant, or Chroma.

The Importance of Chunking Strategies

LLMs have finite context windows. Feeding a 100-page PDF directly is slow and expensive. Therefore, we split documents into smaller chunks. The optimal strategy balances coherence against context limit:

• Fixed-size chunking: Splits at exact character counts (e.g. 500 characters). Simple but splits sentences mid-thought.
• Recursive character chunking: Evaluates markdown syntax, double newlines, single newlines, and space characters sequentially to find natural splits.
• Semantic chunking: Computes sentence-level embedding similarities and groups sentences together until similarity drops below a threshold.

// Recursive Chunking logic preview
function recursiveSplit(text, separators, maxChunkSize, overlap = 50) {
  let chunks = [];
  // Split on paragraph boundaries, sentences, or word spaces
  // then merge items back under maxChunkSize limits...
  return chunks;
}

Retrieval Metrics

Retrieved chunks are ranked using similarity functions like Cosine Similarity, Dot Product, or Euclidean Distance. Additionally, metadata filtering (e.g., matching client IDs or category tags) should be executed beforehand to narrow search spaces.