RAG & Vector Search

Naive string comparison:

  query: "cat"
  doc_1: "cat" → string match: 0% (different strings)
  doc_2: "kitten" → string match: 0% (different strings)
  doc_3: "cat" → string match: 100% (match)
  doc_4: "pet" → string match: 0% (although the meaning is close!)

Problem: semantic proximity ≠ lexical match

Evolution of solutions:
  1960s Boolean search (AND/OR/NOT keywords)
  1970s Bag of Words → word frequency vector
  1980s TF-IDF → importance weighting
  2000s LSA/LSI → hidden semantics via SVD
  2013 Word2Vec → neural word embeddings
  2018 BERT → contextual offer embeddings
  2020+ Bi-Encoder models → embeddings for search (E5, BGE, nomic)

Body of 2 documents:
  d1: "the cat is sitting at home"
  d2: "the dog is running home"

Vocabulary (vocab):
  {cat:0, sitting:1, at home:2, dog:3, running:4, home:5}

Vectors (BoW):
       cat sitting at home dog running home
  d1: [ 2, 1, 1, 0, 0, 0 ]
  d2: [ 0, 0, 0, 1, 1, 1 ]

query: "the cat is running"
  q: [ 1, 0, 0, 0, 1, 0 ]

Matches with d1: "cat" → 1
Matches with d2: "runs" → 1
→ draw. But "the cat is running" is semantically closer to d1?

TF(word, document) = count(word in doc) / len(doc)
IDF(word) = log( N / df(word) )
  N = number of documents in the corpus
  df = in how many documents the word appears

TF-IDF(word, doc) = TF × IDF

Example, corpus = 1000 documents:
  word "and": df=1000 → IDF=log(1000/1000)=0
  word "algorithm": df=50 → IDF=log(1000/50) =3.0
  word "catboost": df=3 → IDF=log(1000/3) =5.8

→ a rare and specific word receives high weight
→ stop words will automatically be zeroed

BM25(q, d) = Σ_{word∈q} IDF(word) ·
  TF(word,d) · (k₁+1)
  ─────────────────────────────────
  TF(word,d) + k₁·(1-b+b·|d|/avgdl)

k₁ ∈ [1.2, 2.0] – TF saturation
b ∈ [0, 1] - length normalization (usually 0.75)
|d| = document length, avgdl = average length

Key properties:
✓ TF saturation: 100 mentions ≠ 10× more important than 10
✓ long docs do not automatically dominate
✓ fast: inverted index → O(|q|) search

Imagine a 2D space (actually 768–3072 dimensions):

  X-axis → “animal/non-animal”
  Y-axis → domestic/wild

                       homemade
                          ▲
          cat [0.8, 0.9] ●
          cat [0.7, 0.9] ● dog [0.6, 0.8] ●
                          │
  ──────────────────── ───────────────────── animal
                          │
          lion [0.9, 0.1] ● wolf [0.8, 0.1] ●
                          │
                         wild

  table [-0.9, 0.0] ● ← far from animals
  algorithm [-0.8, 0.0] ●

query: "pet"
  embedding ≈ [0.75, 0.88]
  → nearest points: cat, cat, dog
  → although the word "pet" does not appear in these documents!
  → this is the power of semantic search

Task: predict neighbors based on the central word

  Window = 2:
  "the big [cat] is sitting on the sofa"
  → predict: "big", "sitting" from the word "cat"

Neural network (2 layers):
  input: one-hot(cat) ∈ ℝ^vocab_size
  hidden: W_embed · one-hot → ℝ^300 ← embedding!
  output: softmax → P(each word as a neighbor)

After training on a billion tokens:
  king − man + woman ≈ queen
  → the arithmetic of meanings works!

Word2Vec problem:
  "onion" (vegetable) = "onion" (weapon)
  → one word = one vector, context not taken into account

Word2Vec (static):
  "bow" → [0.3, -0.5, 0.8, …] ← always one vector

BERT (contextual):
  "bought onions for soup" → [0.7, 0.2, -0.1, …]
  “took a bow and arrows” → [-0.4, 0.8, 0.3, …]
  ← different vectors for one word!

Bi-Encoder for search (Sentence-BERT, E5, BGE):

  ENCODER ENCODER
     ↑ ↑
  query document
  "price of a cat" "cost of a kitten"

  q_emb = [0.2, 0.8, …] ∈ ℝ^768
  d_emb = [0.3, 0.7, …] ∈ ℝ^768

  score = cosine(q_emb, d_emb) = 0.91 ← high!

Entry: "the cat is sitting at home"
Tokens: [CLS] cat sitting at home [SEP]
Model output: vector for each token ℝ^768

[CLS] pooling (BERT-style):
  doc_emb = hidden_state[CLS]
  ↑ a special token is trained to aggregate meaning
  ↑ used in BERT fine-tuned models

Mean pooling (Sentence-BERT, E5):
  doc_emb = mean(hidden_states[1:-1])
  ↑ average for all tokens (without CLS and SEP)
  ↑ in practice it is better for long documents

Weighted mean pooling:
  ↑ tokens with attention weights are strengthened
  ↑ used in some modern models

The result of any pooling:
  doc_emb ∈ ℝ^768 ← one vector per document

Training batch (triplet loss):

  anchor (query): "How to treat a cat for a cold?"
  positive (relevant): "Treatment of cats with ARVI includes..."
  negative (not rel): "Recipe for borscht with beef"

Learning Objective:
  cosine(anchor, positive) → maximum high (~1.0)
  cosine(anchor, negative) → lowest possible (~0.0 or negative)

Triple Loss:
  L = max(0, cosine(a,n) − cosine(a,p) + margin)
  ↑ penalty if negative is closer to positive than by margin

More modern - InfoNCE / NT-Xent (in-batch negatives):
  for each query: all other documents in the batch = negatives
  batch_size=256 → 255 negatives for free
  → modeli: E5, BGE, nomic-embed, text-embedding-3

#E5-instruct:
query_emb = encode(
  "query: " + "how to treat a cat for a cold"
)
doc_emb = encode(
  "passage: " + "For viral infections in cats..."
)

# nomic-embed-text:
query_emb = encode("search_query: " + text)
doc_emb = encode("search_document: " + text)

# Without the correct prefix → quality degradation
# Read the documentation for each model!

Asymmetric matryoshka (OpenAI text-embedding-3):
→ you can trim the vector to the desired length
   3072 → 1536 → 512 → 256 without retraining

In a body of 100M sentences:

"my cat meows" meets with the same neighbors as
"my cat is meowing"

→ contrastive loss: these two phrases = positive pair
→ gradient descent moves their vectors closer

Likewise:
"cost of goods" ↔ "price of goods" → close
"acquire" ↔ "buy" → close
"expensive" ↔ "cheap" → FAR
                                            (antonyms)

Bottom line: vector space is
“map of meanings” learned from language statistics

Formula:
  cosine(A, B) = (A B) / (|A| × |B|)
             = Σᵢ(Aᵢ×Bᵢ) / (√Σᵢ Aᵢ² × √Σᵢ Bᵢ²)

Range: -1 (opposite) to +1 (identical)
For normalized vectors: cosine = dot product

Example (simplified, 3 dimensions):
  query: q = [0.6, 0.8, 0.0] |q| = 1.0
  doc_1: d1 = [0.5, 0.7, 0.1] |d1| = 0.87
  doc_2: d2 = [0.1, 0.1, 0.9] |d2| = 0.91
  doc_3: d3 = [0.6, 0.8, 0.0] |d3| = 1.0

  cosine(q, d1) = (0.6×0.5 + 0.8×0.7 + 0.0×0.1) / (1.0×0.87)
               = (0.30 + 0.56 + 0.00) / 0.87 = 0.874 ← relevant

  cosine(q, d2) = (0.6×0.1 + 0.8×0.1 + 0.0×0.9) / (1.0×0.91)
               = (0.06 + 0.08 + 0.00) / 0.91 = 0.154 ← irrelevant

  cosine(q, d3) = (0.6×0.6 + 0.8×0.8 + 0.0×0.0) / (1.0×1.0)
               = (0.36 + 0.64 + 0.00) / 1.0 = 1.000 ← identical

Result: we rank the documents: d3 > d1 > d2
Return top-k → this is the nearest neighbor search

Exact KNN (brute force):
  for doc in all_docs:
    score = cosine(query_emb, doc_emb)
  return top_k(scores)

  n=1M docs, d=768 → 768M multiplications per request
  → ~2-5 seconds on CPU. Unacceptable.

ANN (HNSW, IVF, ScaNN, FAISS):
  → build the index in advance (~minutes/hours)
  → search: ~1-10ms even for 100M vectors
  → recall@10 = 95-99% (we skip 1-5% of accurate answers)

Trade-off:
  accuracy ↑ ↔ speed ↓
  memory ↑ ↔ accuracy ↑

HNSW structure (simplified):

Level 2 (few nodes - “high-speed highway”):
  A ──────────────────── E
  ↑ long jumps, rough navigation

Level 1 (more nodes):
  A ──── B ──── C ──── D ──── E
  ↑ more detailed search

Level 0 (all nodes - exact search):
  A─B─C─D─E─F─G─H─I─J─K─L─M
  ↑ all documents, each connected to the k closest ones

Search Query Q:
  1. Enter the top level → find the nearest node
  2. Go down one level → clarify
  3. At level 0 → bypass nearest neighbors
  4. Return top-k

Parameters:
  M - number of connections per node (16–64)
  ef_construction — accuracy during construction
  ef_search — search accuracy (recall/speed)

Strategies:

① Fixed-size (naive):
  chunk_size = 512 tokens, overlap = 64
  → fast, does not take into account structure
  → can cut a sentence in the middle

② Sentence/Paragraph splitting:
  split by "\n\n" or punkt sentence tokenizer
  → each chunk = semantic unit
  → unequal length (from 20 to 400 tokens)

③ Semantic chunking:
  embed each sentence → find “gaps” by cosine distance
  → chunks = thematically homogeneous pieces
  → expensive (every offer needs to be embedded)

④ Hierarchical (Parent-Child):
  Small chunks for searching (128 tokens) → exact match
  Large chunks for context (512 tokens) → rich response
  → search for small ones, return large ones

⑤ Document structure-aware:
  Markdown h1/h2/h3 → sections as chunks
  PDF → paragraphs from layout
  → best result for structured docs

{
  # Identification
  "doc_id": "wb-seller-guide-2024",
  "chunk_id": "wb-seller-guide-2024-ch-047",

  # Document navigation
  "title": "WB Seller's Guide",
  "section": "3.2 Pricing",
  "page": 47,

  # Filtering
  "category": "pricing",
  "language": "ru",
  "date": "2024-11-01",

  # Content
  "text": "When calculating the price, take into account...",
  "summary": "About WB pricing methods"
}

# 1. Prepend title into chunk text before embedding:
text_for_embedding = f"""
Section: {section_title}
Document: {doc_title}

{chunk_text}"""
# → title affects embedding through pooling

# 2. Separate “title embedding” + weighted sum:
emb_title = encode(title)
emb_body = encode(body)
emb_final = 0.4 emb_title + 0.6 emb_body
# → explicit weighting

#3. HyDE (Hypothetical Document Embedding):
# query → LLM generates a hypothetical answer
# embed a hypothetical response, not a request
# → embedding is already in the “document space”

User request:
  "How to calculate the WB commission when selling clothes?"

① Encode query:
  q_emb = embed_model.encode(
    "query: How to calculate WB commission when selling clothes?"
  )
  q_emb ∈ ℝ^768 ← specific vector of 768 numbers

② Chunks are stored in the vector database:
  chunk_42: "Wildberries commission is 12-25%..."
    emb_42 = [0.31, -0.12, 0.78, …] (768 numbers)
    cosine(q, emb_42) = 0.891 ← HIGH

  chunk_91: "Returning goods to Wildberries: instructions..."
    emb_91 = [0.15, 0.44, -0.21, …]
    cosine(q, emb_91) = 0.543 ← average

  chunk_3: "Italian pasta recipes"
    emb_3 = [-0.67, 0.03, 0.11, …]
    cosine(q, emb_3) = 0.041 ← LOW

③ ANN search → top-3:
  rank 1: chunk_42 → score 0.891 ← WINNER
  rank 2: chunk_91 → score 0.543
  rank 3: chunk_3 → score 0.041

Why chunk_42 won:
  ✓ contains the words "commission", "Wildberries" → similar context
  ✓ chunk title: “Tariffs and commissions WB” was in the embedding
  ✓ clothing is mentioned as a category → vector close to query

Blind spots of each method:

BM25 defeats Dense when:
  query: "article WB 187234561"
  → exact number - dense embedding "averages" it
  → BM25 finds an exact match using the inverted index

  query: "Government Decree No. 44-FZ"
  → rare specific terms → BM25 wins

Dense defeats BM25 when:
  query: "how to reduce storage costs"
  doc: "methods for reducing warehouse costs"
  → there is not a single common word!
  → BM25: score = 0. Dense: cosine = 0.82

  query: "pet"
  doc: "pet care"
  → synonymy → only Dense will find

Hybrid search:
  score_final = α·BM25_score + (1-α)·dense_score
  → or via RRF (Reciprocal Rank Fusion)
  → consistently better than both separately on most tasks

RRF_score(doc) = Σ_methods 1 / (k + rank_method(doc))
k = 60 (constant, reduces the influence of top 1)

Example:
         BM25_rank Dense_rank RRF_score
chunk_A: 1 3 1/61 + 1/63 = 0.032
chunk_B: 5 1 1/65 + 1/61 = 0.032
chunk_C: 2 2 1/62 + 1/62 = 0.032
chunk_D: 100 50 1/160 + 1/110 = 0.015

chunk_C (good at both) → up!
chunk_D (bad in both) -> down

In Qdrant: built-in hybrid search with RRF
In pgvector: manually via UNION + RRF calculation

Stage 1: fast retrieval (bi-encoder + ANN)
  K = 100 candidates in ~5ms
  ↑ approximately, but quickly

Stage 2: reranking (cross-encoder)
  for each candidate_chunk:
    score = cross_encoder(query, chunk)
    # query and chunk together via BERT → more precisely
  top_k_reranked = sorted by score[:k]
  ↑ slow (100 × forward pass), but accurate

Popular rerankers:
cohere rerank-3 ← API
bge-reranker-v2-m3 ← self-hosted
ms-marco-MiniLM-L-6 ← fast, easy

NDCG@10 gain: +5–15% over bi-encoder

┌─ RAW DOCUMENTS ──────────────────────────────────┐
│ PDF, DOCX, HTML, Markdown, JSON, DB records │
└──────────────────────┬── ─────────────────────────┘
                       ↓
┌─ PARSE & CLEAN ──────────────────────────────────┐
│ extract text: pypdf, unstructured, markitdown │
│ clean: remove headers/footers, normalize spaces │
└──────────────────────┬── ─────────────────────────┘
                       ↓
┌─ CHUNK ───────────────────── ─────────────────────┐
│ split: 512 tokens, overlap 64 │
│ enrich: prepend section title + doc title │
└──────────────────────┬── ─────────────────────────┘
                       ↓
┌─ EMBED ───────────────────── ─────────────────────┐
│ embed_model.encode(batch_of_chunks) │
│ → vectors ∈ ℝ^768, shape [N × 768] │
└──────────────────────┬── ─────────────────────────┘
                       ↓
┌─ STORE ───────────────────── ─────────────────────┐
│ qdrant.upsert(vectors, payloads) │
│ HNSW index is built automatically │
└───────────────────────── ─────────────────────────┘

QUERY: "What is WB's commission for selling jeans in the clothing category?"

───────────────────────────────── ─────────────────────────────────
①QUERY ENCODING
───────────────────────────────── ─────────────────────────────────
  q_emb = embed_model.encode("query: What is the WB commission...")
  q_emb ∈ ℝ^768 ← [0.31, -0.07, 0.54, 0.12, …]

───────────────────────────────── ─────────────────────────────────
② HYBRID SEARCH (parallel)
───────────────────────────────── ─────────────────────────────────
  Dense: HNSW.search(q_emb, top_k=20) → 20 candidates
    q_emb ↔ each chunk_emb is compared via cosine
    chunk_42 "WB tariffs: clothes 12%..." → cos: 0.891 ← 1st place
    chunk_17 "Commission for accessories..." → cos: 0.821 ← 2nd

  BM25: inverted_index.search("commission WB clothes", top_k=20)
    term match: "commission" df=120, IDF=2.9
    term match: "WB" df=890, IDF=1.1
    chunk_42 → BM25: 8.4 ← 1st place

  RRF: merge, chunk_42 top in both → score 0.033 → top 1 final

───────────────────────────────── ─────────────────────────────────
③ METADATA FILTER (if set)
───────────────────────────────── ─────────────────────────────────
  WHERE category = "pricing" AND language = "ru"
  → out of 20 candidates, 8 remain

───────────────────────────────── ─────────────────────────────────
④ RERANKING (cross-encoder)
───────────────────────────────── ─────────────────────────────────
  reranker.predict(query, chunk_42) → 0.94 ← winner
  reranker.predict(query, chunk_17) → 0.71
  reranker.predict(query, chunk_8) → 0.32
  top_3 for LLM: [chunk_42, chunk_17]

───────────────────────────────── ─────────────────────────────────
⑤CONTEXT ASSEMBLY
───────────────────────────────── ─────────────────────────────────
  context = f"""
  [Source: WB Tariffs - Section 4.1 Clothing]
  Wildberries commission for clothes is 12–17%
  depending on the category. Jeans are...
  """

───────────────────────────────── ─────────────────────────────────
⑥LLM GENERATION
───────────────────────────────── ─────────────────────────────────
  system: "Only respond based on the context provided"
  user: query + context
  → "WB commission for jeans (category "Clothing") is 15%..."