{ "type": "new-runtime-knowledge-guide", "version": 1, "generated_at": "2026-07-23", "volume": 9, "slug": "rag-vector-search-reference", "title": "RAG & Vector Search", "description": "Full route: why words cannot be compared directly → Bag of Words → TF-IDF → Word2Vec → contextual embeddings → cosine distance → how exactly the chunk “matches” with the query → HNSW → hybrid search → the full RAG pipeline parsed down to the byte.", "track": { "slug": "retrieval-and-rag", "title": "Retrieval & RAG", "route": "https://newruntime.com/learn/tracks/retrieval-and-rag/", "position": 1 }, "counts": { "sections": 7, "cards": 24 }, "routes": { "html": "https://newruntime.com/learn/rag-vector-search-reference/", "json": "https://newruntime.com/learn/rag-vector-search-reference.json" }, "sections": [ { "number": "01", "title": "Sparse methods: from words to numbers", "intro": "", "cards": [ { "number": "01", "title": "Fundamental problem: text cannot be directly compared", "tag": "Sparse", "description": "The computer works with numbers. The strings “cat” and “cat” are different byte sequences with no default proximity between them. The entire history of information retrieval is a history of attempts to turn the meaning of text into numbers that can be compared mathematically.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Naive string comparison:\n\n query: \"cat\"\n doc_1: \"cat\" → string match: 0% (different strings)\n doc_2: \"kitten\" → string match: 0% (different strings)\n doc_3: \"cat\" → string match: 100% (match)\n doc_4: \"pet\" → string match: 0% (although the meaning is close!)\n\nProblem: semantic proximity ≠ lexical match\n\nEvolution of solutions:\n 1960s Boolean search (AND/OR/NOT keywords)\n 1970s Bag of Words → word frequency vector\n 1980s TF-IDF → importance weighting\n 2000s LSA/LSI → hidden semantics via SVD\n 2013 Word2Vec → neural word embeddings\n 2018 BERT → contextual offer embeddings\n 2020+ Bi-Encoder models → embeddings for search (E5, BGE, nomic)" } ] }, { "number": "02", "title": "Bag of Words: frequency vector", "tag": "Sparse", "description": "Bag of Words is the first way to turn text into vector. We build a dictionary of all words in the corpus, representing each document as a vector: how many times each word appears. Word order is ignored - hence the name \"bag\".", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Body of 2 documents:\n d1: \"the cat is sitting at home\"\n d2: \"the dog is running home\"\n\nVocabulary (vocab):\n {cat:0, sitting:1, at home:2, dog:3, running:4, home:5}\n\nVectors (BoW):\n cat sitting at home dog running home\n d1: [ 2, 1, 1, 0, 0, 0 ]\n d2: [ 0, 0, 0, 1, 1, 1 ]\n\nquery: \"the cat is running\"\n q: [ 1, 0, 0, 0, 1, 0 ]\n\nMatches with d1: \"cat\" → 1\nMatches with d2: \"runs\" → 1\n→ draw. But \"the cat is running\" is semantically closer to d1?" }, { "kind": "list", "label": "BoW problems", "items": [ "“and”, “in”, “on” - frequent, but useless", "vector size = dictionary size (100k+)", "99% of values = 0 (sparse vector)", "word order lost: “the cat eats the fish” = “the fish eats the cat”" ] } ] }, { "number": "03", "title": "TF-IDF: weighted by rarity", "tag": "Sparse", "description": "TF-IDF solves the problem of stop words: a word is important if it occurs frequently in the document (TF) and rarely in the rest of the corpus (IDF). “Neural network” in a text about ML is more important than “is” - although both can appear 5 times.", "blocks": [ { "kind": "code", "label": "formula and intuition", "language": "tex", "value": "TF(word, document) = count(word in doc) / len(doc)\nIDF(word) = log( N / df(word) )\n N = number of documents in the corpus\n df = in how many documents the word appears\n\nTF-IDF(word, doc) = TF × IDF\n\nExample, corpus = 1000 documents:\n word \"and\": df=1000 → IDF=log(1000/1000)=0\n word \"algorithm\": df=50 → IDF=log(1000/50) =3.0\n word \"catboost\": df=3 → IDF=log(1000/3) =5.8\n\n→ a rare and specific word receives high weight\n→ stop words will automatically be zeroed" }, { "kind": "list", "label": "used in", "items": [ "BM25 - improved TF-IDF (Elasticsearch)", "sparse retrieval in hybrid search", "still doesn't understand synonyms", "“cat” and “cat” are completely different words" ] } ] }, { "number": "04", "title": "BM25: industrial sparse search", "tag": "Sparse", "description": "BM25 (Best Match 25) - improved TF-IDF with normalization by document length. A long document does not benefit simply from size. This is a standard in Elasticsearch, Opensearch and the baseline for RAG retrieval.", "blocks": [ { "kind": "code", "label": "formula BM25", "language": "tex", "value": "BM25(q, d) = Σ_{word∈q} IDF(word) ·\n TF(word,d) · (k₁+1)\n ─────────────────────────────────\n TF(word,d) + k₁·(1-b+b·|d|/avgdl)\n\nk₁ ∈ [1.2, 2.0] – TF saturation\nb ∈ [0, 1] - length normalization (usually 0.75)\n|d| = document length, avgdl = average length\n\nKey properties:\n✓ TF saturation: 100 mentions ≠ 10× more important than 10\n✓ long docs do not automatically dominate\n✓ fast: inverted index → O(|q|) search" }, { "kind": "list", "label": "when BM25 beats vectors", "items": [ "exact names: “WB-article 12345678”", "codes, abbreviations, proper names", "rare specific terms", "synonyms, paraphrases - does not see" ] } ] } ] }, { "number": "02", "title": "Dense Embeddings: meaning in space", "intro": "", "cards": [ { "number": "05", "title": "What is embedding: meaning as a point in space", "tag": "Dense", "description": "Embedding is a vector of several hundred or thousand numbers that encodes the meaning of the text so that semantically similar texts appear close in vector space. This is the key idea: distance in space equals semantic distance in language.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "Imagine a 2D space (actually 768–3072 dimensions):\n\n X-axis → “animal/non-animal”\n Y-axis → domestic/wild\n\n homemade\n ▲\n cat [0.8, 0.9] ●\n cat [0.7, 0.9] ● dog [0.6, 0.8] ●\n │\n ──────────────────── ───────────────────── animal\n │\n lion [0.9, 0.1] ● wolf [0.8, 0.1] ●\n │\n wild\n\n table [-0.9, 0.0] ● ← far from animals\n algorithm [-0.8, 0.0] ●\n\nquery: \"pet\"\n embedding ≈ [0.75, 0.88]\n → nearest points: cat, cat, dog\n → although the word \"pet\" does not appear in these documents!\n → this is the power of semantic search" }, { "kind": "list", "label": "key properties", "items": [ "“cat” and “cat” - side by side in space", "“pet” and “cat” are semantically close", "dense: all 768 values are non-zero", "size: 384–3072 depending on model" ] } ] }, { "number": "06", "title": "Word2Vec: neural word embeddings", "tag": "Dense", "description": "Word2Vec (2013, Google) is the first widely used neural embedding approach. Idea: a word is determined by the company in which it occurs. “Cat” and “cat” appear in similar contexts → their vectors will be similar.", "blocks": [ { "kind": "code", "label": "Skip-gram architecture", "language": "sql", "value": "Task: predict neighbors based on the central word\n\n Window = 2:\n \"the big [cat] is sitting on the sofa\"\n → predict: \"big\", \"sitting\" from the word \"cat\"\n\nNeural network (2 layers):\n input: one-hot(cat) ∈ ℝ^vocab_size\n hidden: W_embed · one-hot → ℝ^300 ← embedding!\n output: softmax → P(each word as a neighbor)\n\nAfter training on a billion tokens:\n king − man + woman ≈ queen\n → the arithmetic of meanings works!\n\nWord2Vec problem:\n \"onion\" (vegetable) = \"onion\" (weapon)\n → one word = one vector, context not taken into account" } ] }, { "number": "07", "title": "Contextual embeddings: BERT and Bi-Encoder", "tag": "Dense", "description": "BERT (2018) solved the problem of polysemy: the embedding of a word depends on the context. The “bow” in “bought a bow for soup” and “took a bow and arrow” receives different vectors. For RAG, Bi-Encoder is used: we encode the request and the document separately, and compare them with cosine.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Word2Vec (static):\n \"bow\" → [0.3, -0.5, 0.8, …] ← always one vector\n\nBERT (contextual):\n \"bought onions for soup\" → [0.7, 0.2, -0.1, …]\n “took a bow and arrows” → [-0.4, 0.8, 0.3, …]\n ← different vectors for one word!\n\nBi-Encoder for search (Sentence-BERT, E5, BGE):\n\n ENCODER ENCODER\n ↑ ↑\n query document\n \"price of a cat\" \"cost of a kitten\"\n\n q_emb = [0.2, 0.8, …] ∈ ℝ^768\n d_emb = [0.3, 0.7, …] ∈ ℝ^768\n\n score = cosine(q_emb, d_emb) = 0.91 ← high!" } ] }, { "number": "08", "title": "Pooling: from tokens to document vector", "tag": "Dense", "description": "The embedding model produces a vector for each token. How to make one document vector from N token vectors? This is called pooling - and the choice of strategy affects the quality of the search.", "blocks": [ { "kind": "code", "label": "pooling strategies", "language": "text", "value": "Entry: \"the cat is sitting at home\"\nTokens: [CLS] cat sitting at home [SEP]\nModel output: vector for each token ℝ^768\n\n[CLS] pooling (BERT-style):\n doc_emb = hidden_state[CLS]\n ↑ a special token is trained to aggregate meaning\n ↑ used in BERT fine-tuned models\n\nMean pooling (Sentence-BERT, E5):\n doc_emb = mean(hidden_states[1:-1])\n ↑ average for all tokens (without CLS and SEP)\n ↑ in practice it is better for long documents\n\nWeighted mean pooling:\n ↑ tokens with attention weights are strengthened\n ↑ used in some modern models\n\nThe result of any pooling:\n doc_emb ∈ ℝ^768 ← one vector per document" } ] } ] }, { "number": "03", "title": "How Embedding Models are Trained for Search", "intro": "", "cards": [ { "number": "09", "title": "Contrastive Learning: learning through attraction and repulsion", "tag": "Training", "description": "The embedding model for search is trained so that relevant pairs (query, document) are close in space, and irrelevant ones are far away. This is called contrastive learning, and it explains why “cat price” and “kitten price” give high similarity.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "python", "value": "Training batch (triplet loss):\n\n anchor (query): \"How to treat a cat for a cold?\"\n positive (relevant): \"Treatment of cats with ARVI includes...\"\n negative (not rel): \"Recipe for borscht with beef\"\n\nLearning Objective:\n cosine(anchor, positive) → maximum high (~1.0)\n cosine(anchor, negative) → lowest possible (~0.0 or negative)\n\nTriple Loss:\n L = max(0, cosine(a,n) − cosine(a,p) + margin)\n ↑ penalty if negative is closer to positive than by margin\n\nMore modern - InfoNCE / NT-Xent (in-batch negatives):\n for each query: all other documents in the batch = negatives\n batch_size=256 → 255 negatives for free\n → modeli: E5, BGE, nomic-embed, text-embedding-3" }, { "kind": "list", "label": "training data", "items": [ "MS-MARCO: 500k query-passage pairs", "NQ, TriviaQA: question and answer from Wikipedia", "synthetic pairs via LLM (for domains)", "hard negatives: similar but irrelevant" ] } ] }, { "number": "10", "title": "Asymmetry between query and document", "tag": "Training", "description": "Query and document are texts of different nature. Query is short, incomplete, contains intent. The document is long, self-contained, and contains facts. Modern embedding models are trained on special prefixes in order to process them differently.", "blocks": [ { "kind": "code", "label": "instruction-prefixes (E5, nomic)", "language": "text", "value": "#E5-instruct:\nquery_emb = encode(\n \"query: \" + \"how to treat a cat for a cold\"\n)\ndoc_emb = encode(\n \"passage: \" + \"For viral infections in cats...\"\n)\n\n# nomic-embed-text:\nquery_emb = encode(\"search_query: \" + text)\ndoc_emb = encode(\"search_document: \" + text)\n\n# Without the correct prefix → quality degradation\n# Read the documentation for each model!\n\nAsymmetric matryoshka (OpenAI text-embedding-3):\n→ you can trim the vector to the desired length\n 3072 → 1536 → 512 → 256 without retraining" } ] }, { "number": "11", "title": "Why embeddings “understand” synonyms", "tag": "Training", "description": "It's not magic, it's statistics. The words \"cat\" and \"cat\" appear in the same contexts in billions of sentences. After contrastive training, the model learned: if the texts are about the same thing, their vectors should be close. Synonymy arises as a side effect of these statistics.", "blocks": [ { "kind": "code", "label": "from statistics to space", "language": "sql", "value": "In a body of 100M sentences:\n\n\"my cat meows\" meets with the same neighbors as\n\"my cat is meowing\"\n\n→ contrastive loss: these two phrases = positive pair\n→ gradient descent moves their vectors closer\n\nLikewise:\n\"cost of goods\" ↔ \"price of goods\" → close\n\"acquire\" ↔ \"buy\" → close\n\"expensive\" ↔ \"cheap\" → FAR\n (antonyms)\n\nBottom line: vector space is\n“map of meanings” learned from language statistics" } ] } ] }, { "number": "04", "title": "Vector Search: how to find the nearest neighbor", "intro": "", "cards": [ { "number": "12", "title": "Cosine similarity: how exactly vectors are compared", "tag": "Search", "description": "Cosine similarity measures the angle between two vectors—not the distance between points, but the direction. This is important: two vectors can be far apart in absolute units, but pointing in the same direction means talking about the same thing.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "tex", "value": "Formula:\n cosine(A, B) = (A B) / (|A| × |B|)\n = Σᵢ(Aᵢ×Bᵢ) / (√Σᵢ Aᵢ² × √Σᵢ Bᵢ²)\n\nRange: -1 (opposite) to +1 (identical)\nFor normalized vectors: cosine = dot product\n\nExample (simplified, 3 dimensions):\n query: q = [0.6, 0.8, 0.0] |q| = 1.0\n doc_1: d1 = [0.5, 0.7, 0.1] |d1| = 0.87\n doc_2: d2 = [0.1, 0.1, 0.9] |d2| = 0.91\n doc_3: d3 = [0.6, 0.8, 0.0] |d3| = 1.0\n\n cosine(q, d1) = (0.6×0.5 + 0.8×0.7 + 0.0×0.1) / (1.0×0.87)\n = (0.30 + 0.56 + 0.00) / 0.87 = 0.874 ← relevant\n\n cosine(q, d2) = (0.6×0.1 + 0.8×0.1 + 0.0×0.9) / (1.0×0.91)\n = (0.06 + 0.08 + 0.00) / 0.91 = 0.154 ← irrelevant\n\n cosine(q, d3) = (0.6×0.6 + 0.8×0.8 + 0.0×0.0) / (1.0×1.0)\n = (0.36 + 0.64 + 0.00) / 1.0 = 1.000 ← identical\n\nResult: we rank the documents: d3 > d1 > d2\nReturn top-k → this is the nearest neighbor search" }, { "kind": "list", "label": "alternative distance metrics", "items": [ "cosine - standard for text", "dot product - faster if vectors are normalized", "L2 (Euclidean) - for images, not text", "for normalized vectors all three are equivalent" ] } ] }, { "number": "13", "title": "Exact KNN vs Approximate Nearest Neighbor", "tag": "Search", "description": "An exact search of all vectors in the database (Exact KNN) guarantees the correct answer, but it is slow for large collections. ANN - approximate search algorithms: sometimes they miss the perfect result, but hundreds of times faster.", "blocks": [ { "kind": "code", "label": "comparison of approaches", "language": "python", "value": "Exact KNN (brute force):\n for doc in all_docs:\n score = cosine(query_emb, doc_emb)\n return top_k(scores)\n\n n=1M docs, d=768 → 768M multiplications per request\n → ~2-5 seconds on CPU. Unacceptable.\n\nANN (HNSW, IVF, ScaNN, FAISS):\n → build the index in advance (~minutes/hours)\n → search: ~1-10ms even for 100M vectors\n → recall@10 = 95-99% (we skip 1-5% of accurate answers)\n\nTrade-off:\n accuracy ↑ ↔ speed ↓\n memory ↑ ↔ accuracy ↑" } ] }, { "number": "14", "title": "HNSW: Hierarchical Nearest Neighbor Graph", "tag": "Search", "description": "HNSW (Hierarchical Navigable Small World) is a standard ANN algorithm in most vector databases (Qdrant, Weaviate, pgvector). Builds a multi-level graph: the top level is large jumps, the bottom is fine tuning.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "HNSW structure (simplified):\n\nLevel 2 (few nodes - “high-speed highway”):\n A ──────────────────── E\n ↑ long jumps, rough navigation\n\nLevel 1 (more nodes):\n A ──── B ──── C ──── D ──── E\n ↑ more detailed search\n\nLevel 0 (all nodes - exact search):\n A─B─C─D─E─F─G─H─I─J─K─L─M\n ↑ all documents, each connected to the k closest ones\n\nSearch Query Q:\n 1. Enter the top level → find the nearest node\n 2. Go down one level → clarify\n 3. At level 0 → bypass nearest neighbors\n 4. Return top-k\n\nParameters:\n M - number of connections per node (16–64)\n ef_construction — accuracy during construction\n ef_search — search accuracy (recall/speed)" } ] }, { "number": "15", "title": "Vector databases: overview", "tag": "Search", "description": "Vector database = HNSW/IVF index + metadata storage + filtering + API. Different databases are optimized for different trade-offs in terms of speed, memory and scaling.", "blocks": [ { "kind": "table", "headers": [ "Base", "Algorithm", "Filtration", "Feature" ], "rows": [ [ "Qdrant", "HNSW", "during search", "Rust, payload filters" ], [ "pgvector", "HNSW / IVFFlat", "SQL WHERE", "in PostgreSQL" ], [ "Weaviate", "HNSW", "GraphQL", "hybrid built" ], [ "Chroma", "HNSW", "post-filter", "lightweight, dev" ], [ "FAISS", "IVF/PQ/HNSW", "no", "library, not database" ] ] } ] } ] }, { "number": "05", "title": "Chunking, Metadata and Matching", "intro": "", "cards": [ { "number": "16", "title": "Chunking Strategies: How to Break Up a Document", "tag": "Chunking", "description": "Chunk is a unit that we encode into a vector and which we return to the LLM context. A chunk that is too small loses context. Too large - it dilutes the signal, embedding becomes “average in a hospital”. The size and chunking strategy is one of the main parameters of RAG quality.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "Strategies:\n\n① Fixed-size (naive):\n chunk_size = 512 tokens, overlap = 64\n → fast, does not take into account structure\n → can cut a sentence in the middle\n\n② Sentence/Paragraph splitting:\n split by \"\\n\\n\" or punkt sentence tokenizer\n → each chunk = semantic unit\n → unequal length (from 20 to 400 tokens)\n\n③ Semantic chunking:\n embed each sentence → find “gaps” by cosine distance\n → chunks = thematically homogeneous pieces\n → expensive (every offer needs to be embedded)\n\n④ Hierarchical (Parent-Child):\n Small chunks for searching (128 tokens) → exact match\n Large chunks for context (512 tokens) → rich response\n → search for small ones, return large ones\n\n⑤ Document structure-aware:\n Markdown h1/h2/h3 → sections as chunks\n PDF → paragraphs from layout\n → best result for structured docs" }, { "kind": "list", "label": "empirical recommendations", "items": [ "Q&A: 256–512 tokens", "technical documents: 512–1024", "overlap: 10–20% of chunk_size", "eval in different sizes - no one size fits all" ] } ] }, { "number": "17", "title": "Chunk metadata: what to store next to the vector", "tag": "Chunking", "description": "Next to each vector, metadata is stored in the database - structured fields for filtering and for adding context to LLM. The right metadata allows you to combine semantic search with precise filters.", "blocks": [ { "kind": "code", "label": "typical payload structure", "language": "json", "value": "{\n # Identification\n \"doc_id\": \"wb-seller-guide-2024\",\n \"chunk_id\": \"wb-seller-guide-2024-ch-047\",\n\n # Document navigation\n \"title\": \"WB Seller's Guide\",\n \"section\": \"3.2 Pricing\",\n \"page\": 47,\n\n # Filtering\n \"category\": \"pricing\",\n \"language\": \"ru\",\n \"date\": \"2024-11-01\",\n\n # Content\n \"text\": \"When calculating the price, take into account...\",\n \"summary\": \"About WB pricing methods\"\n}" }, { "kind": "list", "label": "use in search", "items": [ "filter: category=\"pricing\" AND date>\"2024\"", "title in LLM context: \"from section 3.2\"", "summary for hybrid search" ] } ] }, { "number": "18", "title": "How exactly does the chunk title affect matching?", "tag": "Chunking", "description": "The title of a document or section is a powerful signal. If both the title and the body of the chunk are included in the embedding, then a search for the topic “WB pricing” will find the chunk “3.2 Pricing,” even if the word “pricing” does not appear explicitly in the body of the text.", "blocks": [ { "kind": "code", "label": "header inclusion strategies", "language": "text", "value": "# 1. Prepend title into chunk text before embedding:\ntext_for_embedding = f\"\"\"\nSection: {section_title}\nDocument: {doc_title}\n\n{chunk_text}\"\"\"\n# → title affects embedding through pooling\n\n# 2. Separate “title embedding” + weighted sum:\nemb_title = encode(title)\nemb_body = encode(body)\nemb_final = 0.4 emb_title + 0.6 emb_body\n# → explicit weighting\n\n#3. HyDE (Hypothetical Document Embedding):\n# query → LLM generates a hypothetical answer\n# embed a hypothetical response, not a request\n# → embedding is already in the “document space”" } ] }, { "number": "19", "title": "Atomic parsing: why THIS particular chunk was selected", "tag": "Chunking", "description": "Step-by-step analysis of real matching: the query is received → encoded → compared with each vector in the database → the one closest in cosine wins. It is here that it is clear what is being compared to what.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "User request:\n \"How to calculate the WB commission when selling clothes?\"\n\n① Encode query:\n q_emb = embed_model.encode(\n \"query: How to calculate WB commission when selling clothes?\"\n )\n q_emb ∈ ℝ^768 ← specific vector of 768 numbers\n\n② Chunks are stored in the vector database:\n chunk_42: \"Wildberries commission is 12-25%...\"\n emb_42 = [0.31, -0.12, 0.78, …] (768 numbers)\n cosine(q, emb_42) = 0.891 ← HIGH\n\n chunk_91: \"Returning goods to Wildberries: instructions...\"\n emb_91 = [0.15, 0.44, -0.21, …]\n cosine(q, emb_91) = 0.543 ← average\n\n chunk_3: \"Italian pasta recipes\"\n emb_3 = [-0.67, 0.03, 0.11, …]\n cosine(q, emb_3) = 0.041 ← LOW\n\n③ ANN search → top-3:\n rank 1: chunk_42 → score 0.891 ← WINNER\n rank 2: chunk_91 → score 0.543\n rank 3: chunk_3 → score 0.041\n\nWhy chunk_42 won:\n ✓ contains the words \"commission\", \"Wildberries\" → similar context\n ✓ chunk title: “Tariffs and commissions WB” was in the embedding\n ✓ clothing is mentioned as a category → vector close to query" } ] } ] }, { "number": "06", "title": "Hybrid search: Dense + Sparse", "intro": "", "cards": [ { "number": "20", "title": "Why neither Dense nor Sparse are perfect individually", "tag": "Hybrid", "description": "Dense search is good for semantic proximity, but loses precision terms. BM25 is good for exact matches, but doesn't understand synonyms. Hybrid search combines both signals and consistently outperforms each individual signal.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "Blind spots of each method:\n\nBM25 defeats Dense when:\n query: \"article WB 187234561\"\n → exact number - dense embedding \"averages\" it\n → BM25 finds an exact match using the inverted index\n\n query: \"Government Decree No. 44-FZ\"\n → rare specific terms → BM25 wins\n\nDense defeats BM25 when:\n query: \"how to reduce storage costs\"\n doc: \"methods for reducing warehouse costs\"\n → there is not a single common word!\n → BM25: score = 0. Dense: cosine = 0.82\n\n query: \"pet\"\n doc: \"pet care\"\n → synonymy → only Dense will find\n\nHybrid search:\n score_final = α·BM25_score + (1-α)·dense_score\n → or via RRF (Reciprocal Rank Fusion)\n → consistently better than both separately on most tasks" } ] }, { "number": "21", "title": "RRF: Ranking Fusion", "tag": "Hybrid", "description": "Reciprocal Rank Fusion is a simple and reliable way to combine two lists of results without normalizing the absolute scores. The document that gets to the top of both methods rises to the top.", "blocks": [ { "kind": "code", "label": "RRF algorithm", "language": "tex", "value": "RRF_score(doc) = Σ_methods 1 / (k + rank_method(doc))\nk = 60 (constant, reduces the influence of top 1)\n\nExample:\n BM25_rank Dense_rank RRF_score\nchunk_A: 1 3 1/61 + 1/63 = 0.032\nchunk_B: 5 1 1/65 + 1/61 = 0.032\nchunk_C: 2 2 1/62 + 1/62 = 0.032\nchunk_D: 100 50 1/160 + 1/110 = 0.015\n\nchunk_C (good at both) → up!\nchunk_D (bad in both) -> down\n\nIn Qdrant: built-in hybrid search with RRF\nIn pgvector: manually via UNION + RRF calculation" } ] }, { "number": "22", "title": "Reranking: rechecking top K", "tag": "Hybrid", "description": "Bi-encoder is fast, but crude - it doesn't compare query and document together. Reranker (Cross-Encoder) takes the top K candidates and evaluates each more accurately: it sees query and document simultaneously through full attention.", "blocks": [ { "kind": "code", "label": "two-stage retrieval", "language": "python", "value": "Stage 1: fast retrieval (bi-encoder + ANN)\n K = 100 candidates in ~5ms\n ↑ approximately, but quickly\n\nStage 2: reranking (cross-encoder)\n for each candidate_chunk:\n score = cross_encoder(query, chunk)\n # query and chunk together via BERT → more precisely\n top_k_reranked = sorted by score[:k]\n ↑ slow (100 × forward pass), but accurate\n\nPopular rerankers:\ncohere rerank-3 ← API\nbge-reranker-v2-m3 ← self-hosted\nms-marco-MiniLM-L-6 ← fast, easy\n\nNDCG@10 gain: +5–15% over bi-encoder" } ] } ] }, { "number": "07", "title": "Complete RAG Pipeline", "intro": "", "cards": [ { "number": "23", "title": "Indexing Pipeline: preparing the base", "tag": "Pipeline", "description": "Before searching, you need to prepare the database: upload documents, cut them into chunks, enrich them with metadata, encode them into vectors and write them into a vector database. This is an offline process that runs when documents are updated.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "text", "value": "┌─ RAW DOCUMENTS ──────────────────────────────────┐\n│ PDF, DOCX, HTML, Markdown, JSON, DB records │\n└──────────────────────┬── ─────────────────────────┘\n ↓\n┌─ PARSE & CLEAN ──────────────────────────────────┐\n│ extract text: pypdf, unstructured, markitdown │\n│ clean: remove headers/footers, normalize spaces │\n└──────────────────────┬── ─────────────────────────┘\n ↓\n┌─ CHUNK ───────────────────── ─────────────────────┐\n│ split: 512 tokens, overlap 64 │\n│ enrich: prepend section title + doc title │\n└──────────────────────┬── ─────────────────────────┘\n ↓\n┌─ EMBED ───────────────────── ─────────────────────┐\n│ embed_model.encode(batch_of_chunks) │\n│ → vectors ∈ ℝ^768, shape [N × 768] │\n└──────────────────────┬── ─────────────────────────┘\n ↓\n┌─ STORE ───────────────────── ─────────────────────┐\n│ qdrant.upsert(vectors, payloads) │\n│ HNSW index is built automatically │\n└───────────────────────── ─────────────────────────┘" } ] }, { "number": "24", "title": "Query Pipeline: from question to answer - every step", "tag": "Pipeline", "description": "This is a complete breakdown of what happens when a user asks a question in the RAG system. Each step is specific: what it takes as input, what it returns, what it is compared with. This is where everything the previous cards have said comes together.", "blocks": [ { "kind": "code", "label": "Diagram", "language": "sql", "value": "QUERY: \"What is WB's commission for selling jeans in the clothing category?\"\n\n───────────────────────────────── ─────────────────────────────────\n①QUERY ENCODING\n───────────────────────────────── ─────────────────────────────────\n q_emb = embed_model.encode(\"query: What is the WB commission...\")\n q_emb ∈ ℝ^768 ← [0.31, -0.07, 0.54, 0.12, …]\n\n───────────────────────────────── ─────────────────────────────────\n② HYBRID SEARCH (parallel)\n───────────────────────────────── ─────────────────────────────────\n Dense: HNSW.search(q_emb, top_k=20) → 20 candidates\n q_emb ↔ each chunk_emb is compared via cosine\n chunk_42 \"WB tariffs: clothes 12%...\" → cos: 0.891 ← 1st place\n chunk_17 \"Commission for accessories...\" → cos: 0.821 ← 2nd\n\n BM25: inverted_index.search(\"commission WB clothes\", top_k=20)\n term match: \"commission\" df=120, IDF=2.9\n term match: \"WB\" df=890, IDF=1.1\n chunk_42 → BM25: 8.4 ← 1st place\n\n RRF: merge, chunk_42 top in both → score 0.033 → top 1 final\n\n───────────────────────────────── ─────────────────────────────────\n③ METADATA FILTER (if set)\n───────────────────────────────── ─────────────────────────────────\n WHERE category = \"pricing\" AND language = \"ru\"\n → out of 20 candidates, 8 remain\n\n───────────────────────────────── ─────────────────────────────────\n④ RERANKING (cross-encoder)\n───────────────────────────────── ─────────────────────────────────\n reranker.predict(query, chunk_42) → 0.94 ← winner\n reranker.predict(query, chunk_17) → 0.71\n reranker.predict(query, chunk_8) → 0.32\n top_3 for LLM: [chunk_42, chunk_17]\n\n───────────────────────────────── ─────────────────────────────────\n⑤CONTEXT ASSEMBLY\n───────────────────────────────── ─────────────────────────────────\n context = f\"\"\"\n [Source: WB Tariffs - Section 4.1 Clothing]\n Wildberries commission for clothes is 12–17%\n depending on the category. Jeans are...\n \"\"\"\n\n───────────────────────────────── ─────────────────────────────────\n⑥LLM GENERATION\n───────────────────────────────── ─────────────────────────────────\n system: \"Only respond based on the context provided\"\n user: query + context\n → \"WB commission for jeans (category \"Clothing\") is 15%...\"" }, { "kind": "list", "label": "where quality is lost", "items": [ "chunk is too big → embedding is blurry", "the title was not included in the embedding text", "no metadata → cannot be filtered", "no reranker → bi-encoder misses nuances", "embed_model is not trained on the domain → low recall", "hybrid + rerank → best result" ] } ] } ] } ] }