{
  "type": "new-runtime-knowledge-guide",
  "version": 1,
  "generated_at": "2026-07-23",
  "volume": 18,
  "slug": "rag-runtime-reference",
  "title": "RAG Runtime: From Cosine Similarity to the Final Answer",
  "description": "Not “what is RAG” in general terms, but what actually happens at the code and object level: what data structures are created during indexing, what exactly the vector store returns, who reacts to `score`, what if/else branches are triggered after retrieval, how rerank, packing, refusal, citations and debugging are done. This volume is about RAG as an executable program.",
  "track": {
    "slug": "retrieval-and-rag",
    "title": "Retrieval & RAG",
    "route": "https://newruntime.com/learn/tracks/retrieval-and-rag/",
    "position": 3
  },
  "counts": {
    "sections": 7,
    "cards": 33
  },
  "routes": {
    "html": "https://newruntime.com/learn/rag-runtime-reference/",
    "json": "https://newruntime.com/learn/rag-runtime-reference.json"
  },
  "sections": [
    {
      "number": "01",
      "title": "RAG As An Executable Program",
      "intro": "",
      "cards": [
        {
          "number": "01",
          "title": "RAG is not “one model”, but a chain of services and branches",
          "tag": "Runtime",
          "description": "In production RAG there is almost never a magic `rag(query)`. Usually there is an API handler, chunk store, embedder, vector DB, sometimes BM25, sometimes reranker, prompt builder, LLM client and post-processing. After each step, the code decides whether to move on, do a fallback, refuse a response, or ask for more data.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "ROUGH DIAGRAM RAG RUNTIME\n\nHTTP request\n  ↓\nAPI handler\n  ↓\nquery preprocessing\n  ↓\nembed(query)\n  ↓\nvector search / hybrid search\n  ↓\nbranching on scores\n  ↓\nrerank/dedup/expand\n  ↓\ncontext packing\n  ↓\nLLM prompt\n  ↓\nanswers + citations + logs\n\nKey point: each rectangle is ordinary code, not “model magic”"
            }
          ]
        },
        {
          "number": "02",
          "title": "Two different pipelines: indexing-time and query-time",
          "tag": "Runtime",
          "description": "A common thinking mistake: mixing indexing and query processing. Indexing-time makes documents suitable for retrieval. Query-time uses an already built index and makes decisions on a specific user question. These are different budgets, different bugs and different logs.",
          "blocks": [
            {
              "kind": "table",
              "headers": [
                "Phase",
                "What it does",
                "Main artifacts"
              ],
              "rows": [
                [
                  "Indexing-time",
                  "chunk, embed, store",
                  "chunk records, vectors, metadata, ANN index"
                ],
                [
                  "Query-time",
                  "search, filter, rank, generate",
                  "search hits, rerank scores, packed context, answer"
                ]
              ]
            }
          ]
        },
        {
          "number": "03",
          "title": "What objects actually live in code?",
          "tag": "Runtime",
          "description": "To understand RAG, it's useful to think not about “the documents found something”, but about “what specific objects are running between functions.” Typically these are chunk record, query embedding, search hit, rerank hit, context packed entry and final answer object.",
          "blocks": [
            {
              "kind": "code",
              "label": "typical structures",
              "language": "text",
              "value": "ChunkRecord:\n  chunk_id\n  doc_id\n  text\n  embedding\n  metadata\n\nSearchHit:\n  chunk_id\n  score\n  payload\n\nRerankHit:\n  chunk_id\n  retrieval_score\n  rerank_score\n\nContextItem:\n  chunk_id\n  text\n  token_count\n  citation_label"
            }
          ]
        },
        {
          "number": "04",
          "title": "Who is “triggered” after retrieval",
          "tag": "Runtime",
          "description": "The base usually does not “react” to `cosine similarity`. Your orchestrator code in the application reacts. Vector DB simply returns a list of hits. Then the Python/TypeScript/Go code decides: whether to filter by threshold, whether to rerank, whether to expand to parent-doc, whether to refuse or call LLM.",
          "blocks": [
            {
              "kind": "code",
              "label": "important idea",
              "language": "python",
              "value": "vector_db.search(...)\n  → returns hits[]\n\nnot vector DB does:\n  if top_score < threshold: ...\n  if use_reranker: ...\n  if not enough_context: ...\n\nthis is what the application layer/retrieval service does"
            }
          ]
        }
      ]
    },
    {
      "number": "02",
      "title": "Indexing-Time: How Documents Become an Index",
      "intro": "",
      "cards": [
        {
          "number": "05",
          "title": "Loader -> parser -> chunker: first real code route",
          "tag": "Indexing",
          "description": "Indexing usually begins not with embedding, but with document cleaning. Text is extracted from PDF/HTML/Markdown, cut by structure or by tokens, sometimes adding headings and a path to a section. The output is a list of chunk objects, not one big document string.",
          "blocks": [
            {
              "kind": "code",
              "label": "simplified indexing loop",
              "language": "python",
              "value": "raw_docs = loader.load()\n\nfor doc in raw_docs:\n  parsed = parser.parse(doc)\n  chunks = chunker.split(parsed)\n  for chunk in chunks:\n    enqueue_for_embedding(chunk)"
            }
          ]
        },
        {
          "number": "06",
          "title": "Chunk record stores more than just text",
          "tag": "Indexing",
          "description": "A good RAG will almost never store just `text + vector` in its index. Usually there are `doc_id`, `section_title`, `source_url`, `page`, `created_at`, `access_scope`, `token_count`, offsets and the document version nearby. Then it is these fields that decide how filter, dedup, citations and ACL will work.",
          "blocks": [
            {
              "kind": "code",
              "label": "example payload",
              "language": "text",
              "value": "payload = {\n  \"doc_id\": \"handbook-42\",\n  \"chunk_id\": \"handbook-42::17\",\n  \"text\": \"...\",\n  \"section\": \"Vacation Policy\",\n  \"page\": 8,\n  \"token_count\": 173,\n  \"source_url\": \"...\",\n  \"tenant_id\": \"acme\",\n  \"updated_at\": \"2026-03-07\"\n}"
            }
          ]
        },
        {
          "number": "07",
          "title": "Embedding stage: text turns into a dense array of numbers",
          "tag": "Indexing",
          "description": "At the embedding stage, the code calls the embedding model in batches and receives a vector for each chunk. Often, vectors are additionally normalized, because for text, cosine similarity after L2 normalization is convenient to interpret and quickly calculate as a dot product.",
          "blocks": [
            {
              "kind": "code",
              "label": "typical batch pipeline",
              "language": "python",
              "value": "batch_texts = [chunk.text for chunk in chunks]\nvectors = embedder.embed_documents(batch_texts)\n\nfor chunk, vec in zip(chunks, vectors):\n  vec = l2_normalize(vec)\n  store.append({\n    \"chunk_id\": chunk.id,\n    \"embedding\": vec,\n    \"payload\": chunk.payload,\n  })"
            },
            {
              "kind": "list",
              "label": "what is important to log",
              "items": [
                "embedding_model",
                "embedding_dim",
                "doc_version",
                "you cannot silently mix different embedding models in the same index"
              ]
            }
          ]
        },
        {
          "number": "08",
          "title": "Upsert in vector store: what is actually saved",
          "tag": "Infra",
          "description": "Upsert typically stores three things: ID, vector, and payload. This could be pgvector, Qdrant, Weaviate, Pinecone, FAISS + a separate KV table. An important point: similarity is not calculated in advance. The index contains only vectors and search service structures.",
          "blocks": [
            {
              "kind": "code",
              "label": "meaning of upsert",
              "language": "text",
              "value": "upsert(\n  id=\"handbook-42::17\",\n  vector=[0.018, -0.331, ...],\n  payload={...}\n)\n\nAt this moment there is no query and no cosine score\nThere is only a willingness to quickly compare the future query vector with stored vectors"
            }
          ]
        },
        {
          "number": "09",
          "title": "ANN index stores shortcuts for fast top-k",
          "tag": "Infra",
          "description": "If the collection is large, a complete search of all vectors is too expensive. Therefore, vector DB builds an ANN structure like HNSW or IVF. It doesn’t “know the meaning”, but simply helps you quickly navigate to similar vectors without viewing all the elements.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "WHAT APPEARS AFTER INDEX BUILD\n\nbefore build:\n  raw vectors + payload\n\nafter build:\n  raw vectors + payload\n  + graph / inverted lists / neighbor links\n\nat query-time:\n  search traverses index\n  → gets candidate ids\n  → calculates distance/similarity for candidates\n  → returns top-k"
            }
          ]
        }
      ]
    },
    {
      "number": "03",
      "title": "Retrieval-Time: From Question to Search Hits",
      "intro": "",
      "cards": [
        {
          "number": "10",
          "title": "Query preprocessing: rewrite and normalize are optional, but code-based",
          "tag": "Retrieval",
          "description": "Before embedding, queries are sometimes rewritten: abbreviations are expanded, noise is reduced, entity names are normalized, and the user’s working context is added. This is not a “product-level idea”, but a specific function before `embed_query(query)`.",
          "blocks": [
            {
              "kind": "code",
              "label": "typical pre-retrieval hook",
              "language": "python",
              "value": "query = request.user_query\n\nif use_rewrite:\n  query = query_rewriter(query, chat_history)\n\nfilters = build_metadata_filters(request)\ntenant  = request.tenant_id"
            }
          ]
        },
        {
          "number": "11",
          "title": "Embed query: we get one query vector",
          "tag": "Retrieval",
          "description": "A request usually produces one dense vector. Sometimes there are several of them: multi-query retrieval, HyDE, query decomposition. But the basic path is simple: the question string turns into `q_emb`, which then goes into vector search.",
          "blocks": [
            {
              "kind": "code",
              "label": "the most common code",
              "language": "text",
              "value": "q_emb = embedder.embed_query(query)\nq_emb = l2_normalize(q_emb)\n\nhits = vector_store.search(\n  vector=q_emb,\n  limit=20,\n  filters=filters\n)"
            }
          ]
        },
        {
          "number": "12",
          "title": "What vector search actually does and what it returns",
          "tag": "Retrieval",
          "description": "Search does not return the “correct answer”. It returns a list of closest candidates. This is usually an array of objects with `id`, `score`, `payload`. In FAISS these can be separate `distances` and `indices`; in pgvector it is SQL rows; in Qdrant these are points with `score` and `payload`.",
          "blocks": [
            {
              "kind": "code",
              "label": "three typical results",
              "language": "sql",
              "value": "FAISS:\nD, I = index.search(q_emb, k)\n# D = distances/scores, I = vector ids\n\npgvector:\nSELECT chunk_id, text, 1 - (embedding <=> :q) AS cosine_sim\nFROM chunks\nORDER BY embedding <=> :q\nLIMIT 20;\n\nQdrant-style:\n[\n  {id: \"42\", score: 0.812, payload: {...}},\n  {id: \"17\", score: 0.791, payload: {...}}\n]"
            }
          ]
        },
        {
          "number": "13",
          "title": "Cosine similarity score is not a probability score or a truth score",
          "tag": "Retrieval",
          "description": "The returned `0.82` does not mean “82% that this chunk is correct”. This is simply a measure of proximity in embedding space for this model and this collection. Different embedding models give different score ranges, so the thresholds are calibrated empirically based on your task.",
          "blocks": [
            {
              "kind": "list",
              "label": "what is important to remember",
              "items": [
                "0.82 in one model is not equal to 0.82 in another",
                "score doesn't say whether this is enough for an answer",
                "threshold is selected on the eval set"
              ]
            }
          ]
        },
        {
          "number": "14",
          "title": "The most common question is: “cosine similarity has returned - and what next?”",
          "tag": "Retrieval",
          "description": "Then the orchestrator receives a list of hits and begins the usual branching logic. It does not have to immediately call LLM. First it can filter, deduplicate, expand to parent-doc, run a reranker, or even return `no answer` if the evidence is too weak.",
          "blocks": [
            {
              "kind": "code",
              "label": "minimal post-search if/else",
              "language": "python",
              "value": "hits = vector_store.search(..., limit=20)\n\nif not hits:\n  return no_context_response()\n\nhits = [h for h in hits if h.score >= RETRIEVAL_MIN_SCORE]\n\nif not hits:\n  return low_confidence_response()\n\nif use_reranker:\n  hits = rerank(query, hits)\n\ncontext = pack_context(hits, token_budget=3000)\nreturn answer_with_llm(query, context)"
            }
          ]
        }
      ]
    },
    {
      "number": "04",
      "title": "After Search: Ranking, Branching, Packing",
      "intro": "",
      "cards": [
        {
          "number": "15",
          "title": "Thresholding: first runtime gate",
          "tag": "Ranking",
          "description": "The first thing that is often done after retrieval is to cut off frankly weak candidates. Threshold can be by top-1, by each hit, or by the difference between the first and second. This is simple logic, but it is what often decides whether a system will hallucinate or honestly refuse to respond.",
          "blocks": [
            {
              "kind": "code",
              "label": "multiple threshold strategies",
              "language": "python",
              "value": "top1 gate:\n  if hits[0].score < 0.72: refuse\n\nper-hit gate:\n  keep only hits with score >= 0.68\n\nmargin gate:\n  if hits[0].score - hits[1].score < 0.02:\n    route_to_reranker()"
            }
          ]
        },
        {
          "number": "16",
          "title": "Dedup and grouping: top 20 hits are not equal to top 20 pieces of context",
          "tag": "Ranking",
          "description": "Search often returns several adjacent or almost identical chunks from the same document. If you insert them as is, LLM will receive noise and waste tokens. Therefore, the code often merges hits by `doc_id`, throws out near-duplicates and leaves only the best representatives or merges contiguous chunks.",
          "blocks": [
            {
              "kind": "code",
              "label": "typical post-search cleanup",
              "language": "python",
              "value": "hits = dedup_by_chunk_id(hits)\nhits = dedup_by_similarity(hits, cutoff=0.97)\nhits = group_by_doc(hits)\n\nif expand_neighbors:\n  hits = add_adjacent_chunks(hits, window=1)"
            }
          ]
        },
        {
          "number": "17",
          "title": "Hybrid fusion: dense score may not be the only one",
          "tag": "Ranking",
          "description": "In many RAG systems, dense retrieval is only one channel. BM25, keyword search or metadata filters can run in parallel. Then the code does the fusion: for example, through RRF or simply through heuristic merge. This is important because exact IDs, error codes and rare names are sometimes missed by dense retrieval.",
          "blocks": [
            {
              "kind": "code",
              "label": "simplified merge",
              "language": "text",
              "value": "dense_hits = vector_store.search(...)\nsparse_hits = bm25.search(...)\n\nmerged = reciprocal_rank_fusion(dense_hits, sparse_hits)\nhits = merged[:20]"
            }
          ]
        },
        {
          "number": "18",
          "title": "Reranker: who exactly rewrites the order of candidates",
          "tag": "Ranking",
          "description": "Dense search is usually good at recall, but not great at final ranking. Therefore, the next runtime-step is often a reranker. It receives `query + candidate text` and returns a new score for each candidate. After this, the application sorts by `rerank_score`, and not by the original `cosine`.",
          "blocks": [
            {
              "kind": "code",
              "label": "what changes after rerank",
              "language": "text",
              "value": "# retrieval output\n[\n  {chunk_id: 42, score: 0.81},\n  {chunk_id: 17, score: 0.80},\n  {chunk_id: 91, score: 0.79}\n]\n\n# reranker output\n[\n  {chunk_id: 17, retrieval_score: 0.80, rerank_score: 0.94},\n  {chunk_id: 42, retrieval_score: 0.81, rerank_score: 0.71},\n  {chunk_id: 91, retrieval_score: 0.79, rerank_score: 0.19}\n]\n\nfinal order = sort by rerank_score desc"
            },
            {
              "kind": "list",
              "label": "runtime trade-off",
              "items": [
                "better precision",
                "more expensive latency",
                "usually run on top-20 / top-50, not on the entire base"
              ]
            }
          ]
        },
        {
          "number": "19",
          "title": "Token budget: even the best rank is not equal to a ready-made prompt",
          "tag": "Generation",
          "description": "After reranking, you don’t need to “transfer everything to LLM,” but stay within the token budget. Therefore, the code counts tokens, trims long fragments, sometimes compresses them, or takes only top-3/top-5. This step is called context packing and is critical to the quality of the response.",
          "blocks": [
            {
              "kind": "code",
              "label": "packing loop",
              "language": "python",
              "value": "budget = 3000\npacked = []\n\nfor hit in hits:\n  n = count_tokens(hit.text)\n  if n > budget:\n    continue\n  packed.append(hit)\n  budget -= n\n\nif not packed:\n  return refusal(\"No supporting context\")"
            }
          ]
        },
        {
          "number": "20",
          "title": "The order of chunks is also a coding decision",
          "tag": "Generation",
          "description": "Chunks can be placed in the prompt in different ways: strictly according to rerank score, grouping by documents, placing the best one at the end, inserting headers and source labels. This is not a layout detail. This function determines which pieces will be closer to the last prompt tokens and will have a stronger influence on the model.",
          "blocks": [
            {
              "kind": "code",
              "label": "render example",
              "language": "python",
              "value": "for i, hit in enumerate(packed, start=1):\n  prompt_context += f\"[SOURCE {i}]\\\\n\"\n  prompt_context += f\"title: {hit.payload['section']}\\\\n\"\n  prompt_context += hit.payload[\"text\"] + \"\\\\n\\\\n\""
            }
          ]
        }
      ]
    },
    {
      "number": "05",
      "title": "LLM Stage: Prompt, Answer, Refusal",
      "intro": "",
      "cards": [
        {
          "number": "21",
          "title": "Prompt builder: the most important glue code between retrieval and generation",
          "tag": "Generation",
          "description": "Prompt builder doesn't just “insert chunks of text.” It defines the response format, rejection rules, quoting method, the distinction between instructions and retrieved context, and sometimes also the schema for structured output. Very often the quality of RAG breaks down here, although the retrieval was good.",
          "blocks": [
            {
              "kind": "code",
              "label": "typical prompt template",
              "language": "sql",
              "value": "SYSTEM:\n  Answer only from the provided sources.\n  If evidence is insufficient, say you do not know.\n  Cite source ids in square brackets.\n\nUSER QUESTION:\n  {query}\n\nCONTEXT:\n  {packed_sources}"
            }
          ]
        },
        {
          "number": "22",
          "title": "LLM call - a regular API call with a long prefill",
          "tag": "Generation",
          "description": "After packing and prompt build, everything usually comes down to the usual chat completion call. But this is where retrieval gets expensive in terms of tokens: the retrieved context often makes up the bulk of the input prompt. Therefore, the RAG system pays not only for search latency, but also for the long prefill in LLM.",
          "blocks": [
            {
              "kind": "code",
              "label": "call example",
              "language": "text",
              "value": "response = llm.chat(\n  model=\"...\",\n  temperature=0,\n  messages=[\n    {\"role\": \"system\", \"content\": system_prompt},\n    {\"role\": \"user\", \"content\": user_block_with_context}\n  ]\n)"
            }
          ]
        },
        {
          "number": "23",
          "title": "Refusal branch: a good RAG knows how not to respond",
          "tag": "Generation",
          "description": "If the retrieval is weak, it is better to refuse or ask for clarification than to force the LLM to “think it through.” Therefore, there is almost always a separate branch: `no hits`, `low confidence`, `context not enough`, `conflicting evidence`. This is not a UX detail, but a part of core logic.",
          "blocks": [
            {
              "kind": "code",
              "label": "typical refusal gate",
              "language": "python",
              "value": "if not packed:\n  return {\n    \"answer\": \"I don't see enough data in the sources.\",\n    \"citations\": [],\n    \"reason\": \"no_context\"\n  }\n\nif top_rerank_score < 0.35:\n  return clarification_request()"
            }
          ]
        },
        {
          "number": "24",
          "title": "Citations do not appear on their own - they need to be pulled through the entire pipeline",
          "tag": "Generation",
          "description": "In order for answer to have links to sources, chunk ids and source labels must be saved before the LLM call. Then either the model itself writes `[SOURCE 2]`, or the post-processor maps the claims back to the context. If the payload does not contain stable source labels, normal grounding will not work.",
          "blocks": [
            {
              "kind": "code",
              "label": "minimal citation path",
              "language": "text",
              "value": "packed = [\n  {label: \"SOURCE 1\", chunk_id: \"42\", text: \"...\"},\n  {label: \"SOURCE 2\", chunk_id: \"17\", text: \"...\"}\n]\n\nprompt tells model:\n  cite labels like [SOURCE 1]\n\npostprocess:\n  map labels → real URLs/pages"
            }
          ]
        },
        {
          "number": "25",
          "title": "Full query-time pseudocode: from request to answer",
          "tag": "Generation",
          "description": "Here is the most useful mental model for a developer: RAG is a regular function with clear inputs, intermediate objects and branches. Below is not a concept, but an almost skeleton production handler.",
          "blocks": [
            {
              "kind": "code",
              "label": "end-to-end skeleton",
              "language": "python",
              "value": "def answer_question(request):\n    query = preprocess_query(request.query, request.history)\n    filters = build_filters(request)\n\n    q_emb = embedder.embed_query(query)\n    q_emb = l2_normalize(q_emb)\n\n    hits = vector_store.search(q_emb, limit=20, filters=filters)\n    if not hits:\n        return no_context_response()\n\n    hits = [h for h in hits if h.score >= RETRIEVAL_MIN_SCORE]\n    if not hits:\n        return low_confidence_response()\n\n    hits = dedup_hits(hits)\n    hits = maybe_expand_neighbors(hits)\n    hits = maybe_merge_sparse(query, hits)\n\n    if use_reranker:\n        hits = rerank(query, hits)\n\n    packed = pack_context(hits, token_budget=MAX_CONTEXT_TOKENS)\n    if not packed:\n        return refusal_response()\n\n    prompt = render_prompt(query, packed)\n    raw_answer = llm.generate(prompt)\n    answer = postprocess(raw_answer, packed)\n\n    log_trace(query, hits, packed, answer)\n    return answer"
            }
          ]
        }
      ]
    },
    {
      "number": "06",
      "title": "Advanced Branches: More than One Happy Path",
      "intro": "",
      "cards": [
        {
          "number": "26",
          "title": "Parent-document retrieval: retrieval in small pieces, answer in large pieces",
          "tag": "Bridge",
          "description": "Common production logic: index small chunks for the sake of recall, but after a hit, expand them to the parent block or neighboring chunks. This is a separate runtime branch after retrieval, and not “a kind of embedding”.",
          "blocks": [
            {
              "kind": "code",
              "label": "example expand step",
              "language": "text",
              "value": "hits = vector_store.search(...)\nparent_ids = unique(h.payload[\"parent_id\"] for h in hits[:5])\nexpanded = doc_store.fetch_parent_blocks(parent_ids)\n\npacked = rerank_or_pack(expanded)"
            }
          ]
        },
        {
          "number": "27",
          "title": "Iterative retrieval: first pass didn't find enough",
          "tag": "Bridge",
          "description": "Some RAG systems don't stop at just search. If the context is weak, LLM or deterministic code builds an additional query, extends filters, includes another index, or makes a web/tool ​​branch. This is another regular control-flow branch.",
          "blocks": [
            {
              "kind": "code",
              "label": "simplified branch",
              "language": "python",
              "value": "hits = search(query)\n\nif evidence_is_weak(hits):\n  follow_up_query = reformulate(query)\n  more_hits = search(follow_up_query)\n  hits = merge(hits, more_hits)"
            }
          ]
        },
        {
          "number": "28",
          "title": "Cheap-first routing: retrieval sometimes completes the task without LLM",
          "tag": "Bridge",
          "description": "If the top-1 hit is very strong and the payload contains a ready-made FAQ-answer or structured field, the application may not call LLM at all. This is especially useful for FAQ, knowledge lookup and semantic cache. That is, after cosine similarity, not only the “in prompt” branch is possible, but also the “return a ready-made answer” branch.",
          "blocks": [
            {
              "kind": "code",
              "label": "simple early-exit branch",
              "language": "python",
              "value": "hits = search(query)\n\nif hits[0].score > FAQ_DIRECT_THRESHOLD and hits[0].payload[\"answer\"]:\n  return {\n    \"answer\": hits[0].payload[\"answer\"],\n    \"source\": hits[0].payload[\"source_url\"],\n    \"mode\": \"retrieval_only\"\n  }"
            }
          ]
        },
        {
          "number": "29",
          "title": "Streaming and post-processing occur after the LLM, but are also important",
          "tag": "Bridge",
          "description": "When LLM starts streaming tokens, the retrieval has already ended, but the pipeline is not yet there. Next can be markdown cleanup, schema validation, extraction of cited source labels, moderation, analytics events and fallback if the model output does not pass the test.",
          "blocks": [
            {
              "kind": "code",
              "label": "last layer",
              "language": "text",
              "value": "raw_answer = llm.stream(...)\nanswer = collect_stream(raw_answer)\nanswer = validate_schema(answer)\nanswer = attach_citation_urls(answer, packed)\nanswer = moderate(answer)\nreturn answer"
            }
          ]
        }
      ]
    },
    {
      "number": "07",
      "title": "Debugging, Metrics, Failure Tree",
      "intro": "",
      "cards": [
        {
          "number": "30",
          "title": "What to log at each step of RAG runtime",
          "tag": "Ops",
          "description": "If you don't log intermediate artifacts, RAG becomes a black box. To debug, you need to see separately: input query, rewritten query, filters, top-k retrieval hits, scores, rerank scores, packed chunk ids, final prompt size and answer. Without this, it is impossible to understand whether retrieval failed or generation.",
          "blocks": [
            {
              "kind": "code",
              "label": "minimal trace schema",
              "language": "text",
              "value": "trace = {\n  \"query\": \"...\",\n  \"rewritten_query\": \"...\",\n  \"filters\": {...},\n  \"retrieval_hits\": [...],\n  \"rerank_hits\": [...],\n  \"packed_chunk_ids\": [...],\n  \"packed_token_count\": 2714,\n  \"llm_model\": \"...\",\n  \"answer\": \"...\",\n  \"latency_ms\": {...}\n}"
            }
          ]
        },
        {
          "number": "31",
          "title": "Failure tree: error in retrieval, ranking or generation?",
          "tag": "Ops",
          "description": "When the response is bad, the stage needs to be localized. If the required chunk is not in top-k, the problem is in indexing or retrieval. If there is a chunk, but it has sunk below, the problem is rerank or packing. If the chunk is in prompt, but the answer still does not follow the source, the problem is already in prompt or generation.",
          "blocks": [
            {
              "kind": "code",
              "label": "Diagram",
              "language": "text",
              "value": "QUICK DEBUG\n\n1. Was the required fact even in the database?\n   no → indexing bug\n\n2. Did he get into the top-k retrieval?\n   no → retrieval / embedding / chunking bug\n\n3. Did he get into a packed context?\n   no → rerank / token budget / ordering bug\n\n4. Was it in the prompt, but LLM ignored it?\n   yes → prompt / refusal / generation bug"
            }
          ]
        },
        {
          "number": "32",
          "title": "What metrics monitor RAG as a system?",
          "tag": "Ops",
          "description": "RAG needs to be measured layer by layer. Retrieval has its own recall/precision metrics. Ranking has its own hit@k and rerank gain. Generation - faithfulness, answer relevance, refusal accuracy. Plus infrastructure metrics: latency, cost, cache hit rate, index freshness.",
          "blocks": [
            {
              "kind": "list",
              "label": "minimum set",
              "items": [
                "retrieval recall@k",
                "MRR / NDCG / hit@k",
                "faithfulness / citation coverage",
                "refusal precision",
                "end-to-end latency"
              ]
            }
          ]
        },
        {
          "number": "33",
          "title": "The main engineering thought of the volume",
          "tag": "Ops",
          "description": "When you see `cosine similarity = 0.81`, this is not an answer or a solution. It's just one of the inputs to the next layer of code. Everything important in production RAG happens next: thresholds, rerank, dedup, packing, prompting, refusal, citations and observability. Therefore, RAG should be designed not as a “vector database connected”, but as a full-fledged runtime pipeline.",
          "blocks": [
            {
              "kind": "list",
              "label": "short reminder",
              "items": [
                "score → branch",
                "hits → objects, not an answer",
                "LLM is not always called",
                "the main bugs live between stages"
              ]
            }
          ]
        }
      ]
    }
  ]
}