RAG Runtime: From Cosine Similarity to the Final Answer

In production RAG there is almost never a magic `rag(query)`. Usually there is an API handler, chunk store, embedder, vector DB, sometimes BM25, sometimes reranker, prompt builder, LLM client and post-processing. After each step, the code decides whether to move on, do a fallback, refuse a response, or ask for more data.

ROUGH DIAGRAM RAG RUNTIME

HTTP request
  ↓
API handler
  ↓
query preprocessing
  ↓
embed(query)
  ↓
vector search / hybrid search
  ↓
branching on scores
  ↓
rerank/dedup/expand
  ↓
context packing
  ↓
LLM prompt
  ↓
answers + citations + logs

Key point: each rectangle is ordinary code, not “model magic”

A common thinking mistake: mixing indexing and query processing. Indexing-time makes documents suitable for retrieval. Query-time uses an already built index and makes decisions on a specific user question. These are different budgets, different bugs and different logs.

To understand RAG, it's useful to think not about “the documents found something”, but about “what specific objects are running between functions.” Typically these are chunk record, query embedding, search hit, rerank hit, context packed entry and final answer object.

ChunkRecord:
  chunk_id
  doc_id
  text
  embedding
  metadata

SearchHit:
  chunk_id
  score
  payload

RerankHit:
  chunk_id
  retrieval_score
  rerank_score

ContextItem:
  chunk_id
  text
  token_count
  citation_label

The base usually does not “react” to `cosine similarity`. Your orchestrator code in the application reacts. Vector DB simply returns a list of hits. Then the Python/TypeScript/Go code decides: whether to filter by threshold, whether to rerank, whether to expand to parent-doc, whether to refuse or call LLM.

vector_db.search(...)
  → returns hits[]

not vector DB does:
  if top_score < threshold: ...
  if use_reranker: ...
  if not enough_context: ...

this is what the application layer/retrieval service does

Indexing usually begins not with embedding, but with document cleaning. Text is extracted from PDF/HTML/Markdown, cut by structure or by tokens, sometimes adding headings and a path to a section. The output is a list of chunk objects, not one big document string.

raw_docs = loader.load()

for doc in raw_docs:
  parsed = parser.parse(doc)
  chunks = chunker.split(parsed)
  for chunk in chunks:
    enqueue_for_embedding(chunk)

A good RAG will almost never store just `text + vector` in its index. Usually there are `doc_id`, `section_title`, `source_url`, `page`, `created_at`, `access_scope`, `token_count`, offsets and the document version nearby. Then it is these fields that decide how filter, dedup, citations and ACL will work.

payload = {
  "doc_id": "handbook-42",
  "chunk_id": "handbook-42::17",
  "text": "...",
  "section": "Vacation Policy",
  "page": 8,
  "token_count": 173,
  "source_url": "...",
  "tenant_id": "acme",
  "updated_at": "2026-03-07"
}

At the embedding stage, the code calls the embedding model in batches and receives a vector for each chunk. Often, vectors are additionally normalized, because for text, cosine similarity after L2 normalization is convenient to interpret and quickly calculate as a dot product.

batch_texts = [chunk.text for chunk in chunks]
vectors = embedder.embed_documents(batch_texts)

for chunk, vec in zip(chunks, vectors):
  vec = l2_normalize(vec)
  store.append({
    "chunk_id": chunk.id,
    "embedding": vec,
    "payload": chunk.payload,
  })

Upsert typically stores three things: ID, vector, and payload. This could be pgvector, Qdrant, Weaviate, Pinecone, FAISS + a separate KV table. An important point: similarity is not calculated in advance. The index contains only vectors and search service structures.

upsert(
  id="handbook-42::17",
  vector=[0.018, -0.331, ...],
  payload={...}
)

At this moment there is no query and no cosine score
There is only a willingness to quickly compare the future query vector with stored vectors

If the collection is large, a complete search of all vectors is too expensive. Therefore, vector DB builds an ANN structure like HNSW or IVF. It doesn’t “know the meaning”, but simply helps you quickly navigate to similar vectors without viewing all the elements.

WHAT APPEARS AFTER INDEX BUILD

before build:
  raw vectors + payload

after build:
  raw vectors + payload
  + graph / inverted lists / neighbor links

at query-time:
  search traverses index
  → gets candidate ids
  → calculates distance/similarity for candidates
  → returns top-k

Before embedding, queries are sometimes rewritten: abbreviations are expanded, noise is reduced, entity names are normalized, and the user’s working context is added. This is not a “product-level idea”, but a specific function before `embed_query(query)`.

query = request.user_query

if use_rewrite:
  query = query_rewriter(query, chat_history)

filters = build_metadata_filters(request)
tenant  = request.tenant_id

A request usually produces one dense vector. Sometimes there are several of them: multi-query retrieval, HyDE, query decomposition. But the basic path is simple: the question string turns into `q_emb`, which then goes into vector search.

q_emb = embedder.embed_query(query)
q_emb = l2_normalize(q_emb)

hits = vector_store.search(
  vector=q_emb,
  limit=20,
  filters=filters
)

Search does not return the “correct answer”. It returns a list of closest candidates. This is usually an array of objects with `id`, `score`, `payload`. In FAISS these can be separate `distances` and `indices`; in pgvector it is SQL rows; in Qdrant these are points with `score` and `payload`.

FAISS:
D, I = index.search(q_emb, k)
# D = distances/scores, I = vector ids

pgvector:
SELECT chunk_id, text, 1 - (embedding <=> :q) AS cosine_sim
FROM chunks
ORDER BY embedding <=> :q
LIMIT 20;

Qdrant-style:
[
  {id: "42", score: 0.812, payload: {...}},
  {id: "17", score: 0.791, payload: {...}}
]

The returned `0.82` does not mean “82% that this chunk is correct”. This is simply a measure of proximity in embedding space for this model and this collection. Different embedding models give different score ranges, so the thresholds are calibrated empirically based on your task.

Then the orchestrator receives a list of hits and begins the usual branching logic. It does not have to immediately call LLM. First it can filter, deduplicate, expand to parent-doc, run a reranker, or even return `no answer` if the evidence is too weak.

hits = vector_store.search(..., limit=20)

if not hits:
  return no_context_response()

hits = [h for h in hits if h.score >= RETRIEVAL_MIN_SCORE]

if not hits:
  return low_confidence_response()

if use_reranker:
  hits = rerank(query, hits)

context = pack_context(hits, token_budget=3000)
return answer_with_llm(query, context)

The first thing that is often done after retrieval is to cut off frankly weak candidates. Threshold can be by top-1, by each hit, or by the difference between the first and second. This is simple logic, but it is what often decides whether a system will hallucinate or honestly refuse to respond.

top1 gate:
  if hits[0].score < 0.72: refuse

per-hit gate:
  keep only hits with score >= 0.68

margin gate:
  if hits[0].score - hits[1].score < 0.02:
    route_to_reranker()

Search often returns several adjacent or almost identical chunks from the same document. If you insert them as is, LLM will receive noise and waste tokens. Therefore, the code often merges hits by `doc_id`, throws out near-duplicates and leaves only the best representatives or merges contiguous chunks.

hits = dedup_by_chunk_id(hits)
hits = dedup_by_similarity(hits, cutoff=0.97)
hits = group_by_doc(hits)

if expand_neighbors:
  hits = add_adjacent_chunks(hits, window=1)

In many RAG systems, dense retrieval is only one channel. BM25, keyword search or metadata filters can run in parallel. Then the code does the fusion: for example, through RRF or simply through heuristic merge. This is important because exact IDs, error codes and rare names are sometimes missed by dense retrieval.

dense_hits = vector_store.search(...)
sparse_hits = bm25.search(...)

merged = reciprocal_rank_fusion(dense_hits, sparse_hits)
hits = merged[:20]

Dense search is usually good at recall, but not great at final ranking. Therefore, the next runtime-step is often a reranker. It receives `query + candidate text` and returns a new score for each candidate. After this, the application sorts by `rerank_score`, and not by the original `cosine`.

# retrieval output
[
  {chunk_id: 42, score: 0.81},
  {chunk_id: 17, score: 0.80},
  {chunk_id: 91, score: 0.79}
]

# reranker output
[
  {chunk_id: 17, retrieval_score: 0.80, rerank_score: 0.94},
  {chunk_id: 42, retrieval_score: 0.81, rerank_score: 0.71},
  {chunk_id: 91, retrieval_score: 0.79, rerank_score: 0.19}
]

final order = sort by rerank_score desc

After reranking, you don’t need to “transfer everything to LLM,” but stay within the token budget. Therefore, the code counts tokens, trims long fragments, sometimes compresses them, or takes only top-3/top-5. This step is called context packing and is critical to the quality of the response.

budget = 3000
packed = []

for hit in hits:
  n = count_tokens(hit.text)
  if n > budget:
    continue
  packed.append(hit)
  budget -= n

if not packed:
  return refusal("No supporting context")

Chunks can be placed in the prompt in different ways: strictly according to rerank score, grouping by documents, placing the best one at the end, inserting headers and source labels. This is not a layout detail. This function determines which pieces will be closer to the last prompt tokens and will have a stronger influence on the model.

for i, hit in enumerate(packed, start=1):
  prompt_context += f"[SOURCE {i}]\\n"
  prompt_context += f"title: {hit.payload['section']}\\n"
  prompt_context += hit.payload["text"] + "\\n\\n"

Prompt builder doesn't just “insert chunks of text.” It defines the response format, rejection rules, quoting method, the distinction between instructions and retrieved context, and sometimes also the schema for structured output. Very often the quality of RAG breaks down here, although the retrieval was good.

SYSTEM:
  Answer only from the provided sources.
  If evidence is insufficient, say you do not know.
  Cite source ids in square brackets.

USER QUESTION:
  {query}

CONTEXT:
  {packed_sources}

After packing and prompt build, everything usually comes down to the usual chat completion call. But this is where retrieval gets expensive in terms of tokens: the retrieved context often makes up the bulk of the input prompt. Therefore, the RAG system pays not only for search latency, but also for the long prefill in LLM.

response = llm.chat(
  model="...",
  temperature=0,
  messages=[
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_block_with_context}
  ]
)

If the retrieval is weak, it is better to refuse or ask for clarification than to force the LLM to “think it through.” Therefore, there is almost always a separate branch: `no hits`, `low confidence`, `context not enough`, `conflicting evidence`. This is not a UX detail, but a part of core logic.

if not packed:
  return {
    "answer": "I don't see enough data in the sources.",
    "citations": [],
    "reason": "no_context"
  }

if top_rerank_score < 0.35:
  return clarification_request()

In order for answer to have links to sources, chunk ids and source labels must be saved before the LLM call. Then either the model itself writes `[SOURCE 2]`, or the post-processor maps the claims back to the context. If the payload does not contain stable source labels, normal grounding will not work.

packed = [
  {label: "SOURCE 1", chunk_id: "42", text: "..."},
  {label: "SOURCE 2", chunk_id: "17", text: "..."}
]

prompt tells model:
  cite labels like [SOURCE 1]

postprocess:
  map labels → real URLs/pages

Here is the most useful mental model for a developer: RAG is a regular function with clear inputs, intermediate objects and branches. Below is not a concept, but an almost skeleton production handler.

def answer_question(request):
    query = preprocess_query(request.query, request.history)
    filters = build_filters(request)

    q_emb = embedder.embed_query(query)
    q_emb = l2_normalize(q_emb)

    hits = vector_store.search(q_emb, limit=20, filters=filters)
    if not hits:
        return no_context_response()

    hits = [h for h in hits if h.score >= RETRIEVAL_MIN_SCORE]
    if not hits:
        return low_confidence_response()

    hits = dedup_hits(hits)
    hits = maybe_expand_neighbors(hits)
    hits = maybe_merge_sparse(query, hits)

    if use_reranker:
        hits = rerank(query, hits)

    packed = pack_context(hits, token_budget=MAX_CONTEXT_TOKENS)
    if not packed:
        return refusal_response()

    prompt = render_prompt(query, packed)
    raw_answer = llm.generate(prompt)
    answer = postprocess(raw_answer, packed)

    log_trace(query, hits, packed, answer)
    return answer

Common production logic: index small chunks for the sake of recall, but after a hit, expand them to the parent block or neighboring chunks. This is a separate runtime branch after retrieval, and not “a kind of embedding”.

hits = vector_store.search(...)
parent_ids = unique(h.payload["parent_id"] for h in hits[:5])
expanded = doc_store.fetch_parent_blocks(parent_ids)

packed = rerank_or_pack(expanded)

Some RAG systems don't stop at just search. If the context is weak, LLM or deterministic code builds an additional query, extends filters, includes another index, or makes a web/tool ​​branch. This is another regular control-flow branch.

hits = search(query)

if evidence_is_weak(hits):
  follow_up_query = reformulate(query)
  more_hits = search(follow_up_query)
  hits = merge(hits, more_hits)

If the top-1 hit is very strong and the payload contains a ready-made FAQ-answer or structured field, the application may not call LLM at all. This is especially useful for FAQ, knowledge lookup and semantic cache. That is, after cosine similarity, not only the “in prompt” branch is possible, but also the “return a ready-made answer” branch.

hits = search(query)

if hits[0].score > FAQ_DIRECT_THRESHOLD and hits[0].payload["answer"]:
  return {
    "answer": hits[0].payload["answer"],
    "source": hits[0].payload["source_url"],
    "mode": "retrieval_only"
  }

When LLM starts streaming tokens, the retrieval has already ended, but the pipeline is not yet there. Next can be markdown cleanup, schema validation, extraction of cited source labels, moderation, analytics events and fallback if the model output does not pass the test.

raw_answer = llm.stream(...)
answer = collect_stream(raw_answer)
answer = validate_schema(answer)
answer = attach_citation_urls(answer, packed)
answer = moderate(answer)
return answer

If you don't log intermediate artifacts, RAG becomes a black box. To debug, you need to see separately: input query, rewritten query, filters, top-k retrieval hits, scores, rerank scores, packed chunk ids, final prompt size and answer. Without this, it is impossible to understand whether retrieval failed or generation.

trace = {
  "query": "...",
  "rewritten_query": "...",
  "filters": {...},
  "retrieval_hits": [...],
  "rerank_hits": [...],
  "packed_chunk_ids": [...],
  "packed_token_count": 2714,
  "llm_model": "...",
  "answer": "...",
  "latency_ms": {...}
}

When the response is bad, the stage needs to be localized. If the required chunk is not in top-k, the problem is in indexing or retrieval. If there is a chunk, but it has sunk below, the problem is rerank or packing. If the chunk is in prompt, but the answer still does not follow the source, the problem is already in prompt or generation.

QUICK DEBUG

1. Was the required fact even in the database?
   no → indexing bug

2. Did he get into the top-k retrieval?
   no → retrieval / embedding / chunking bug

3. Did he get into a packed context?
   no → rerank / token budget / ordering bug

4. Was it in the prompt, but LLM ignored it?
   yes → prompt / refusal / generation bug

RAG needs to be measured layer by layer. Retrieval has its own recall/precision metrics. Ranking has its own hit@k and rerank gain. Generation - faithfulness, answer relevance, refusal accuracy. Plus infrastructure metrics: latency, cost, cache hit rate, index freshness.

When you see `cosine similarity = 0.81`, this is not an answer or a solution. It's just one of the inputs to the next layer of code. Everything important in production RAG happens next: thresholds, rerank, dedup, packing, prompting, refusal, citations and observability. Therefore, RAG should be designed not as a “vector database connected”, but as a full-fledged runtime pipeline.