RAG Runtime: From Cosine Similarity to the Final Answer

In production RAG there is almost never a magic `rag(query)`. Usually there is an API handler, chunk store, embedder, vector DB, sometimes BM25, sometimes reranker, prompt builder, LLM client and post-processing. After each step, the code decides whether to move on, do a fallback, refuse a response, or ask for more data.

A common thinking mistake: mixing indexing and query processing. Indexing-time makes documents suitable for retrieval. Query-time uses an already built index and makes decisions on a specific user question. These are different budgets, different bugs and different logs.

To understand RAG, it's useful to think not about “the documents found something”, but about “what specific objects are running between functions.” Typically these are chunk record, query embedding, search hit, rerank hit, context packed entry and final answer object.

The base usually does not “react” to `cosine similarity`. Your orchestrator code in the application reacts. Vector DB simply returns a list of hits. Then the Python/TypeScript/Go code decides: whether to filter by threshold, whether to rerank, whether to expand to parent-doc, whether to refuse or call LLM.

Indexing usually begins not with embedding, but with document cleaning. Text is extracted from PDF/HTML/Markdown, cut by structure or by tokens, sometimes adding headings and a path to a section. The output is a list of chunk objects, not one big document string.

A good RAG will almost never store just `text + vector` in its index. Usually there are `doc_id`, `section_title`, `source_url`, `page`, `created_at`, `access_scope`, `token_count`, offsets and the document version nearby. Then it is these fields that decide how filter, dedup, citations and ACL will work.

At the embedding stage, the code calls the embedding model in batches and receives a vector for each chunk. Often, vectors are additionally normalized, because for text, cosine similarity after L2 normalization is convenient to interpret and quickly calculate as a dot product.

Upsert typically stores three things: ID, vector, and payload. This could be pgvector, Qdrant, Weaviate, Pinecone, FAISS + a separate KV table. An important point: similarity is not calculated in advance. The index contains only vectors and search service structures.

If the collection is large, a complete search of all vectors is too expensive. Therefore, vector DB builds an ANN structure like HNSW or IVF. It doesn’t “know the meaning”, but simply helps you quickly navigate to similar vectors without viewing all the elements.

Before embedding, queries are sometimes rewritten: abbreviations are expanded, noise is reduced, entity names are normalized, and the user’s working context is added. This is not a “product-level idea”, but a specific function before `embed_query(query)`.

A request usually produces one dense vector. Sometimes there are several of them: multi-query retrieval, HyDE, query decomposition. But the basic path is simple: the question string turns into `q_emb`, which then goes into vector search.

Search does not return the “correct answer”. It returns a list of closest candidates. This is usually an array of objects with `id`, `score`, `payload`. In FAISS these can be separate `distances` and `indices`; in pgvector it is SQL rows; in Qdrant these are points with `score` and `payload`.

The returned `0.82` does not mean “82% that this chunk is correct”. This is simply a measure of proximity in embedding space for this model and this collection. Different embedding models give different score ranges, so the thresholds are calibrated empirically based on your task.

Then the orchestrator receives a list of hits and begins the usual branching logic. It does not have to immediately call LLM. First it can filter, deduplicate, expand to parent-doc, run a reranker, or even return `no answer` if the evidence is too weak.

The first thing that is often done after retrieval is to cut off frankly weak candidates. Threshold can be by top-1, by each hit, or by the difference between the first and second. This is simple logic, but it is what often decides whether a system will hallucinate or honestly refuse to respond.

Search often returns several adjacent or almost identical chunks from the same document. If you insert them as is, LLM will receive noise and waste tokens. Therefore, the code often merges hits by `doc_id`, throws out near-duplicates and leaves only the best representatives or merges contiguous chunks.

In many RAG systems, dense retrieval is only one channel. BM25, keyword search or metadata filters can run in parallel. Then the code does the fusion: for example, through RRF or simply through heuristic merge. This is important because exact IDs, error codes and rare names are sometimes missed by dense retrieval.

Dense search is usually good at recall, but not great at final ranking. Therefore, the next runtime-step is often a reranker. It receives `query + candidate text` and returns a new score for each candidate. After this, the application sorts by `rerank_score`, and not by the original `cosine`.

After reranking, you don’t need to “transfer everything to LLM,” but stay within the token budget. Therefore, the code counts tokens, trims long fragments, sometimes compresses them, or takes only top-3/top-5. This step is called context packing and is critical to the quality of the response.

Chunks can be placed in the prompt in different ways: strictly according to rerank score, grouping by documents, placing the best one at the end, inserting headers and source labels. This is not a layout detail. This function determines which pieces will be closer to the last prompt tokens and will have a stronger influence on the model.

Prompt builder doesn't just “insert chunks of text.” It defines the response format, rejection rules, quoting method, the distinction between instructions and retrieved context, and sometimes also the schema for structured output. Very often the quality of RAG breaks down here, although the retrieval was good.

After packing and prompt build, everything usually comes down to the usual chat completion call. But this is where retrieval gets expensive in terms of tokens: the retrieved context often makes up the bulk of the input prompt. Therefore, the RAG system pays not only for search latency, but also for the long prefill in LLM.

If the retrieval is weak, it is better to refuse or ask for clarification than to force the LLM to “think it through.” Therefore, there is almost always a separate branch: `no hits`, `low confidence`, `context not enough`, `conflicting evidence`. This is not a UX detail, but a part of core logic.

In order for answer to have links to sources, chunk ids and source labels must be saved before the LLM call. Then either the model itself writes `[SOURCE 2]`, or the post-processor maps the claims back to the context. If the payload does not contain stable source labels, normal grounding will not work.

Here is the most useful mental model for a developer: RAG is a regular function with clear inputs, intermediate objects and branches. Below is not a concept, but an almost skeleton production handler.

Common production logic: index small chunks for the sake of recall, but after a hit, expand them to the parent block or neighboring chunks. This is a separate runtime branch after retrieval, and not “a kind of embedding”.

Some RAG systems don't stop at just search. If the context is weak, LLM or deterministic code builds an additional query, extends filters, includes another index, or makes a web/tool ​​branch. This is another regular control-flow branch.

If the top-1 hit is very strong and the payload contains a ready-made FAQ-answer or structured field, the application may not call LLM at all. This is especially useful for FAQ, knowledge lookup and semantic cache. That is, after cosine similarity, not only the “in prompt” branch is possible, but also the “return a ready-made answer” branch.

When LLM starts streaming tokens, the retrieval has already ended, but the pipeline is not yet there. Next can be markdown cleanup, schema validation, extraction of cited source labels, moderation, analytics events and fallback if the model output does not pass the test.

If you don't log intermediate artifacts, RAG becomes a black box. To debug, you need to see separately: input query, rewritten query, filters, top-k retrieval hits, scores, rerank scores, packed chunk ids, final prompt size and answer. Without this, it is impossible to understand whether retrieval failed or generation.

When the response is bad, the stage needs to be localized. If the required chunk is not in top-k, the problem is in indexing or retrieval. If there is a chunk, but it has sunk below, the problem is rerank or packing. If the chunk is in prompt, but the answer still does not follow the source, the problem is already in prompt or generation.

RAG needs to be measured layer by layer. Retrieval has its own recall/precision metrics. Ranking has its own hit@k and rerank gain. Generation - faithfulness, answer relevance, refusal accuracy. Plus infrastructure metrics: latency, cost, cache hit rate, index freshness.

When you see `cosine similarity = 0.81`, this is not an answer or a solution. It's just one of the inputs to the next layer of code. Everything important in production RAG happens next: thresholds, rerank, dedup, packing, prompting, refusal, citations and observability. Therefore, RAG should be designed not as a “vector database connected”, but as a full-fledged runtime pipeline.