{
"type": "new-runtime-knowledge-guide",
"version": 1,
"generated_at": "2026-07-23",
"volume": 23,
"slug": "prompt-dialog-mode-reference",
"title": "What a Prompt Is and Why a Question Becomes an Answer",
"description": "The main paradox of LLM is this: if the model is just predicting the next token, why does it answer the question meaningfully, and not just continue our text? This volume analyzes the prompt as the model's actual input, chat mode as a special data format, instruction tuning as a shift in behavior towards the “assistant”, and the connection of prompt with the model weights.",
"track": {
"slug": "llm-foundations",
"title": "LLM Foundations",
"route": "https://newruntime.com/learn/tracks/llm-foundations/",
"position": 5
},
"counts": {
"sections": 7,
"cards": 34
},
"routes": {
"html": "https://newruntime.com/learn/prompt-dialog-mode-reference/",
"json": "https://newruntime.com/learn/prompt-dialog-mode-reference.json"
},
"sections": [
{
"number": "01",
"title": "The Great Paradox",
"intro": "",
"cards": [
{
"number": "01",
"title": "LLM really predicts the next token. But the next token counts after the whole context.",
"tag": "Paradox",
"description": "The most important thing is that the model doesn’t just look at your last sentence. It looks at the entire prefix that was provided as input. If the input already has a history of dialogue and a label that it is the assistant’s turn, then the “next token” should already be the assistant’s response token, rather than a continuation of the user’s question.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "Not like that.\n\nuser: \"Why is the sky blue?\"\n ↓\nThe model is supposed to continue this question.\n\nAnd so.\n\nsystem: \"You're a useful assistant\"\nuser: \"Why is the sky blue?\"\nassistant:\n ↓\nNow the next token should continue the assistant's move."
}
]
},
{
"number": "02",
"title": "The model does not answer the question. It continues the format in which questions are usually answered.",
"tag": "Paradox",
"description": "This is a subtle but important shift. The LLM does not think of “I have a question, I have to answer.” It sees a pattern where after a user message in the assistant slot, there is usually a response, explanation, clarification or refusal. The answer appears as a statistically plausible continuation of this format.",
"blocks": [
{
"kind": "list",
"label": "Useful framing",
"items": [
"The model continues the interaction format",
"The answer is continuation in the right mode.",
"The model understands the question as a person in an explicit form"
]
}
]
},
{
"number": "03",
"title": "One meaningful answer is many small next-token steps in a row.",
"tag": "Paradox",
"description": "The answer is not born entirely. The model selects the first token, then this token is added to the input, then the second, third and so on are selected. Because of this, early tokens very strongly set the trajectory of the entire response.",
"blocks": [
{
"kind": "code",
"label": "step by step",
"language": "text",
"value": "prompt\nChoose the first answer token\nAppend it to the input\nChoose a second token\nAppend it to the input\nRepeat to stop token/limit"
}
]
},
{
"number": "04",
"title": "Therefore, the “next token” and the “meaningful response” do not contradict each other.",
"tag": "Paradox",
"description": "A meaningful answer is simply a long chain of locally believable tokens within the desired mode. There's no contradiction. The fact that only one token is predicted at each step does not prevent the whole sequence from appearing as an explanation, dialogue, code, or step list.",
"blocks": [
{
"kind": "code",
"label": "formula",
"language": "tex",
"value": "Reasonable answer = many times in a row\nWhich token is most appropriate now\nin this conversational state?"
}
]
}
]
},
{
"number": "02",
"title": "What is Prompt by Essence?",
"intro": "",
"cards": [
{
"number": "05",
"title": "Prompt is not a “query idea” but a specific tokenized prefix",
"tag": "Prompt",
"description": "From an engineering point of view, prompt is everything that got into the model before the next token was generated. Not just your question, but the entire input: system instruction, dialogue history, retrieved context, tool results, separators, service tokens and sometimes hidden chat template. After tokenization, all this becomes a token ID sequence.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "PROMPT = Everything That's Already in\n\nsystem instruction\n+ user message\n+ previous assistant messages\n+ tool outputs\n+ retrieved docs\n+ role tokens / separators\n= final token prefix"
}
]
},
{
"number": "06",
"title": "Prompt is visible and invisible",
"tag": "Prompt",
"description": "The user sees only their own text, but the model almost always gets more. An SDK or server can automatically add system prompt, role markers, chat template, safety instructions, tool schema, and chunks of history. So “my prompt” and “real prompt” are often not the same thing.",
"blocks": [
{
"kind": "list",
"label": "frequently",
"items": [
"system message",
"assistant prefix",
"tool schema",
"history trimming / formatting"
]
}
]
},
{
"number": "07",
"title": "For the model, prompt is not “meaning points”, but a sequence of tokens.",
"tag": "Prompt",
"description": "People think of prompts as intentions and instructions. But the model at the input sees only tokens and their order. This is why wording, separators, role marking, JSON frames, and sample patterns influence behavior so much.",
"blocks": [
{
"kind": "code",
"label": "inside",
"language": "text",
"value": "\"Be brief and get JSON back.\"\n→ tokenizer\n→ [tokens...]\n→ embeddings\n→ transformer\n→ next-token distribution"
}
]
},
{
"number": "08",
"title": "Prompt is a temporal context, not a brain change model",
"tag": "Prompt",
"description": "When you write a good prompt, you are not rewriting the weights of the network. You temporarily set the context within which the model unfolds one trajectory of behavior rather than another. At the end of the request, this context disappears and the weights remain the same.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "WEIGHT AND PROMPT\n\nweights = long-term abilities and statistics\nprompt = current task and current scene\n\nPrompt guides the use of weights, but does not replace learning"
}
]
},
{
"number": "09",
"title": "A good prompt “chooses mode” rather than “adds intelligence”",
"tag": "Prompt",
"description": "If the scales already have the right skills, prompt helps to activate them in the right form: briefly, in a table, as a tutor, as a reviewer, with quotes, step by step. But if there is no knowledge or skill in the scales at all, prompt will not create it out of thin air.",
"blocks": [
{
"kind": "list",
"label": "prompt is good.",
"items": [
"style",
"format",
"choose",
"Embedding a new area of knowledge in weight"
]
}
]
}
]
},
{
"number": "03",
"title": "Why the question turns into an answer",
"intro": "",
"cards": [
{
"number": "10",
"title": "Pretraining already teaches the model a huge number of templates “question → answer”",
"tag": "Dialog",
"description": "At the pretraining stage, the model sees a sea of texts with FAQs, forums, references, textbooks, documentation, interviews and dialogues. Therefore, even before instruction tuning, she already knows that after many question constructs, there are often explanations, definitions, lists or solutions.",
"blocks": [
{
"kind": "code",
"label": "pattern",
"language": "text",
"value": "Q: ... A: ...\nQuestion: ... Answer: ...\nUser: ... Assistant: ...\nTask: ... Solution:\nWhy ... ? Because..."
}
]
},
{
"number": "11",
"title": "Instruction tuning shifts the model from “continue text” to “help the user”",
"tag": "Dialog",
"description": "After pretraining, the model has many language patterns, but does not have to be a convenient assistant. Instruction tuning shows her special pairs like “user instruction → helpful assistant response.” In this way, she learns that in a certain format of dialogue, the continuation should not be an abstract text, but a useful response.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "Before and after\n\npretrained LM:\n It can continue many different texts.\n\ninstruction-tuned LM:\n In the chat-like format, the\n helpful / harmless / answer-like continuation"
}
]
},
{
"number": "12",
"title": "RLHF/preference tuning reinforces “assistance” behavior",
"tag": "Dialog",
"description": "In the next stages, the model is further pushed toward answers that people rate as more helpful, safe, polite, and understandable. Therefore, in chat mode, she can not just respond, but prefers a response over a strange continuation or aggressive style.",
"blocks": [
{
"kind": "list",
"label": "reinforcement",
"items": [
"utility",
"politeness",
"formatting",
"Absolute truth is not guaranteed."
]
}
]
},
{
"number": "13",
"title": "The model is not “obliged” to continue the question with the question mark, because the assistant is in line.",
"tag": "Dialog",
"description": "This is the main answer to the initial confusion. In chat, the model doesn’t try to guess “what the end of your line will be.” Your line is over. Now you have to guess what usually comes after her in assistant-turn.",
"blocks": [
{
"kind": "code",
"label": "literally",
"language": "text",
"value": "User Message Ended\nslot assistant\nThe next token will start the assistant reply\nSo it's possible:\n\"Of course,\" \"Therefore,\" \"This,\" \"Neuronet,\"\nNot a continuation of the user question."
}
]
},
{
"number": "14",
"title": "If the prompt format is knocked down, the model may begin to behave “not like a chat”",
"tag": "Dialog",
"description": "If you submit raw text without role markers, mix user and assistant into one thread, break a template or give a strange few-shot format, the model can go into another continuation mode: start copying the style of the case, finishing the document, repeating the question or continuing the list instead of answering.",
"blocks": [
{
"kind": "list",
"label": "hence",
"items": [
"format",
"broken format = strange behavior"
]
}
]
}
]
},
{
"number": "04",
"title": "What is Dialog Mode Technically?",
"intro": "",
"cards": [
{
"number": "15",
"title": "Chat mode almost always turns into a regular text sequence through a template.",
"tag": "Dialog",
"description": "Most chat models inside still get a linear sequence of tokens. Just before that, the list of messages is serialized by the chat template: role markers, separators, sometimes special start/end tokens, and often an explicit prefix for the assistant’s response are added.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "sql",
"value": "From Messages to Raw Text\n\n[\n {role: \"system\", content: \"...\"},\n {role: \"user\", content: \"...\"}\n]\n\n↓ chat template\n\n ... \n ... \n"
}
]
},
{
"number": "16",
"title": "System, user, assistant are not magical entities, but part of a format.",
"tag": "Dialog",
"description": "These are roles for us. For the model, signals in the token sequence. They help to distinguish: where is the top-level instruction, where is the user's request, where is the assistant's past response, where is the tool's result. This dramatically streamlines the behavior of the network.",
"blocks": [
{
"kind": "code",
"label": "role-play",
"language": "sql",
"value": "The system defines the policy/framework\nuser sets the task\nThe place where the model continues the answer\nThe external result, which then reads the model"
}
]
},
{
"number": "17",
"title": "Assistant prefix is one of the most important hidden pieces of prompt. and",
"tag": "Dialog",
"description": "In many chat templates, the last piece of prompt looks like an open assistant-turn. That's the signal: now continue on behalf of the assistant. Therefore, the model did not “guess to start answering”; it was literally shown that now it is necessary to fill this slot.",
"blocks": [
{
"kind": "code",
"label": "The latest prompt tokens are often similar to",
"language": "text",
"value": "...\n Why is the sky blue? >\n\n\nAfter that, the next token should be an assistant reply token."
}
]
},
{
"number": "18",
"title": "A multi-way dialogue is just a longer and longer prompt.",
"tag": "Dialog",
"description": "Each new move does not create a new personality. It simply extends the current prompt: it now includes past answers, past errors, refinements, and tool outputs. That's why history is so powerful in behavior, and that's why it's sometimes cut or summarized.",
"blocks": [
{
"kind": "list",
"label": "consequences",
"items": [
"history = part of the context",
"A dirty story pollutes the answer",
"Too long a story is expensive."
]
}
]
},
{
"number": "19",
"title": "Tool-calling is also a special dialogue format.",
"tag": "Dialog",
"description": "When a model invokes an instrument, it is not “agent magic.” This is again a special pattern in the sequence of messages: the assistant asks for a tool call, runtime executes it, puts the tool result in the story, and then the model continues with this new text.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "TOOL LOOP\n\nuser\n→ assistant(tool call)\n→ tool result\n→ assistant(final answer)\n\nAgain, everything is resolved through the next token inside the right format."
}
]
}
]
},
{
"number": "05",
"title": "How Prompt Is Linked to Libra",
"intro": "",
"cards": [
{
"number": "20",
"title": "Weights are compressed statistics and skills. Prompt is the current task and the current data",
"tag": "Weights",
"description": "Weights contain long-term patterns: language, style, code, facts, reasoning patterns, frequency, propensity for certain answers. Prompt contains a short-lived scene: what exactly needs to be done now, for whom, in what format and on what material. Weights without prompt-and do not know which mode to choose now. Prompt without weights can't do anything by itself.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "Separation of work\n\nweights:\n What model does\n\nprompt:\n What they want now\n\ngeneration:\n How these two layers interact in a particular launch"
}
]
},
{
"number": "21",
"title": "Prompt is not recorded in weight during normal inference",
"tag": "Weights",
"description": "During normal communication, the model is not trained on your request in the classical sense. It just makes a forward pass on fixed weights. So after a query, it doesn’t become forever smarter or forever better at your topic.",
"blocks": [
{
"kind": "code",
"label": "what is/what is not",
"language": "text",
"value": "inference:\n weights fixed\n activations change\n context changes\n\ntraining / fine-tuning:\n weights update\n The model's ability to change"
}
]
},
{
"number": "22",
"title": "It is better to think that prompt “swifts” the weights.",
"tag": "Weights",
"description": "A useful intuition is this: weights set a huge field of possible continuations, and prompt brings the model to one area of this field. Therefore, the same network can look like a tutor, code assistant, summarizer or reviewer, although the weights are the same.",
"blocks": [
{
"kind": "list",
"label": "metaphor",
"items": [
"weights = landscape of possibilities",
"prompt = route through the landscape"
]
}
]
},
{
"number": "23",
"title": "RAGs and prompt examples do not “teach the model” but temporarily supply it with facts and templates.",
"tag": "Weights",
"description": "When you add retrieved passages or few-shot examples, you’re not recording those facts in weight. You just put them in the current prompt so that at that launch the model builds on them as part of the context. At the next launch without these pieces, the model may no longer have the same support.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "RAG / FEW-SHOT\n\nAdd context to prompt\nThe model reads it\nChange the distribution of the next token\nThe answer becomes more grounded.\n\nBut the weights themselves were not rewritten."
}
]
},
{
"number": "24",
"title": "Fine-tuning and prompt engineering solve different problems",
"tag": "Weights",
"description": "Prompt engineering changes behavior on the fly for a specific challenge. Fine tuning changes the weights themselves and thus changes the model more steadily. Therefore, prompt is good for instructions and format, and fine tuning is needed when you want a systemic shift in behavior or a specialized skill.",
"blocks": [
{
"kind": "table",
"headers": [
"Method",
"What changes"
],
"rows": [
[
"Prompt",
"current call context"
],
[
"Fine-tuning",
"weight and long-term behavior"
]
]
}
]
}
]
},
{
"number": "06",
"title": "What Happens at Runtime",
"intro": "",
"cards": [
{
"number": "25",
"title": "During the answer, the model constantly recalculates the distribution by dictionary.",
"tag": "Runtime",
"description": "At each step, the network outputs logits throughout the dictionary. After softmax, the probability distribution is obtained, and then the system selects the following token: either the most likely or sampling strategy. After that, the token is added to the input, and the cycle repeats.",
"blocks": [
{
"kind": "code",
"label": "runtime loop",
"language": "text",
"value": "prompt\n→ logits over vocab\n→ probabilities\n→ choose token\n→ append token\n→ repeat"
}
]
},
{
"number": "26",
"title": "Temperature and sampling do not affect knowledge, but the choice of possible continuations.",
"tag": "Runtime",
"description": "If the temperature is lower, the model often takes the most likely tokens and behaves more predictably. If above, it explores the tail of the distribution more. It doesn’t make it smarter or less intelligent directly, it only changes the style of choice.",
"blocks": [
{
"kind": "list",
"label": "usually",
"items": [
"Low temperature is more stable",
"High temperature is more varied but noisier"
]
}
]
},
{
"number": "27",
"title": "The first 5-20 answer tokens set the entire future mode",
"tag": "Runtime",
"description": "If the beginning of the answer went in the direction of “Briefly:”, “Of course!”, JSON format, list of points or refusal, then the model will continue inside this corridor. This is why prompt engineering has a particularly strong impact on the start of the response.",
"blocks": [
{
"kind": "code",
"label": "example",
"language": "text",
"value": "first token \"{\"\nThe answer goes into the JSON trajectory.\n\nfirst token \"1.\"\nThe answer goes into the list trajectory\n\nfirst token \"Sorry\"\nThe answer goes into a refusal trajectory."
}
]
},
{
"number": "28",
"title": "Sometimes the model does not answer immediately, but clarifies the question. It's just another possible follow-up.",
"tag": "Runtime",
"description": "If prompt and training signal that there is not enough data, the more plausible continuation may not be the answer, but the clarification: “Which framework do you mean?” or “Clarify the period.” It's not a separate magic. This is again the choice of the most appropriate next move assistant.",
"blocks": [
{
"kind": "list",
"label": "what it looks like",
"items": [
"answer continuation",
"clarification continuation",
"refusal continuation"
]
}
]
},
{
"number": "29",
"title": "This is why prompt engineering works.",
"tag": "Runtime",
"description": "Prompt engineering does not change the “soul of the model,” but rather the local distribution of probabilities in the first and next steps of generation. Added role, format, example, constraints, context, and made some sequels more likely and others less likely.",
"blocks": [
{
"kind": "code",
"label": "mechanics",
"language": "text",
"value": "changed promptly\nChanged internal activations\nChanged the logits\nChanged the probability of tokens\nChanged the trajectory of the response"
}
]
}
]
},
{
"number": "07",
"title": "Myths, Boundaries and Final Framework",
"intro": "",
"cards": [
{
"number": "30",
"title": "Myth: “If it’s a next-token prediction, then the model can’t do anything complicated.”",
"tag": "Myth",
"description": "The complexity of behavior does not have to be a separate “thinking module.” It can arise from a very powerful model, which is able at every step to assess well which continuation in this context will be logical. Many local predictions in a row produce globally complex behaviors.",
"blocks": [
{
"kind": "list",
"label": "correctly",
"items": [
"Complex behavior can be emergent",
"Next-token = keyboard autocomplete only"
]
}
]
},
{
"number": "31",
"title": "Prompt does not guarantee the truth. It only changes the probability of response forms.",
"tag": "Boundaries",
"description": "You can fine-tune the format, style, and Useful framing of the answer, but that doesn’t mean the model will always be factually right. If knowledge is lacking or the context is ambiguous, it can still generate a very convincing but incorrect continuation.",
"blocks": [
{
"kind": "list",
"label": "consequence",
"items": [
"prompting - fact checking",
"Facts often require RAG/retrieval/tools"
]
}
]
},
{
"number": "32",
"title": "The most useful final formula",
"tag": "Conclusion",
"description": "To avoid confusion, it is convenient to keep in mind this chain: weights store abilities and patterns, chat template designs the scene, prompt sets the current task and context, and generation by one token unfolds the most likely move of the assistant inside this scene.",
"blocks": [
{
"kind": "code",
"label": "Diagram",
"language": "text",
"value": "Tom's Main Formula\n\nweights\n + chat format\n + current prompt\n + decoding strategy\n =\ntrajectory of the next response"
}
]
},
{
"number": "33",
"title": "Explain very briefly",
"tag": "Conclusion",
"description": "Prompt is the entire current context fed to the model prior to generation. In chat mode, this context is already framed in a way that the assistant is now saying. Therefore, the next-token prediction naturally gives not a continuation of your question, but the beginning of the assistant’s answer.",
"blocks": [
{
"kind": "code",
"label": "phrase",
"language": "sql",
"value": "The model doesn't continue with your question.\nAnd all the dialogue in the dot.\nwhere the assistant's turn came"
}
]
},
{
"number": "34",
"title": "The main engineering idea",
"tag": "Conclusion",
"description": "If you want to run an LLM, think not “how to get her to understand me,” but “what token context I’m creating for her now and what continuation mode I’m making most likely.” It is a more accurate and more useful mental model for prompt engineering, RAG, agents and structured output.",
"blocks": [
{
"kind": "list",
"label": "short reminder",
"items": [
"format",
"history",
"Slot is important.",
"Weights and prompt must be distinguished"
]
}
]
}
]
}
]
}