What a Prompt Is and Why a Question Becomes an Answer

The most important thing is that the model doesn’t just look at your last sentence. It looks at the entire prefix that was provided as input. If the input already has a history of dialogue and a label that it is the assistant’s turn, then the “next token” should already be the assistant’s response token, rather than a continuation of the user’s question.

This is a subtle but important shift. The LLM does not think of “I have a question, I have to answer.” It sees a pattern where after a user message in the assistant slot, there is usually a response, explanation, clarification or refusal. The answer appears as a statistically plausible continuation of this format.

The answer is not born entirely. The model selects the first token, then this token is added to the input, then the second, third and so on are selected. Because of this, early tokens very strongly set the trajectory of the entire response.

A meaningful answer is simply a long chain of locally believable tokens within the desired mode. There's no contradiction. The fact that only one token is predicted at each step does not prevent the whole sequence from appearing as an explanation, dialogue, code, or step list.

From an engineering point of view, prompt is everything that got into the model before the next token was generated. Not just your question, but the entire input: system instruction, dialogue history, retrieved context, tool results, separators, service tokens and sometimes hidden chat template. After tokenization, all this becomes a token ID sequence.

The user sees only their own text, but the model almost always gets more. An SDK or server can automatically add system prompt, role markers, chat template, safety instructions, tool schema, and chunks of history. So “my prompt” and “real prompt” are often not the same thing.

People think of prompts as intentions and instructions. But the model at the input sees only tokens and their order. This is why wording, separators, role marking, JSON frames, and sample patterns influence behavior so much.

When you write a good prompt, you are not rewriting the weights of the network. You temporarily set the context within which the model unfolds one trajectory of behavior rather than another. At the end of the request, this context disappears and the weights remain the same.

If the weights already encode the right skills, a prompt helps activate them in the right form: briefly, in a table, as a tutor, as a reviewer, with quotes, or step by step. But if the knowledge or skill is not encoded in the weights at all, a prompt will not create it out of thin air.

At the pretraining stage, the model sees a sea of texts with FAQs, forums, references, textbooks, documentation, interviews and dialogues. Therefore, even before instruction tuning, she already knows that after many question constructs, there are often explanations, definitions, lists or solutions.

After pretraining, the model has many language patterns, but does not have to be a convenient assistant. Instruction tuning shows her special pairs like “user instruction → helpful assistant response.” In this way, she learns that in a certain format of dialogue, the continuation should not be an abstract text, but a useful response.

In the next stages, the model is further pushed toward answers that people rate as more helpful, safe, polite, and understandable. Therefore, in chat mode, she can not just respond, but prefers a response over a strange continuation or aggressive style.

This is the main answer to the initial confusion. In chat, the model doesn’t try to guess “what the end of your line will be.” Your line is over. Now you have to guess what usually comes after her in assistant-turn.

If you submit raw text without role markers, mix user and assistant into one thread, break a template or give a strange few-shot format, the model can go into another continuation mode: start copying the style of the case, finishing the document, repeating the question or continuing the list instead of answering.

Most chat models inside still get a linear sequence of tokens. Just before that, the list of messages is serialized by the chat template: role markers, separators, sometimes special start/end tokens, and often an explicit prefix for the assistant’s response are added.

These are roles for us. For the model, signals in the token sequence. They help to distinguish: where is the top-level instruction, where is the user's request, where is the assistant's past response, where is the tool's result. This dramatically streamlines the behavior of the network.

In many chat templates, the last piece of prompt looks like an open assistant-turn. That's the signal: now continue on behalf of the assistant. Therefore, the model did not “guess to start answering”; it was literally shown that now it is necessary to fill this slot.

Each new move does not create a new personality. It simply extends the current prompt: it now includes past answers, past errors, refinements, and tool outputs. That's why history is so powerful in behavior, and that's why it's sometimes cut or summarized.

When a model invokes an instrument, it is not “agent magic.” This is again a special pattern in the sequence of messages: the assistant asks for a tool call, runtime executes it, puts the tool result in the story, and then the model continues with this new text.

Weights contain long-term patterns: language, style, code, facts, reasoning patterns, frequency, propensity for certain answers. Prompt contains a short-lived scene: what exactly needs to be done now, for whom, in what format and on what material. Weights without prompt-and do not know which mode to choose now. Prompt without weights can't do anything by itself.

During normal communication, the model is not trained on your request in the classical sense. It just makes a forward pass on fixed weights. So after a query, it doesn’t become forever smarter or forever better at your topic.

A useful intuition is this: weights set a huge field of possible continuations, and prompt brings the model to one area of this field. Therefore, the same network can look like a tutor, code assistant, summarizer or reviewer, although the weights are the same.

When you add retrieved passages or few-shot examples, you’re not recording those facts in weight. You just put them in the current prompt so that at that launch the model builds on them as part of the context. At the next launch without these pieces, the model may no longer have the same support.

Prompt engineering changes behavior on the fly for a specific challenge. Fine tuning changes the weights themselves and thus changes the model more steadily. Therefore, prompt is good for instructions and format, and fine tuning is needed when you want a systemic shift in behavior or a specialized skill.

At each step, the network outputs logits throughout the dictionary. After softmax, the probability distribution is obtained, and then the system selects the following token: either the most likely or sampling strategy. After that, the token is added to the input, and the cycle repeats.

If the temperature is lower, the model often takes the most likely tokens and behaves more predictably. If above, it explores the tail of the distribution more. It doesn’t make it smarter or less intelligent directly, it only changes the style of choice.

If the beginning of the answer went in the direction of “Briefly:”, “Of course!”, JSON format, list of points or refusal, then the model will continue inside this corridor. This is why prompt engineering has a particularly strong impact on the start of the response.

If prompt and training signal that there is not enough data, the more plausible continuation may not be the answer, but the clarification: “Which framework do you mean?” or “Clarify the period.” It's not a separate magic. This is again the choice of the most appropriate next move assistant.

Prompt engineering does not change the “soul of the model,” but rather the local distribution of probabilities in the first and next steps of generation. Added role, format, example, constraints, context, and made some sequels more likely and others less likely.

The complexity of behavior does not have to be a separate “thinking module.” It can arise from a very powerful model, which is able at every step to assess well which continuation in this context will be logical. Many local predictions in a row produce globally complex behaviors.

You can fine-tune the format, style, and Useful framing of the answer, but that doesn’t mean the model will always be factually right. If knowledge is lacking or the context is ambiguous, it can still generate a very convincing but incorrect continuation.

To avoid confusion, it is convenient to keep in mind this chain: weights store abilities and patterns, chat template designs the scene, prompt sets the current task and context, and generation by one token unfolds the most likely move of the assistant inside this scene.

Prompt is the entire current context fed to the model prior to generation. In chat mode, this context is already framed in a way that the assistant is now saying. Therefore, the next-token prediction naturally gives not a continuation of your question, but the beginning of the assistant’s answer.

If you want to run an LLM, think not “how to get her to understand me,” but “what token context I’m creating for her now and what continuation mode I’m making most likely.” It is a more accurate and more useful mental model for prompt engineering, RAG, agents and structured output.