What a Prompt Is and Why a Question Becomes an Answer

The most important thing is that the model doesn’t just look at your last sentence. It looks at the entire prefix that was provided as input. If the input already has a history of dialogue and a label that it is the assistant’s turn, then the “next token” should already be the assistant’s response token, rather than a continuation of the user’s question.

Not like that.

user: "Why is the sky blue?"
  ↓
The model is supposed to continue this question.

And so.

system: "You're a useful assistant"
user: "Why is the sky blue?"
assistant:
  ↓
Now the next token should continue the assistant's move.

This is a subtle but important shift. The LLM does not think of “I have a question, I have to answer.” It sees a pattern where after a user message in the assistant slot, there is usually a response, explanation, clarification or refusal. The answer appears as a statistically plausible continuation of this format.

The answer is not born entirely. The model selects the first token, then this token is added to the input, then the second, third and so on are selected. Because of this, early tokens very strongly set the trajectory of the entire response.

prompt
Choose the first answer token
Append it to the input
Choose a second token
Append it to the input
Repeat to stop token/limit

A meaningful answer is simply a long chain of locally believable tokens within the desired mode. There's no contradiction. The fact that only one token is predicted at each step does not prevent the whole sequence from appearing as an explanation, dialogue, code, or step list.

Reasonable answer = many times in a row
Which token is most appropriate now
in this conversational state?

From an engineering point of view, prompt is everything that got into the model before the next token was generated. Not just your question, but the entire input: system instruction, dialogue history, retrieved context, tool results, separators, service tokens and sometimes hidden chat template. After tokenization, all this becomes a token ID sequence.

PROMPT = Everything That's Already in

system instruction
+ user message
+ previous assistant messages
+ tool outputs
+ retrieved docs
+ role tokens / separators
= final token prefix

The user sees only their own text, but the model almost always gets more. An SDK or server can automatically add system prompt, role markers, chat template, safety instructions, tool schema, and chunks of history. So “my prompt” and “real prompt” are often not the same thing.

People think of prompts as intentions and instructions. But the model at the input sees only tokens and their order. This is why wording, separators, role marking, JSON frames, and sample patterns influence behavior so much.

"Be brief and get JSON back."
→ tokenizer
→ [tokens...]
→ embeddings
→ transformer
→ next-token distribution

When you write a good prompt, you are not rewriting the weights of the network. You temporarily set the context within which the model unfolds one trajectory of behavior rather than another. At the end of the request, this context disappears and the weights remain the same.

WEIGHT AND PROMPT

weights = long-term abilities and statistics
prompt = current task and current scene

Prompt guides the use of weights, but does not replace learning

If the scales already have the right skills, prompt helps to activate them in the right form: briefly, in a table, as a tutor, as a reviewer, with quotes, step by step. But if there is no knowledge or skill in the scales at all, prompt will not create it out of thin air.

At the pretraining stage, the model sees a sea of texts with FAQs, forums, references, textbooks, documentation, interviews and dialogues. Therefore, even before instruction tuning, she already knows that after many question constructs, there are often explanations, definitions, lists or solutions.

Q: ... A: ...
Question: ... Answer: ...
User: ... Assistant: ...
Task: ... Solution:
Why ... ? Because...

After pretraining, the model has many language patterns, but does not have to be a convenient assistant. Instruction tuning shows her special pairs like “user instruction → helpful assistant response.” In this way, she learns that in a certain format of dialogue, the continuation should not be an abstract text, but a useful response.

Before and after

pretrained LM:
  It can continue many different texts.

instruction-tuned LM:
  In the chat-like format, the
  helpful / harmless / answer-like continuation

In the next stages, the model is further pushed toward answers that people rate as more helpful, safe, polite, and understandable. Therefore, in chat mode, she can not just respond, but prefers a response over a strange continuation or aggressive style.

This is the main answer to the initial confusion. In chat, the model doesn’t try to guess “what the end of your line will be.” Your line is over. Now you have to guess what usually comes after her in assistant-turn.

User Message Ended
slot assistant
The next token will start the assistant reply
So it's possible:
"Of course," "Therefore," "This," "Neuronet,"
Not a continuation of the user question.

If you submit raw text without role markers, mix user and assistant into one thread, break a template or give a strange few-shot format, the model can go into another continuation mode: start copying the style of the case, finishing the document, repeating the question or continuing the list instead of answering.

Most chat models inside still get a linear sequence of tokens. Just before that, the list of messages is serialized by the chat template: role markers, separators, sometimes special start/end tokens, and often an explicit prefix for the assistant’s response are added.

From Messages to Raw Text

[
  {role: "system", content: "..."},
  {role: "user", content: "..."}
]

↓ chat template

<system> ... </system>
<user> ... </user>
<assistant>

These are roles for us. For the model, signals in the token sequence. They help to distinguish: where is the top-level instruction, where is the user's request, where is the assistant's past response, where is the tool's result. This dramatically streamlines the behavior of the network.

The system defines the policy/framework
user sets the task
The place where the model continues the answer
The external result, which then reads the model

In many chat templates, the last piece of prompt looks like an open assistant-turn. That's the signal: now continue on behalf of the assistant. Therefore, the model did not “guess to start answering”; it was literally shown that now it is necessary to fill this slot.

...
<user> Why is the sky blue? </user >>
<assistant>

After that, the next token should be an assistant reply token.

Each new move does not create a new personality. It simply extends the current prompt: it now includes past answers, past errors, refinements, and tool outputs. That's why history is so powerful in behavior, and that's why it's sometimes cut or summarized.

When a model invokes an instrument, it is not “agent magic.” This is again a special pattern in the sequence of messages: the assistant asks for a tool call, runtime executes it, puts the tool result in the story, and then the model continues with this new text.

TOOL LOOP

user
→ assistant(tool call)
→ tool result
→ assistant(final answer)

Again, everything is resolved through the next token inside the right format.

Weights contain long-term patterns: language, style, code, facts, reasoning patterns, frequency, propensity for certain answers. Prompt contains a short-lived scene: what exactly needs to be done now, for whom, in what format and on what material. Weights without prompt-and do not know which mode to choose now. Prompt without weights can't do anything by itself.

Separation of work

weights:
  What model does

prompt:
  What they want now

generation:
  How these two layers interact in a particular launch

During normal communication, the model is not trained on your request in the classical sense. It just makes a forward pass on fixed weights. So after a query, it doesn’t become forever smarter or forever better at your topic.

inference:
  weights fixed
  activations change
  context changes

training / fine-tuning:
  weights update
  The model's ability to change

A useful intuition is this: weights set a huge field of possible continuations, and prompt brings the model to one area of this field. Therefore, the same network can look like a tutor, code assistant, summarizer or reviewer, although the weights are the same.

When you add retrieved passages or few-shot examples, you’re not recording those facts in weight. You just put them in the current prompt so that at that launch the model builds on them as part of the context. At the next launch without these pieces, the model may no longer have the same support.

RAG / FEW-SHOT

Add context to prompt
The model reads it
Change the distribution of the next token
The answer becomes more grounded.

But the weights themselves were not rewritten.

Prompt engineering changes behavior on the fly for a specific challenge. Fine tuning changes the weights themselves and thus changes the model more steadily. Therefore, prompt is good for instructions and format, and fine tuning is needed when you want a systemic shift in behavior or a specialized skill.

At each step, the network outputs logits throughout the dictionary. After softmax, the probability distribution is obtained, and then the system selects the following token: either the most likely or sampling strategy. After that, the token is added to the input, and the cycle repeats.

prompt
→ logits over vocab
→ probabilities
→ choose token
→ append token
→ repeat

If the temperature is lower, the model often takes the most likely tokens and behaves more predictably. If above, it explores the tail of the distribution more. It doesn’t make it smarter or less intelligent directly, it only changes the style of choice.

If the beginning of the answer went in the direction of “Briefly:”, “Of course!”, JSON format, list of points or refusal, then the model will continue inside this corridor. This is why prompt engineering has a particularly strong impact on the start of the response.

first token "{"
The answer goes into the JSON trajectory.

first token "1."
The answer goes into the list trajectory

first token "Sorry"
The answer goes into a refusal trajectory.

If prompt and training signal that there is not enough data, the more plausible continuation may not be the answer, but the clarification: “Which framework do you mean?” or “Clarify the period.” It's not a separate magic. This is again the choice of the most appropriate next move assistant.

Prompt engineering does not change the “soul of the model,” but rather the local distribution of probabilities in the first and next steps of generation. Added role, format, example, constraints, context, and made some sequels more likely and others less likely.

changed promptly
Changed internal activations
Changed the logits
Changed the probability of tokens
Changed the trajectory of the response

The complexity of behavior does not have to be a separate “thinking module.” It can arise from a very powerful model, which is able at every step to assess well which continuation in this context will be logical. Many local predictions in a row produce globally complex behaviors.

You can fine-tune the format, style, and Useful framing of the answer, but that doesn’t mean the model will always be factually right. If knowledge is lacking or the context is ambiguous, it can still generate a very convincing but incorrect continuation.

To avoid confusion, it is convenient to keep in mind this chain: weights store abilities and patterns, chat template designs the scene, prompt sets the current task and context, and generation by one token unfolds the most likely move of the assistant inside this scene.

Tom's Main Formula

weights
  + chat format
  + current prompt
  + decoding strategy
  =
trajectory of the next response

Prompt is the entire current context fed to the model prior to generation. In chat mode, this context is already framed in a way that the assistant is now saying. Therefore, the next-token prediction naturally gives not a continuation of your question, but the beginning of the assistant’s answer.

The model doesn't continue with your question.
And all the dialogue in the dot.
where the assistant's turn came

If you want to run an LLM, think not “how to get her to understand me,” but “what token context I’m creating for her now and what continuation mode I’m making most likely.” It is a more accurate and more useful mental model for prompt engineering, RAG, agents and structured output.