What happened
Alex Wang’s post, “You could have invented transformers from memory,” describes neural architectures through the lens of associative memory. In this framing, a store receives a key and returns its associated value; its capacity depends on how much interference accumulated from other stored associations obscures the desired signal. The post compares an outer-product store, an optimal linear store, several MLP forms, and softmax self-attention using estimates based on this interference-to-signal perspective.
For an outer-product store, the capacity is about the key dimension. With a key dimension of 64, that means roughly 64 memories. An optimal linear store that whitens keys is estimated to hold about twice as many, or roughly 128 at the same key size. These are not claims about total parameters, the number of training examples, or the maximum length of a prompt. They are estimates of the number of associations that can be stored and retrieved under the analysis’s chosen setup.
The distinction matters because the analysis asks a particular question: when a query asks for one stored value, how much do the other stored values interfere with its recovery? It therefore treats capacity as a relationship among keys, values, and retrieval quality. A layer with more parameters is not automatically being assigned a larger figure merely because it is larger. Conversely, a capacity estimate is not a statement that every association will remain useful after training on a real distribution, or that a model will follow instructions based on it.
The essay gives larger estimates for MLP-based stores. A squared-ReLU GLU is estimated at about 2,700 memories, while a SwiGLU-style store is estimated at about 2,900. A cubed ReLU store reaches an estimate of about 35,000. These figures place nonlinear MLP variants in a different range from the simple linear examples in the post’s comparison. The point is not that one activation should be selected from these numbers alone, but that the form of the memory mechanism changes the expected interference behavior.
The post then considers softmax self-attention with sharpness set to the square root of the key dimension, estimating capacity at about 1.2 million memories. When sharpness is scaled with the logarithm of the number of memories, the reported experiments had not saturated and exceeded 2 million. Wang’s central architectural distinction is that self-attention is nonparametric in this account: each context token supplies one key-value memory at inference time. By contrast, linear maps and MLPs store associations in trained weights.
This makes the context itself an active memory surface for attention, rather than requiring every association to be encoded in parameters. It also clarifies why the article’s comparison is about mechanisms, not a leaderboard of complete language models. The supplied source material gives no product release, access conditions, price, API limits, or implementation package; it presents a conceptual analysis and numerical estimates.
What the archive adds
This is a new fact on a known research thread, rather than a restatement. Amazon Science’s priming method also examines how pretrained Transformer weights relate to alternative architectures, though Wang’s essay focuses on associative capacity rather than weight reuse.
The archive also contains work on agent memory, including a study of the memory tax. That is related in vocabulary but studies system-level retrieval over history; Wang is analyzing memory capacity within neural architectural components.
Trend signal
This essay does not materially establish a SITE_PATTERNS thesis. Its claims concern architecture-level associative capacity rather than a shipped task interface, agent runtime, governed knowledge layer, or operational workflow. It is most useful as background for interpreting later claims about model memory, not as evidence that a particular product or organizational pattern is already taking hold.
Why it matters
For builders, the value is conceptual rather than prescriptive. The comparison makes explicit that “memory” can refer to different mechanisms: learned parameters that compress regularities, or context tokens that act as individually addressable key-value entries. That distinction can help teams reason about where a system’s information resides and what changes when context is added, retrieved material is replaced, or weights are updated.
It also cautions against treating a context window and model training as interchangeable ways to provide knowledge. In the essay’s framing, attention can receive a fresh set of key-value memories at inference time, while associations in an MLP or linear map have been committed to learned weights. Product teams deciding whether to retrieve documents, maintain a prompt context, or fine-tune a model still need workload-specific evidence, but they should recognize that these choices operate on different memory surfaces.
The large attention estimates should not be read as a practical guarantee that a model can reliably retrieve millions of useful facts. They come from the post’s chosen assumptions and capacity measure, not from an end-to-end benchmark of a deployed model. Still, the framework is useful when explaining why adding context and training weights are not interchangeable operations, and why architecture comparisons need to specify what counts as a stored memory and what counts as successful retrieval.
Limits and uncertainty
The supplied material provides the essay’s summarized estimates, but not enough detail to independently reconstruct every derivation, experimental condition, distributional assumption, or variance around the figures. Capacity here is specific to an interference-to-signal framing; it is not a direct measure of factual accuracy, robustness, context utilization, instruction following, or production usefulness.
The post’s estimates should therefore not be compared as universal rankings across deployed systems. They do not establish that a real model will retrieve every context entry, remain reliable at a given prompt length, or preserve the same associations through fine-tuning. The source also offers no implementation, benchmark suite, cost comparison, or practical sizing rule for a particular workload. Builders would need separate experiments to connect this theory to a retrieval system or application decision.
Reader takeaway
Treat “memory capacity” as an architecture-dependent concept, not a single number. Wang’s useful distinction is between associations encoded in trained weights and key-value memories supplied by context tokens; the numerical estimates illustrate that distinction but do not promise real-world retrieval performance. Use the lens to ask where information lives before treating any memory claim as an operational guarantee.