---
type: "post"
id: "the-gradient-is-a-covariance-8-minute-read-5099f4"
slug: "the-gradient-is-a-covariance-8-minute-read-5099f4"
title: "The Gradient Is a Covariance (8 minute read)"
description: "**Thesis:** Sparse parameter updates are not explained by small magnitude alone; the relationship between the learning signal and parameter score determines which coordinates move."
retrieval_nugget: "**Thesis:** Sparse parameter updates are not explained by small magnitude alone; the relationship between the learning signal and parameter score determines which coordinates move."
published_at: "2026-10-16"
updated_at: "2026-10-16"
record_date: "2026-10-16"
date_kind: "published_at"
topics: ["models"]
source_urls: ["https://richardli.xyz/post/gradient-is-covariance"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:the-gradient-is-a-covariance-8-minute-read-5099f4"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:32094321378675e846efdbcc6c7ef55f56dd34c0bac0c7b0b179bf31a03ccd22","reviewed_at":"2026-10-16T20:11:23.635Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Sparse parameter updates are not explained by small magnitude alone; the relationship between the learning signal and parameter score determines which coordinates move.","observed_facts":[{"text":"The essay says that on samples drawn from the model, the score has zero mean, making the expected gradient a covariance and the diagonal Fisher a variance.","source_urls":["https://richardli.xyz/post/gradient-is-covariance"]},{"text":"The essay derives the bound |G_i| ≤ std(A)√F_i and explains that a large Fisher is necessary but not sufficient for a large gradient.","source_urls":["https://richardli.xyz/post/gradient-is-covariance"]},{"text":"The essay reports that the top 1% of coordinates by Fisher contain 47.40% of the top 1% by gradient magnitude in cited measurements.","source_urls":["https://richardli.xyz/post/gradient-is-covariance"]}],"mechanism":"Centering of the score under the sampling distribution turns the expected gradient into a covariance; Cauchy–Schwarz bounds it by signal spread and Fisher spread. At token level, common logit movement cancels and only relative candidate sensitivity contributes.","why_now":"The essay distinguishes small post-training updates from sparse updates, a distinction needed to interpret claims about how training changes model parameters.","implications":["Fisher identifies where large updates are possible, while reward alignment determines whether those updates occur.","Common movement of all candidate logits does not change token probabilities; differences among candidates matter."],"evidence_boundary":"This is a summary of one theoretical essay and its reported empirical result. The supplied source excerpt does not independently validate the measurement or establish that the mechanism generalizes across training setups.","watch_conditions":["Check whether the later parts of the series and independent evaluations clarify where the sparsity mechanism holds across algorithms, sampling distributions, and training data."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied, so this source-grounded theoretical explanation starts a distinct research record rather than being attached to an invented predecessor."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/the-gradient-is-a-covariance-8-minute-read-5099f4/","markdown":"https://newruntime.com/posts/the-gradient-is-a-covariance-8-minute-read-5099f4.md","json":"https://newruntime.com/posts/the-gradient-is-a-covariance-8-minute-read-5099f4.json"}
---

# The Gradient Is a Covariance (8 minute read)

## Retrieval answer

**Thesis:** Sparse parameter updates are not explained by small magnitude alone; the relationship between the learning signal and parameter score determines which coordinates move.

**Thesis:** Sparse parameter updates are not explained by small magnitude alone; the relationship between the learning signal and parameter score determines which coordinates move.

Yingru Li’s essay says that on samples drawn from the model, the score has zero mean. **Mechanism:** Centering of the score under the sampling distribution turns the expected gradient into a covariance; Cauchy–Schwarz bounds it by signal spread and Fisher spread. At token level, common logit movement cancels and only relative candidate sensitivity contributes. The essay also derives the bound |G_i| ≤ std(A)√F_i and says a large Fisher is necessary but not sufficient for a large gradient.

**Why now:** The essay distinguishes small post-training updates from sparse updates, a distinction needed to interpret claims about how training changes model parameters. **First implication:** Fisher identifies where large updates are possible, while reward alignment determines whether those updates occur. The essay reports that, in cited measurements, the top 1% of coordinates by Fisher contained 47.40% of the top 1% by gradient magnitude; that is a reported result, not a general guarantee.

**Evidence boundary:** This is a summary of one theoretical essay and its reported empirical result. The supplied source excerpt does not independently validate the measurement or establish that the mechanism generalizes across training setups. It is part one of a three-part series. **First watch condition:** Check whether the later parts of the series and independent evaluations clarify where the sparsity mechanism holds across algorithms, sampling distributions, and training data.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Models](https://newruntime.com/topics/models.md): Explore the models topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)