A self-contained, build-up-from-zero guide · for AI researchers & curious undergrads

The Geometry of Intelligence
Riemannian & Information Geometry for AI

From the parallel postulate to natural gradients, hyperbolic transformers, and flow matching on manifolds — every concept defined from scratch, with intuition, worked numbers, pictures, and the exact papers that turned the math into state-of-the-art AI.

Non-Euclidean geometry Riemannian manifolds Fisher information Information geometry Natural gradient · K-FAC · TRPO Hyperbolic embeddings Flow matching & OT

What's inside

The 60-second map
Non-Euclidean geometry: breaking Euclid
Curvature & the Theorema Egregium
Riemannian geometry: the master framework
The metric tensor (the star of the show)
Geodesics, connections & parallel transport
Fisher information: probability gets a ruler
Information geometry: the grand synthesis
Natural gradient: geometry meets optimization
AI I — Natural gradient, K-FAC, TRPO
AI II — Fisher in deep learning
AI III — Hyperbolic & spherical deep learning
AI IV — Riemannian generative models & OT
Geometry in production — where this ships
Master cheat-sheet & further reading

1 · The 60-second map

Four topics, one idea: distance is not what you think it is. The shortest path between two points, the "angle" between two directions, even the question "are these two things far apart?" — all of these depend on a hidden object called a metric. Change the metric and you change the geometry. These four fields are four answers to "what is the right metric?"

The dependency graph of this document. Non-Euclidean geometry shows that Euclid's flat world is just one option. Riemannian geometry is the general theory of curved spaces. Fisher information discovers that the space of probability distributions is itself curved. Information geometry is what you get when you point Riemannian geometry at that space — and it is the bridge to most of the AI applications.

The one-paragraph version of each

① Non-Euclidean geometry — For 2000 years people tried to prove Euclid's 5th axiom (the parallel postulate) from the other four. They failed because it's independent: you can build perfectly consistent geometries where it's false. On a sphere there are no parallel lines and triangles bulge past 180°; in hyperbolic space there are infinitely many parallels and triangles are pinched below 180°. This was the shock that taught mathematics "flat" is a choice.
② Riemannian geometry — Riemann's leap: don't pick one curved space, give a recipe for all of them. At every point you attach a tiny ruler-and-protractor (the metric tensor $\Met$) that tells you the length of tiny steps. From that local rule alone you derive global length, shortest paths (geodesics), and curvature. This is the language of general relativity — and of modern geometric deep learning.
③ Fisher information — A statistics quantity: how sharply does the likelihood of your data respond to a change in a parameter? High Fisher information = the parameter is easy to pin down from data. Its deep secret: it is exactly the right metric tensor for the space of probability distributions.
④ Information geometry — Put ② and ③ together. The set of all distributions in a model (say, all Gaussians, or all settings of a neural net's softmax) forms a curved manifold whose metric is the Fisher information. Distances become "how distinguishable are these two models?", straight lines become natural-gradient updates, and the Pythagorean theorem reappears as the KL divergence. This is the engine under natural gradient, TRPO, EWC, and much more.

First, the one word everything depends on: "metric." The rest of this document hinges on it, so here it is concretely, no metaphors. A metric at a point is a symmetric, positive-definite $n\times n$ matrix $\Met$. It generalizes the dot product: the squared length of a tiny step $d\vx$ is $$ds^2 = d\vx^\top \Met\, d\vx \quad\text{instead of}\quad d\vx^\top d\vx.$$ Flat space is just $\Met = I$ (the identity matrix). "Curved space" means $\Met$ changes as you move. That's the whole idea — a position-dependent ruler.

30-second worked example. Take $\Met=\begin{pmatrix}4&0\\0&1\end{pmatrix}$ and the step $d\vx=\begin{pmatrix}1\\1\end{pmatrix}$. Ordinary (flat) length would be $\sqrt{1^2+1^2}=\sqrt2\approx1.41$. But with this metric: $$ds^2 = \begin{pmatrix}1&1\end{pmatrix}\begin{pmatrix}4&0\\0&1\end{pmatrix}\begin{pmatrix}1\\1\end{pmatrix}=4\cdot1^2+1\cdot1^2=5,\quad ds=\sqrt5\approx2.24.$$ The $x$-direction is "stretched" — its ruler is twice as long ($4=2^2$). That single idea — a matrix that reweights distance — is non-Euclidean geometry, Riemannian geometry, Fisher information, and information geometry, all four. They differ only in which matrix $\Met$ and where it comes from.

2 · Non-Euclidean geometry: breaking Euclid

Notation key (kept tiny on purpose — refer back anytime):

$\vx$ — a point (bold = vector/position) · $\|\vv\|$ — ordinary length of $\vv$ · $d\vx$ — a tiny step

$ds$ — the true (metric) length of that tiny step · $dA$ — a tiny patch of area · $\Met$ (or $G$) — the metric tensor (the position-dependent ruler) · $K$ — curvature

Everything starts with a 2300-year-old argument about one sentence.

2.1 Euclid's five postulates and the troublemaker

Around 300 BCE Euclid built all of plane geometry from five postulates. The first four are short and "obvious":

You can draw a straight line between any two points.
You can extend a line segment indefinitely.
You can draw a circle with any center and radius.
All right angles are equal.

The fifth is wordy and feels like a theorem in disguise:

The parallel postulate (Playfair's clean form). Given a line $\ell$ and a point $P$ not on it, there is exactly one line through $P$ that never meets $\ell$.

For two millennia mathematicians believed #5 must follow from #1–#4 and tried to prove it. Every "proof" secretly assumed something equivalent to #5. The resolution, reached independently by Gauss (unpublished), Bolyai (1832), and Lobachevsky (1829): the postulate is independent. You can deny it and get a new, fully consistent geometry. There are three outcomes, controlled by how many parallels exist through $P$:

Three geometries, one differing axiom. The number of lines through $P$ parallel to a given line is $1$ (flat), $0$ (sphere — every two "great circles" cross, like meridians at the poles), or $\infty$ (hyperbolic). A great circle is the largest circle on a sphere (the equator, any meridian) — a sphere's version of a straight line. The triangle's angle sum and the sign of the curvature $K$ are bound to that choice.

curvature K:

One triangle, one knob. Instead of three separate universes, here is a single triangle as you sweep the curvature $K$ from $-1$ to $+1$. $K>0$ (sphere-like): the edges bow outward and the angle sum climbs past $180°$. $K=0$: edges straight, sum exactly $180°$. $K<0$ (hyperbolic): edges pinch inward and the sum drops below $180°$. The live readout shows the Gauss–Bonnet law in action: $(\text{angle sum})-180° = K\times\text{Area}$ — curvature is literally the angle you gain (or lose) per unit area.

The three geometries side by side

Property	Euclidean (flat)	Spherical (elliptic)	Hyperbolic
Curvature $K$	$0$	$>0$ (e.g. $+1/R^2$)	$<0$
Parallels through a point	exactly 1	0 (all geodesics meet)	infinitely many
Triangle angle sum	$=180°$	$>180°$ (bulges out)	$<180°$ (pinched in)
"Straight line" (geodesic)	ordinary line	great circle	boundary-⟂ arc (Poincaré)
Circumference of radius-$r$ circle	$2\pi r$	$2\pi\sin r\ (<2\pi r)$	$2\pi\sinh r\ (>2\pi r)$
Disk area grows like	$r^2$ (polynomial)	bounded (closes up)	$e^{r}$ (exponential)
Flatten to paper?	yes (already flat)	no (must tear — maps lie)	no (too much room)
Natural data fit	grids, tabular	directions, similarity	trees, hierarchies
AI appearance	plain SGD, $\ell_2$ embeddings	CLIP, contrastive (§12.2)	Poincaré/Lorentz emb. (§12.1)

The exponential area growth row is the single fact that powers all of hyperbolic AI (§12): a tree's nodes grow like $e^{r}$, so only a space whose room grows like $e^{r}$ can hold it without distortion.

2.2 What is a "straight line" on a curved surface?

The whole trick is to redefine "line" as a geodesic — and the two phrases that sound contradictory are actually the same thing. A geodesic is a path that is straight from the traveler's point of view: walk it without ever turning your steering wheel left or right, and you trace a geodesic. On a flat plane that's an ordinary straight line; on a sphere it's a great circle (the equator, any meridian) — which is why long-haul flights arc toward the poles. Such a no-turning path is automatically the shortest route between nearby points; "locally shortest" means shortest compared to small wiggles around it, even if some longer way around also exists. Two distinct great circles always intersect (at two antipodal points), so there are no parallels at all — and that single fact forces triangles to be fat.

Why "straightest" = "locally shortest." The green arc bends as little as possible given that it must stay on the sphere; any small wiggle off it adds length.

A triangle with three right angles. Start at the North Pole. Walk straight down to the equator (a meridian). Turn 90°, walk a quarter of the way around the equator. Turn 90° again, walk straight back up to the pole. You've traced a triangle with three 90° corners — angle sum $270°$, a full $90°$ over Euclid's $180°$. That excess is not an error; it is the curvature, integrated over the triangle.

This is the seed of the Gauss–Bonnet theorem: $(\text{angle sum}) - 180° = \iint_{\triangle} K \, dA$ (working in radians, $180°=\pi$). Check it: our triple-right-angle triangle has excess $270°-180°=90°=\tfrac\pi2$, and on a unit sphere it covers an octant of area $\tfrac\pi2$, with $K=1$ — so $\iint K\,dA = 1\cdot\tfrac\pi2 = \tfrac\pi2$. Both sides equal. Curvature is literally the angle you gain by going around.

2.3 The Poincaré disk — hyperbolic space you can hold

Hyperbolic space is harder to picture because it has "too much room" to fit in flat paper. The Poincaré disk model is the standard cheat: squash the entire infinite hyperbolic plane into the inside of a unit disk. The price is that the ruler shrinks as you approach the boundary — near the edge, a step that looks tiny is actually enormous. The boundary circle is "infinitely far away."

Left: straight lines (geodesics) in the Poincaré disk are circular arcs that hit the boundary at a right angle; the green diameter is also a geodesic. Through one point you can draw infinitely many arcs that never meet a given arc — that's $\infty$ parallels. Right: the metric $ds = \tfrac{2\,\|d\vx\|}{1-\|\vx\|^2}$ blows up near $\|\vx\|=1$, so distances explode at the rim.

The first real metric, in actual numbers. Decode the symbols: $\vx$ is your current position (distance $\|\vx\|$ from center, between $0$ and $1$); $d\vx$ is a tiny step; $\|d\vx\|$ is its ordinary flat length; $ds$ is its true hyperbolic length. The metric just multiplies flat length by a factor $\dfrac{2}{1-\|\vx\|^2}$ that depends only on how far out you are:

position $\\|\vx\\|$	factor $\frac{2}{1-\\|\vx\\|^2}$	a flat step of 0.01 becomes…
0 (center)	2.0	0.02 — mild
0.9	10.5	0.105 — ~5× longer
0.99	100.5	~1.0 — ~50× longer
→ 1 (rim)	→ ∞	→ ∞ — the boundary is infinitely far

So the rim only looks close: to actually reach it you'd integrate a blowing-up factor, i.e. travel infinitely far. This exploding-room property is the entire reason hyperbolic space is the natural home for trees and hierarchies in AI (§12).

drag the two white points ·

Play with hyperbolic distance yourself. Drag points A and B. The green geodesic is the circular arc that meets the boundary at right angles — the hyperbolic "straight line." Watch the two readouts: the screen (Euclidean) distance barely changes as you slide a point toward the rim, but the true hyperbolic distance $d=\operatorname{arccosh}\!\big(1+\tfrac{2\|a-b\|^2}{(1-\|a\|^2)(1-\|b\|^2)}\big)$ explodes. The evenly-spaced tick dots along the geodesic are equal hyperbolic steps — they visibly bunch up near the edge, making "equal real length looks tiny near the boundary" something you can see, not just read.

Why does a tree "want" hyperbolic space? Count nodes: a binary tree has $2^d$ nodes at depth $d$ — the number of things grows exponentially with distance from the root. In flat space the area of a disk of radius $r$ grows like $r^2$ (polynomially), so you run out of room and neighbors get crushed together. In hyperbolic space the area of a disk grows like $e^{r}$ — exponentially, exactly matching the tree. There is "enough room" to place every node far from its cousins. Embed a hierarchy in flat space and distances lie; embed it in hyperbolic space and they're nearly exact in only 2 dimensions. Hold that thought for §12.

3 · Curvature & the Theorema Egregium

We keep saying "curvature." Let's make it a number. The cleanest entry point is Gaussian curvature $K$ of a 2D surface.

3.1 Two radii, one number

Stand at a point on a surface and look at how it bends in different directions. There's a most-curved direction and a least-curved direction (always at right angles). Each has a signed curvature $\kappa=1/R$ — and the sign is the whole point. Pick a side of the surface to call "up": a direction that curves toward up gets $\kappa>0$, one that curves away (down) gets $\kappa<0$. The two extremes $\kappa_1,\kappa_2$ are the principal curvatures. Gauss's definition:

$$K = \kappa_1 \,\kappa_2 = \frac{1}{R_1 R_2}.$$

Surface	$\kappa_1,\kappa_2$	$K$	Feels like
Plane / cylinder	cylinder: $\kappa_1=1/R$ around, $\kappa_2=0$ along	$K=\frac1R\cdot0=0$	flat (can unroll to paper)
Sphere, radius $R$	both bend same way	$K=+1/R^2 > 0$	dome — surface curves toward you everywhere
Saddle / Pringle	bend opposite ways	$K<0$	hyperbolic — up one way, down the other

The sign rule made visual. On a dome both slices bend the same way (both $+$) so $K=(+)(+)>0$. On a saddle one slice smiles up ($+$) and the perpendicular slice frowns down ($-$), so $K=(+)(-)<0$. That sign flip is exactly the difference between sphere-like and hyperbolic geometry.

A cylinder has $K=0$ even though it looks curved: you can roll a flat sheet of paper into a cylinder without stretching, so "intrinsically" it's still flat. A sphere has $K>0$ and you cannot flatten it without tearing — which is why every world map distorts something.

3.2 Theorema Egregium — the "remarkable theorem"

Gauss's Theorema Egregium (1827). Gaussian curvature $K$ is intrinsic: it can be computed entirely from measurements made within the surface (lengths and angles of tiny triangles), with no reference to the surrounding 3D space.

Imagine a 2D ant living on the surface, who can measure distances but can never look "up" out of its world. The Theorema Egregium says the ant can still determine $K$ — by checking whether the angles of a small triangle sum to more, less, or exactly $180°$, or whether the circumference of a small circle is less than, more than, or exactly $2\pi r$. Curvature is detectable from the inside. This is the conceptual jump that makes everything later possible: we will compute the curvature of the space of probability distributions, a space we obviously cannot step "outside" of. We only need the intrinsic ruler — the metric.

The ant's circle test — "intrinsic" as an actual measurement. A flat 2D ant can't look "up" out of its surface, but it can still detect curvature. Recipe: walk out a distance $r$ in all directions (measuring $r$ along the surface), then measure the circumference $C$ of the circle you've traced.

surface	circumference of radius-$r$ circle	at $r=1$	ant concludes
Plane	$C=2\pi r$	$6.283$	flat ✓
Cylinder (radius $R$)	$C=2\pi r$ (unrolls flat!)	$6.283$	flat — same as a plane!
Sphere (radius 1)	$C=2\pi\sin r < 2\pi r$	$2\pi\sin 1 = 5.29$	curved — a deficit!

The sphere-ant measures $C=5.29$ where it expected $6.283$ — a circumference deficit that no amount of "looking from inside" can explain away except curvature. The cylinder-ant measures exactly $2\pi r$ and correctly concludes "flat," even though to us standing outside, the cylinder looks bent. That gap between what the ant measures and what we see is the entire content of "intrinsic vs extrinsic."

Why a map of Earth always lies. A cylinder ($K=0$) flattens with scissors — no distances change. A sphere ($K=+1/R^2$) cannot be flattened without tearing or stretching, because by the Theorema Egregium that would change $K$ from $+$ to $0$, which is impossible. Every world map is forced to distort something. (And it's why a pizza slice stiffens when you fold it: pinching one curvature to nonzero forces the other to $0$ to keep $K=0$ — geometry you can eat.)

This is also why a flat map of Earth always lies: flattening changes $K$ from $+1/R^2$ to $0$, which by the theorem is impossible without distorting distances. The same theorem is why pizza folds: pinching a slice into a U forces $\kappa_1 \ne 0$ in one direction, and since $K$ must stay $0$ (paper-like dough), the other curvature is forced to $0$ — the slice stiffens and won't flop. You have used the Theorema Egregium to eat lunch.

We now have the two ideas we need to generalize: (a) geometry can be curved, and (b) curvature is intrinsic, derivable from a local ruler. Riemann's framework makes "local ruler" precise and works in any dimension.

4 · Riemannian geometry: the master framework

Riemann's 1854 lecture answered a radical question: instead of studying surfaces sitting inside 3D space, can we define a curved space abstractly, by only specifying how to measure tiny distances at each point? Yes — and that recipe is all of Riemannian geometry. We build it in four layers.

The build order. You only ever choose ① the manifold and ③ the metric. The tangent spaces ② come automatically, and the connection ④ — geodesics, parallel transport, and curvature — is uniquely determined by the metric (the Fundamental Theorem of Riemannian Geometry). So in practice, choosing a geometry = choosing a metric.

4.1 Layer ①: manifolds — spaces that are locally flat

A manifold $M$ of dimension $n$ is a space that, in a small neighborhood of every point, looks like ordinary flat $\RR^n$. You can lay down local coordinates (a "chart") just like latitude/longitude on Earth — but no single chart need cover everything, and the global shape can be curved or topologically nontrivial.

Earth's surface is a 2-manifold: locally it's a flat map, globally it's a sphere. The key word is locally: zoom in far enough and curvature becomes invisible, which is exactly why people once thought the Earth was flat. This "locally flat" property is what lets us do calculus on curved spaces.

Think of a manifold as a patchwork quilt of flat coordinate-charts, stitched together smoothly. Calculus happens chart-by-chart; geometry is about how the charts are stitched. For AI: the "data manifold" hypothesis says natural images / sentences / proteins live on a low-dimensional manifold curled up inside a huge ambient space (pixels, tokens). Generative models try to learn a chart for that manifold.

4.2 Layer ②: tangent spaces — the local linear world

At a point $p\in M$, the tangent space $T_pM$ is the set of all velocity vectors of curves passing through $p$. It is a flat vector space (a copy of $\RR^n$) glued onto the manifold at $p$ — the best linear approximation of $M$ near $p$, like the tangent plane resting on a globe.

Tangent space. Every point gets its own little linear space of directions. Crucially, vectors at different points live in different tangent spaces — you cannot directly subtract a vector at the north pole from one at the equator. Comparing them needs parallel transport (Layer ④). This "can't subtract vectors at different points" fact is the entire reason curvature is subtle, and it reappears in AI as the reason you can't just average gradients across a curved parameter space naively.

Two planes, two worlds. The whole subtlety of curved space in one picture: the green arrows live in different tilted planes, so they cannot be directly added or subtracted. This is exactly why, in AI, you cannot naively average or subtract gradients/parameters that live at different points of a curved model space.

5 · Layer ③: the metric tensor — the star of the show

This is the single most important object in the whole document. Everything else is built from it.

A Riemannian metric $g$ assigns to each point $p$ a smoothly-varying inner product $g_p(\cdot,\cdot)$ on the tangent space $T_pM$. In coordinates it's a symmetric positive-definite matrix $\Met(p)=[g_{ij}(p)]$, and the squared length of an infinitesimal step $d\vx$ is $$ ds^2 \;=\; \sum_{i,j} g_{ij}(\vx)\, dx^i\, dx^j \;=\; d\vx^\top \Met(\vx)\, d\vx. $$

Read that as a position-dependent Pythagorean theorem. In flat space $\Met=I$ (the identity) and you recover $ds^2 = dx^2+dy^2$. Make $\Met$ vary from point to point and you have bent the space — without ever leaving the coordinates.

What each entry $g_{ij}$ actually means. Let $e_1,e_2,\dots$ be the coordinate directions at a point. Then $g_{ij}=g_p(e_i,e_j)$ is literally the inner product of direction $i$ with direction $j$. So:

the diagonal $g_{ii}=\|e_i\|^2$ tells you how long one step along axis $i$ really is;
the off-diagonal $g_{ij}$ tells you how non-perpendicular axes $i$ and $j$ are (zero ⇒ perpendicular).

A general inner product is then just the bookkeeping $g_p(\vv,\vw)=\sum_{ij}g_{ij}v^iw^j=\vv^\top\Met\,\vw$ — that's why an inner product is a matrix. The condition positive-definite (every nonzero step has $\vv^\top\Met\vv>0$, i.e. all eigenvalues positive) just guarantees the ruler never reports zero or negative length.

$ds^2$ with actual numbers (including a nonzero off-diagonal). Take $$\Met=\begin{pmatrix}4&1\\1&9\end{pmatrix},\qquad d\vx=\begin{pmatrix}0.1\\0.2\end{pmatrix}.$$ The double sum $\sum_{i,j}g_{ij}\,dx^i dx^j$ expands to four terms; because $\Met$ is symmetric ($g_{12}=g_{21}=1$) the two cross terms merge into $2g_{12}\,dx^1dx^2$: $$ds^2 = \underbrace{4(0.1)^2}_{0.04}+\underbrace{2(1)(0.1)(0.2)}_{0.04}+\underbrace{9(0.2)^2}_{0.36}=0.44,\qquad ds=\sqrt{0.44}\approx0.663.$$ Compare flat space ($\Met=I$): $ds^2=0.1^2+0.2^2=0.05$, $ds\approx0.224$ — the same coordinate step is 3× longer under this metric. The off-diagonal $g_{12}=1$ is what makes the axes not perpendicular: moving in $x$ now partly counts as moving in $y$. Set $g_{12}=0$ and the cross term vanishes — clean Pythagoras with independently-stretched axes.

5.1 The metric does three jobs

Quantity	Formula from the metric	Meaning
Length of a vector $\vv$	$\\|\vv\\|_g=\sqrt{\vv^\top \Met\, \vv}$	how "big" a step is — and $\Met$ can make the same coordinate-step count as huge or tiny
Angle between $\vv,\vw$	$\cos\theta=\dfrac{\vv^\top \Met\,\vw}{\\|\vv\\|_g\\|\vw\\|_g}$	the protractor
Length of a curve $\gamma$	$L=\displaystyle\int_0^1\sqrt{\dot\gamma^\top \Met(\gamma)\,\dot\gamma}\,dt$	add up infinitesimal lengths along the path

The Poincaré disk metric, concretely. Its metric matrix is a position-dependent rescaling of the identity: $$\Met(\vx)=\left(\frac{2}{1-\|\vx\|^2}\right)^{2} I .$$ At the center $\vx=0$: the factor is $(2/1)^2=4$. At radius $\|\vx\|=0.9$: the factor is $\big(2/(1-0.81)\big)^2=\big(2/0.19\big)^2\approx 111$. So a coordinate-step near the rim costs about $\sqrt{111/4}\approx 5\times$ more "real" length than the same step at the center — and it keeps exploding toward $\|\vx\|\to 1$. That single scalar field is the entire reason hyperbolic space has exponentially expanding room. Choosing this metric = choosing hyperbolic geometry.

Why "tensor"? Because the same geometric length must come out no matter what coordinates you use (Cartesian, polar, whatever). When you change coordinates, the components $g_{ij}$ transform in exactly the compensating way to keep $ds^2$ invariant. "Tensor" is just the bookkeeping rule that makes a quantity coordinate-independent. This invariance is precisely what AI people mean when they praise the natural gradient as "reparameterization-invariant" — it's a metric-based (tensorial) update, so it doesn't care whether you parameterized your softmax one way or another.

Heads-up for ML readers: a geometry tensor is not the same as a torch.Tensor. The array of numbers looks the same, but the geometry word adds a contract: the components must transform under coordinate changes so that physical quantities (lengths, angles) stay invariant. A torch.Tensor is just storage; a geometric tensor is storage + that transformation rule.

A metric is not a distance function directly. The metric measures infinitesimal lengths ($ds$); to get the distance between two far-apart points you must find the shortest curve connecting them and integrate $ds$ along it. That shortest curve is the geodesic — next section. Confusing the local ruler with the global distance is the #1 beginner mistake.

6 · Geodesics, connections & curvature

6.1 Geodesics — straightest/shortest paths

A geodesic is a locally length-minimizing curve: the generalization of a straight line. Equivalently, it's the path a free particle follows with "no turning" — zero acceleration as measured intrinsically. In coordinates it solves the geodesic equation $$ \ddot\gamma^{\,k} + \sum_{i,j}\Gamma^{k}_{ij}\,\dot\gamma^{\,i}\dot\gamma^{\,j}=0, $$ where $\Gamma^{k}_{ij}$ are the Christoffel symbols built from derivatives of the metric.

Reading the symbols (the notation is scarier than the idea): a geodesic is a path $\gamma(t)$. Superscripts here are component labels, not powers: $\gamma^k$ is the $k$-th coordinate of the path, $\dot\gamma^i=d\gamma^i/dt$ is the $i$-th velocity component, $\ddot\gamma^k$ the $k$-th acceleration. The $\Gamma^k_{ij}$ are a table of numbers (one per triple $i,j,k$) saying how much "apparent acceleration" the bent coordinates inject.

The term $\Gamma^k_{ij}\dot\gamma^i\dot\gamma^j$ is the "fictitious force" the curvature exerts. Set $\Gamma=0$ (flat space) and the equation becomes $\ddot\gamma=0$, i.e. $\gamma(t)=a+bt$ — an ordinary straight line. So geodesics are "straight lines bent by the metric." On a globe they're great circles; in the Poincaré disk they're those boundary-perpendicular arcs.

6.2 Christoffel symbols — how the ruler twists

The Christoffel symbols are computed purely from the metric and its first derivatives:

$$\Gamma^{k}_{ij}=\tfrac12\sum_{l} g^{kl}\Big(\partial_i g_{jl}+\partial_j g_{il}-\partial_l g_{ij}\Big),$$

Anatomy of the scary formula (it's smaller than it looks):

$\Gamma^{k}_{ij}$ — the fictitious-force table: apparent acceleration injected in direction $k$ when you move along $i$ and $j$. One number per $(i,j,k)$ triple.

$g^{kl}$ — the inverse metric (raised indices = matrix inverse). It "divides by the ruler," converting raw ruler-change back into acceleration.

$\partial_i g_{jl}$ — how fast the metric entry $g_{jl}$ changes as you step in direction $i$. This is the whole engine: if the ruler never changes, every $\partial g=0$, so $\Gamma=0$, so space is flat.

The specific three-term combination just makes $\Gamma$ symmetric in $i,j$ and metric-compatible — you needn't feel it, only see that it's built only from first derivatives of $g$.

In words: $\Gamma$ = (1 / ruler) × (rate the ruler changes). Constant ruler ⇒ $\Gamma=0$ ⇒ straight lines stay straight.

where $g^{kl}$ is the inverse metric matrix. Don't memorize it — just absorb the message: everything about how to move "straight" is encoded in how the metric changes from point to point. If $\Met$ is constant, all derivatives vanish, $\Gamma=0$, and the space is flat. Curvature is the failure of the metric to be constant.

Worked example you already half-know: the flat plane in polar coordinates. Use $(r,\theta)$ instead of $(x,y)$. The metric is $$\Met=\begin{pmatrix}1&0\\0&r^2\end{pmatrix}\quad\text{i.e.}\quad ds^2=dr^2+r^2\,d\theta^2.$$ It is not constant (it depends on $r$), so the coordinates twist even though the plane is perfectly flat. Plugging into the Christoffel formula, only two symbols survive: $$\Gamma^r_{\theta\theta}=-r,\qquad \Gamma^\theta_{r\theta}=\Gamma^\theta_{\theta r}=\tfrac1r,$$ and the geodesic equation becomes $$\ddot r - r\dot\theta^2 = 0,\qquad \ddot\theta+\tfrac2r\dot r\dot\theta=0.$$ You have seen $\ddot r = r\dot\theta^2$ before — it's exactly the centrifugal term from freshman mechanics. So Christoffel symbols are nothing scarier than the "fictitious forces" that appear when you use curved coordinates, and here the space is still flat (a straight line just looks curved in polar coords). A genuinely curved space — the sphere — adds a $\sin\theta\cos\theta$ term that no change of coordinates can remove. That irremovable piece is real curvature.

Running the machinery on a truly curved space: great circles ARE geodesics. We asserted this in §2.2; here's the one place it's earned. The unit sphere has metric $ds^2=d\theta^2+\sin^2\theta\,d\varphi^2$ ($\theta$ = colatitude, $\varphi$ = longitude). Its surviving Christoffel symbols are $\Gamma^\theta_{\varphi\varphi}=-\sin\theta\cos\theta$ and $\Gamma^\varphi_{\theta\varphi}=\cot\theta$. Test the equator $\theta=\tfrac\pi2$ (constant), traversed at steady $\dot\varphi$: the $\theta$-equation is $$\ddot\theta-\sin\theta\cos\theta\,\dot\varphi^2 = 0-\underbrace{\sin\tfrac\pi2\cos\tfrac\pi2}_{=\,0}\dot\varphi^2=0.\ \checkmark$$ So the equator satisfies the geodesic equation — it's a geodesic. Every other great circle is just a rotated equator, and rotations are isometries (they preserve the metric), so all great circles are geodesics. There's also a 3-line non-calculus proof: any two points lie in a plane through the sphere's center; reflection across that plane is a symmetry fixing both points and fixing exactly the great-circle arc, so the shortest path (unique locally) must be the part it fixes — the great circle.

Parallel transport & holonomy. The connection (encoded by $\Gamma$) defines how to move a tangent vector "without turning." Transport a vector around a closed loop on a curved space and it returns rotated — the deficit angle measures curvature. The Riemann curvature tensor $R^{l}_{ijk}$ is exactly this "rotation per unit area of loop," and it is built from $\Gamma$ and its derivatives (hence from second derivatives of $g$).

loop size (longitude span):

Watch holonomy accumulate. Press play: an orange arrow is carried around a spherical triangle (pole → equator → along the equator → back), always kept "as parallel as possible." A faint ghost stays pinned at the start orientation. When the arrow returns it is visibly rotated — and the readout shows the rotation equals the enclosed area exactly: holonomy $=\Delta\lambda = K\times\text{Area}$ on the unit sphere. Drag the slider to enclose more area and the deficit grows. This rotation-per-loop is the operational definition of the Riemann curvature tensor.

6.3 The curvature ladder & the exp / log maps

The full Riemann tensor $R^l_{ijk}$ is a beast, but its four indices have a concrete operational reading: $i,j$ pick the two directions spanning the little loop you transport around; $k$ is the direction of the arrow you carry; $l$ is the component of the (rotated) arrow that comes back. So $R^l_{ijk}$ = "how much a $k$-pointing arrow tips into the $l$-direction when carried around an $i$–$j$ loop." Four indices = (loop plane: 2) + (input direction: 1) + (output component: 1). It gets contracted (summed down) into friendlier averages:

Object	What it is	One-line meaning
Riemann tensor $R^l_{ijk}$	full curvature, all directions	holonomy per unit loop area
Sectional curvature $K(\vv,\vw)$	curvature of one 2D slice	the Gaussian $K$ of that slice
Ricci curvature $\mathrm{Ric}_{ij}$	average over directions	does a small ball have more/less volume than Euclidean? (drives general relativity & "Ricci flow")
Scalar curvature $S$	one number per point	total local curvature

Finally, two maps let us hop between the curved manifold and its flat tangent spaces — these are the workhorses of every geometric deep-learning library:

Exponential map $\exp_p(\vv)$: start at $p$, shoot off along the geodesic in direction $\vv\in T_pM$ for unit time, and report where you land on $M$. It turns a flat tangent vector into a point on the curved manifold.

Logarithm map $\log_p(q)$: the inverse — the tangent vector at $p$ that points toward $q$ with length equal to the geodesic distance. It turns a manifold point into a flat tangent vector.

$\exp$ and $\log$ are how practitioners "fake" doing linear algebra on a curved space: $\log$ to flatten into a tangent space, do your Euclidean operation (add, average, take a gradient step), then $\exp$ back onto the manifold. Hyperbolic neural networks (§12) literally wrap each layer in $\log/\exp$; Riemannian optimizers replace "$\theta \leftarrow \theta - \eta\,\vg$" with "$\theta\leftarrow \exp_\theta(-\eta\,\vg)$". Master these two maps and most "geometric deep learning" code stops looking mysterious.

The round-trip, as code you could run: averaging two points on a curved manifold. You can't average two surface points by $\tfrac12(A+B)$ — that midpoint floats off the surface (see picture). Instead:

# Riemannian mean of points A, B, around a base point p
a     = log(p, A)        # log_p : manifold point -> flat tangent vector in T_pM
b     = log(p, B)        # both A and B are now ordinary vectors in the SAME flat space
m_tan = 0.5*(a + b)      # ordinary Euclidean average — totally legal here
M     = exp(p, m_tan)    # exp_p : tangent vector -> back onto the manifold

Types: $\log_p:\,M\to T_pM$ (point → flat vector), $\exp_p:\,T_pM\to M$ (flat vector → point), and they're inverses: $\exp_p(\log_p(q))=q$. So in $\theta\leftarrow\exp_\theta(-\eta\vg)$, the object $-\eta\vg$ is a tangent vector and $\exp_\theta$ of it is a new point on the manifold.

Why the machinery exists. The naive Euclidean midpoint of two manifold points generally isn't on the manifold; the $\log$→average→$\exp$ recipe keeps every result where it belongs.

We now have the complete machinery of curved spaces: manifold, tangent space, metric, geodesics, curvature, exp/log. The magic of the next two sections is that the space of probability distributions is secretly one of these manifolds — and its metric has a name you already half-know: Fisher information.

7 · Fisher information: probability gets a ruler

Switch worlds. Forget surfaces for a moment. Consider a statistical model: a family of probability distributions $p(x\mid\vth)$ indexed by parameters $\vth$. Examples: all Gaussians (parameter = mean and variance), all coins (parameter = bias $\theta$), or — the AI case — all next-token distributions a neural net can produce as you vary its weights $\vth$.

The question Fisher asked in 1925: how much does the data tell us about the parameter? The answer is a matrix, and that matrix will turn out to be a metric tensor.

7.1 The score: how sensitive is the likelihood?

The score is the gradient of the log-likelihood with respect to the parameter: $$ s(\vth)=\grad_{\vth}\log p(x\mid\vth). $$ We use the log for the usual ML reason: it turns the product over independent data points into a sum, so the score of a whole dataset is just the sum of per-example scores (and its spread will become the Fisher matrix). It points in the direction in parameter space that would most increase the probability of the data you observed. A key fact: averaged over the data the model itself generates, the score is zero, $\EE_{x\sim p_\vth}[s(\vth)]=0$.

Score of a coin, concretely (and why "mean zero" isn't a slogan). With $\log p(x\mid\theta)=x\log\theta+(1-x)\log(1-\theta)$, differentiate in $\theta$: $$ s(\theta)=\frac{x}{\theta}-\frac{1-x}{1-\theta}. $$

Heads ($x=1$): $s=+1/\theta$ — positive, "the data wants a higher $\theta$."
Tails ($x=0$): $s=-1/(1-\theta)$ — negative, "the data wants a lower $\theta$."

Now average over data the coin actually produces (heads w.p. $\theta$, tails w.p. $1-\theta$): $$\EE[s]=\theta\cdot\frac{1}{\theta}+(1-\theta)\cdot\frac{-1}{1-\theta}=1-1=0.$$ That's mean-zero as arithmetic, not a slogan: heads pull $\theta$ up and tails pull it down, and if the coin really has bias $\theta$ those pulls exactly cancel. They fail to cancel only when the data come from a different $\theta$ — which is precisely how learning gets a signal.

7.2 Fisher information = variance of the score

The Fisher information matrix is the covariance of the score (equivalently, minus the expected curvature of the log-likelihood): $$ \Fish(\vth)\;=\;\EE_{x\sim p_\vth}\!\big[\,s(\vth)\,s(\vth)^\top\big]\;=\;-\,\EE_{x\sim p_\vth}\!\big[\grad^2_\vth \log p(x\mid\vth)\big]. $$ The two forms are equal under mild regularity conditions ("Fisher's identity").

Why the two forms agree (one line, reusing "score has mean zero"). Start from $\EE[s]=\int p\cdot\frac{\grad p}{p}=\grad\!\int p=\grad 1=0$. Differentiate that identity once more in $\theta$; the product rule splits it into exactly $$\EE[\grad^2\log p]+\EE[(\grad\log p)(\grad\log p)^\top]=0,\quad\text{i.e.}\quad -\EE[\text{Hessian}]=\EE[ss^\top].$$ So "variance of score = negative expected curvature" is just "differentiate mean score = 0 one more time." A variance (first derivatives, always $\ge0$) and a curvature (second derivatives) turn out to be the same matrix.

Read both forms intuitively:

As score variance (first form): if the score jumps around a lot as the data varies, then the data is very informative about $\vth$ — small changes in $\vth$ visibly change which data is likely. High variance of the score ⇒ high information.
As negative curvature of log-likelihood (second form): a sharply-peaked log-likelihood (large second derivative) means the data pins down $\vth$ tightly. Flat likelihood ⇒ the data barely constrains $\vth$ ⇒ low information.

Fisher information of a coin. A coin lands heads with probability $\theta$. Observe one flip $x\in\{0,1\}$. Then $\log p = x\log\theta+(1-x)\log(1-\theta)$, and a short calculation gives the (scalar) Fisher information $$ \Fish(\theta)=\frac{1}{\theta(1-\theta)}. $$ At $\theta=0.5$ (fair coin): $\Fish = 1/0.25 = 4$ — minimum information; a 50/50 coin's flips are maximally ambiguous about small changes in $\theta$. At $\theta=0.99$: $\Fish=1/(0.99\cdot0.01)\approx 101$ — huge information; a near-certain coin reveals its bias fast, because seeing even one tail is very surprising and strongly updates $\theta$. The parameter space of coins is "stretched" near the edges 0 and 1 and "compressed" in the middle — it is curved, and $\Fish(\theta)$ is its metric.

7.3 The Cramér–Rao bound — the speed limit of estimation

For any unbiased estimator $\hat\vth$ of $\vth$ from $n$ i.i.d. samples, $$ \mathrm{Cov}(\hat\vth)\;\underbrace{\succeq}_{\text{matrix }\ge}\;\frac{1}{n}\,\Fish(\vth)^{-1}. $$ The symbol $\succeq$ is the matrix version of "$\ge$" (the difference is positive-semidefinite). In words: you cannot estimate a parameter more precisely than the inverse Fisher information allows — high Fisher ⇒ low floor ⇒ precisely learnable.

This is why Fisher information deserves the name "information": it is literally the currency of statistical precision. More Fisher information ⇒ smaller achievable variance ⇒ a more learnable parameter. The maximum-likelihood estimator achieves this bound asymptotically, which is the deep reason MLE (and thus most of deep learning's loss functions) is a sensible thing to do.

Geometrically, the Cramér–Rao bound says: directions in which distributions are easy to tell apart (high Fisher) are directions you can estimate precisely; directions in which distributions look nearly identical (low Fisher) are directions you're doomed to be fuzzy about. The Fisher matrix is the exchange rate between "parameter distance" and "statistical distinguishability." That exchange rate is exactly what a metric tensor is. Hold that thought — §8 makes it literal.

8 · Information geometry: the grand synthesis

Here is the punchline the whole document has been building toward.

The central theorem (in one breath)

The set of all distributions $\{p(x\mid\vth)\}$ in a model is a Riemannian manifold — points are distributions, coordinates are parameters $\vth$ — and its metric tensor is the Fisher information: $$ \boxed{\;g_{ij}(\vth)=\Fish_{ij}(\vth)\;}\qquad\text{the \emph{Fisher–Rao metric}.} $$ Distance on this manifold = statistical distinguishability. Straight lines, curvature, and the Pythagorean theorem all reappear — wearing the costumes of KL divergence, exponential families, and natural gradient.

Why is this the same kind of object as a surface metric? Put the two facts side by side:

$$\underbrace{ds^2=\delta^\top \Met\,\delta}_{\text{length on a curved surface (§5)}}\qquad\Longleftrightarrow\qquad \underbrace{2\,\KL\approx\delta^\top \Fish\,\delta}_{\text{distinguishability of nearby models (§8.1)}}$$

A metric is anything that turns a tiny coordinate step into a squared length. On a surface that length is measured in meters; on a statistical manifold it's measured in "how distinguishable," via KL. Same machine, different ruler — so calling $\Fish$ the metric isn't an analogy, it literally satisfies the definition of $g_{ij}$.

The bridge — the single most important picture in this document. On the left, a curved surface where the metric $g$ makes the orange step "long" and the blue step "short." On the right, the same abstract patch, but now each point is literally a probability distribution (a bell curve). The orange neighbor's distribution is very different from the center (easy to distinguish ⇒ "far" ⇒ high Fisher); the blue neighbor's is almost identical (hard to distinguish ⇒ "near" ⇒ low Fisher). Same step $\delta$, same matrix doing the rescaling — that's why $g_{ij}=\Fish_{ij}$. "The space of distributions is a curved manifold" stops being a slogan once you see the points are distributions.

8.1 Why is Fisher the right metric? KL divergence is secretly quadratic

KL divergence — definition first, since everything below (and all of §9–§10) uses it. Given two distributions $p$ and $q$ over the same outcomes $x$, $$\KL(p\,\|\,q)=\EE_{x\sim p}\!\left[\log\frac{p(x)}{q(x)}\right]=\sum_x p(x)\log\frac{p(x)}{q(x)}.$$ In plain English: the average log-likelihood-ratio — how surprised you'd be, on average, if you believed $q$ but data actually came from $p$ (equivalently, the extra bits wasted compressing $p$'s data with a code built for $q$). Three facts to hold onto:

$\KL(p\|q)\ge0$ always, and $=0$ iff $p=q$ — so it behaves like a "distance-squared" from $p$ to $q$.
It is not symmetric: $\KL(p\|q)\ne\KL(q\|p)$ in general. That's why it's a divergence, not a true distance.
Small when $p,q$ are hard to tell apart; large when one puts probability where the other puts almost none.

Coin check: $p=(\text{heads }0.5)$, $q=(\text{heads }0.9)$. Then $\KL(p\|q)=0.5\log\frac{0.5}{0.9}+0.5\log\frac{0.5}{0.1}\approx0.51$; swapped, $\KL(q\|p)\approx0.37$ — different numbers, confirming asymmetry. (Units: $\log_2\!\to$ bits, $\ln\!\to$ nats.)

The Kullback–Leibler divergence $\KL(p_\vth\,\|\,p_{\vth+\delta})$ measures how distinguishable two nearby distributions are. Taylor-expand it for a tiny step $\delta$. The constant term is 0 (a distribution is identical to itself), the linear term is 0 (KL is minimized at $\delta=0$, so its gradient vanishes there), and the first surviving term is quadratic:

$$ \KL\big(p_\vth \,\big\|\, p_{\vth+\delta}\big)\;=\;\tfrac12\,\delta^\top\,\Fish(\vth)\,\delta\;+\;O(\|\delta\|^3). $$

Watch the Fisher "pop out" — KL between two nearby coins. Coin $\theta$ vs coin $\theta+\delta$: $$\KL=\theta\log\frac{\theta}{\theta+\delta}+(1-\theta)\log\frac{1-\theta}{1-(\theta+\delta)}.$$ Taylor-expand the logs in $\delta$. The $\delta^0$ terms cancel. The $\delta^1$ terms cancel too — they sum to $\big(\tfrac{-\theta}{\theta}+\tfrac{1-\theta}{1-\theta}\big)\delta=0$, which is exactly the score having mean zero (§7.1), now showing up as the linear term dying. The $\delta^2$ terms survive: $$\KL\approx\tfrac12\Big(\tfrac1\theta+\tfrac{1}{1-\theta}\Big)\delta^2=\tfrac12\cdot\frac{1}{\theta(1-\theta)}\,\delta^2.$$ That coefficient $\frac{1}{\theta(1-\theta)}$ is exactly the coin's Fisher information from §7.2 — the same number, reached two completely different ways. Numerically at $\theta=0.5,\ \delta=0.01$: $\KL\approx\tfrac12\cdot4\cdot0.0001=2.0\times10^{-4}$, and the exact KL is $2.0\times10^{-4}$. They match. This single example ties §7.1 (score mean zero ⇒ linear term dies), §7.2 (Fisher of a coin), and §8.1 (quadratic coefficient) into one chain of arithmetic.

Now the matrix version — where the Fisher matrix $\Fish=\diag(1/\sigma^2,\,2/\sigma^2)$ actually comes from. The coin was scalar; here two parameters $(\mu,\sigma)$ show the off-diagonal die too. The exact KL between two Gaussians has a closed form (with $\sigma'=\sigma+\delta\sigma$): $$\KL\big(\mathcal N(\mu,\sigma^2)\,\big\|\,\mathcal N(\mu+\delta\mu,\sigma'^2)\big)=\log\frac{\sigma'}{\sigma}+\frac{\sigma^2+\delta\mu^2}{2\sigma'^2}-\tfrac12.$$ Expand to 2nd order in $(\delta\mu,\delta\sigma)$:

$\delta\mu$ enters only as $\dfrac{\delta\mu^2}{2\sigma^2}$ → coefficient $1/\sigma^2 = \Fish_{\mu\mu}$.
The $\delta\sigma^0$ and $\delta\sigma^1$ terms cancel (KL is minimized at $\delta\sigma=0$), leaving $\dfrac{\delta\sigma^2}{\sigma^2}$ → $\Fish_{\sigma\sigma}=2/\sigma^2$.
The cross term $\delta\mu\,\delta\sigma$ vanishes → $\Fish_{\mu\sigma}=0$.

$$\Rightarrow\quad \KL\approx\tfrac12\Big(\tfrac{\delta\mu^2}{\sigma^2}+\tfrac{2\,\delta\sigma^2}{\sigma^2}\Big)=\tfrac12\,\delta^\top\!\diag\!\big(\tfrac1{\sigma^2},\tfrac2{\sigma^2}\big)\delta.$$ That is exactly the matrix the §8.2 ellipse demo plots — and the off-diagonal-zero is why that ellipse stays axis-aligned. Numeric check: $\sigma=1,\delta\mu=0.1,\delta\sigma=0$ gives $\tfrac12(0.01)=0.005$, matching the exact KL $\approx0.005$.

coin bias θ:

Where the quadratic approximation holds — and where it breaks. The red curve is the exact $\KL(\theta\,\|\,\theta+\delta)$ as you vary the step $\delta$ (x-axis); the blue dashed parabola is the Fisher approximation $\tfrac12\frac{1}{\theta(1-\theta)}\delta^2$. They kiss at $\delta=0$ (Fisher = local curvature of KL) and peel apart as $|\delta|$ grows — that gap is the $O(\delta^3)$ truncation. Two things to notice by dragging $\theta$: near $\theta\to0$ or $1$ the parabola gets steep (high Fisher = the coin is easy to pin down), and the exact red KL is visibly lopsided (asymmetric) while the blue metric is always symmetric — the precise sense in which KL is a "divergence," not a true distance.

So the Fisher information is the second-derivative (Hessian) of the KL divergence at zero separation. KL behaves like one-half a squared distance, and the matrix defining that squared distance is exactly $\Fish$. This is the rigorous version of "Fisher = local ruler on the space of distributions," and it's the equation that powers TRPO, natural gradient, EWC, and the KL penalty in RLHF — all in §10–§11.

Chentsov's uniqueness theorem seals the deal: the Fisher–Rao metric is, up to overall scale, the only Riemannian metric on a statistical manifold that is invariant under "sufficient statistics" — any reprocessing of the data that loses no information. In other words, it is the unique geometry that doesn't care how you encode or reparameterize your data and model. That invariance is precisely the property AI researchers cherish when they say natural gradient is "reparameterization-invariant." The geometry is forced on us by nature, not chosen.

8.2 Visualizing the Fisher metric: the distinguishability ellipse

family mean μ: std σ:

The Fisher metric of the Gaussian family $\mathcal N(\mu,\sigma^2)$, parameterized by $(\mu,\sigma)$. Its Fisher matrix is $\Fish=\diag(1/\sigma^2,\,2/\sigma^2)$. We draw the equal-KL ellipse $\{\delta:\delta^\top\Fish\delta=c\}$ — the set of parameter-steps that are equally distinguishable from the center model. Its $\mu$-semi-axis is $\sqrt{c}\,\sigma$, so the ellipse grows as $\sigma$ grows: when a Gaussian is already wide and blurry (large $\sigma$), you must move $\mu$ or $\sigma$ a lot to make a statistically noticeable change — at $\sigma=2$ you must move $\mu$ four times as far as at $\sigma=0.5$ for the same KL. When $\sigma$ is small (a sharp spike), a tiny nudge to $\mu$ is instantly detectable — the ellipse is tight, the space is "stretched." This anisotropic, position-dependent stretching is the whole point of natural gradient: ordinary gradient descent ignores it and takes mis-sized steps.

8.3 Dually-flat spaces, exponential families & the generalized Pythagorean theorem

This subsection is the most abstract in the document. On a first read you can skip straight to §9 and lose nothing operationally — come back when you're curious why KL behaves like a squared distance.

Information geometry has one more gift, and it's the structural reason KL "feels like" a squared distance. Amari showed that statistical manifolds carry not one but a dual pair of flat connections — the e-connection (exponential/natural coordinates) and the m-connection (mixture/expectation coordinates) — linked by a Legendre transform. A manifold flat under both is called dually flat, and exponential families (Gaussian, Bernoulli, categorical/softmax, Poisson…) are exactly the dually-flat ones.

On a dually-flat manifold the canonical divergence is the Bregman divergence of a convex potential, and for probability distributions that Bregman divergence is the KL divergence. A generalized Pythagorean theorem holds: $$ \KL(p\,\|\,r)=\KL(p\,\|\,q)+\KL(q\,\|\,r) $$ whenever the m-geodesic $p\!\to\!q$ meets the e-geodesic $q\!\to\!r$ at a "right angle" (in the Fisher metric).

The generalized Pythagorean theorem. The whole edifice of maximum-likelihood, the EM algorithm, variational inference, and information projection rests on this: minimizing KL onto an exponential family is an orthogonal projection in Fisher geometry, with a Pythagorean decomposition of the "error." When you fit a softmax, run EM, or do variational inference, you are doing geometry on a dually-flat manifold whether you know it or not.

Three ways to measure the distance between two distributions

The document defines all three in different sections; here's why you'd reach for each. They are three rulers on the same space of distributions.

	KL divergence $\KL(p\\|q)$	Wasserstein $W_2$ (earth-mover)	Fisher–Rao
Measures	extra bits to code $p$ with $q$'s code	cheapest mass transport (mass × distance)	infinitesimal distinguishability of nearby models
Symmetric?	No ($\KL(p\\|q)\ne\KL(q\\|p)$)	Yes (a true metric)	Yes (a true Riemannian distance)
Triangle inequality?	No	Yes	Yes
Finite if supports don't overlap?	No (→ ∞)	Yes (stays finite)	n/a (local)
Relation to the others	$\approx\tfrac12\delta^\top\Fish\delta$ for small $\delta$	geodesic distance under the Otto metric	the metric tensor = Hessian of KL at $\delta\!=\!0$
Scale	global (bits/nats)	global (ground-distance × mass)	local / infinitesimal
AI appearance	TRPO/PPO/GRPO, VAE ELBO, RLHF	WGAN, OT flow matching, diffusion	natural gradient, EWC, Laplace

KL is the local picture (§8), Fisher is its second derivative, Wasserstein is the transport picture (§13) — three views of "how far apart are two distributions?"

9 · Natural gradient: geometry meets optimization

Now the geometry pays off as an algorithm. This single idea is the trunk from which most of the AI applications branch.

9.1 The problem with the ordinary gradient

Gradient descent does $\vth \leftarrow \vth - \eta\,\grad L$. But $\grad L$ is the direction of steepest descent only if you measure distance with the plain Euclidean ruler $\|\delta\vth\|^2$. On a statistical manifold that ruler is wrong — the right notion of "how far did the model move" is the Fisher length $\delta\vth^\top \Fish\,\delta\vth$ (≈ KL change), not the parameter-space length. Using the wrong ruler means the gradient points in a distorted direction and the step sizes are inconsistent across coordinates.

The natural gradient (Amari, 1998) is the steepest-descent direction measured in the Fisher metric: $$ \tilde\grad L \;=\; \Fish(\vth)^{-1}\,\grad L,\qquad \vth \leftarrow \vth - \eta\,\Fish(\vth)^{-1}\grad L. $$ It is the gradient pre-conditioned by the inverse Fisher information.

Where does $\Fish^{-1}$ come from? Derive it in two lines. Don't take the formula on faith — it falls out of one optimization. "Steepest descent" really means: take the step $\delta$ that decreases the loss most, subject to a fixed budget of how far you move. Measure "how far" by KL (the natural distance on distributions), which is $\approx\tfrac12\delta^\top\Fish\delta$ (§8.1). So: $$\min_{\delta}\ \grad L^\top\delta \quad\text{subject to}\quad \tfrac12\delta^\top\Fish\delta\le\varepsilon.$$ Form the Lagrangian $\mathcal L=\grad L^\top\delta+\lambda\big(\tfrac12\delta^\top\Fish\delta-\varepsilon\big)$ and set $\partial\mathcal L/\partial\delta=0$: $$\grad L+\lambda\Fish\delta=0\ \Longrightarrow\ \delta=-\tfrac1\lambda\,\Fish^{-1}\grad L.$$ The optimal direction is $-\Fish^{-1}\grad L$ — the natural gradient — and $\lambda$ (the learning rate) just sizes the step to fill the KL budget. So "steepest descent in the Fisher metric" and "the biggest loss drop within a KL trust region" are literally the same optimization. Solving for the budget gives $\eta=\sqrt{2\varepsilon/(\grad L^\top\Fish^{-1}\grad L)}$ — which is exactly TRPO's step size (§10). The two "superpowers" below are now consequences, not claims:

Two superpowers fall out immediately:

Reparameterization invariance. Because $\Fish$ transforms as a metric tensor, the update $\Fish^{-1}\grad L$ produces the same change in the underlying distribution regardless of how you parameterized the model. Ordinary SGD does not — rescale your weights and SGD takes a different path; natural gradient takes the same one.
It's a trust region in KL. Taking a natural-gradient step of fixed size is equivalent to "improve the loss as much as possible while changing the output distribution by at most a fixed KL amount." This is exactly the principle behind TRPO (§10).

9.2 See it: natural vs. ordinary gradient on a stretched landscape

anisotropy (how stretched the metric is):

Ordinary gradient (orange) vs. natural gradient (blue) descending toward the minimum (★) of an ill-conditioned quadratic — a stand-in for a loss whose Fisher metric stretches one direction far more than another. The ordinary gradient is perpendicular to the contour lines and zig-zags across the narrow valley, wasting steps. The natural gradient pre-multiplies by $\Fish^{-1}$, which "un-stretches" the space, so it points almost straight at the minimum and converges in far fewer steps. This picture is, essentially, why second-order / curvature-aware optimizers (K-FAC, Shampoo, natural policy gradient) beat plain SGD on badly-conditioned problems — drag the slider up to exaggerate the effect. The stretched contours here are the same anisotropy you met as the distinguishability ellipse in §8.2 — that ellipse is a level set of $\delta^\top\Fish\delta$, and $\Fish^{-1}$ is what turns it back into a circle.

Why $\Fish^{-1}$ "un-stretches." $\Fish$ stretches space; $\Fish^{-1}$ is literally the inverse stretch. Natural gradient = (1) temporarily transform to the coordinates where the metric is a plain circle (multiply by $\Fish^{-1/2}$), (2) take the obvious step — in a round space, steepest descent does point at the minimum — and (3) map the direction back. The composition is exactly $\Fish^{-1}\grad L$.

The catch — and the reason half the AI papers below exist — is that $\Fish$ is a $d\times d$ matrix where $d$ = number of parameters. For a modern network $d$ is in the billions, so forming, storing, or inverting $\Fish$ is utterly impossible. Every practical "natural gradient" method is really a clever approximation of $\Fish^{-1}$: diagonal (Adam-ish), block-diagonal Kronecker-factored (K-FAC), low-rank, or a KL trust-region solved without ever materializing $\Fish$ (TRPO's conjugate gradients). Keep this in mind as we tour the applications.

The whole document on one page. You only ever choose ① the metric. Everything in column ② — geodesics, curvature, exp/log maps, the Fisher metric on distributions, the Wasserstein metric — is derived from it. And every AI method in column ③ (§10–§13) hangs off one of those derived objects. "Choosing a geometry = choosing a metric" is the single sentence this map encodes.

That completes the mathematical core. From here on, every section is the same four ideas — metric, geodesic, curvature, natural gradient — wearing the clothes of a specific state-of-the-art AI method. Each connection box names the exact papers (verified, with arXiv IDs) so you can go deep.

10 · AI I — Natural gradient, K-FAC & trust-region RL

The most direct application: use the Fisher metric to optimize. This thread runs from a 1998 theorem straight into how every modern reasoning LLM (DeepSeek-R1, etc.) is trained.

The founding result — natural gradient

Amari, "Natural Gradient Works Efficiently in Learning" 1998Neural Computation 10(2)very-high >5000

Proves that when parameters live on a statistical (Riemannian) manifold, the steepest-descent direction is $\Fish^{-1}\grad L$, not $\grad L$, and that online natural-gradient learning is asymptotically Fisher-efficient (it attains the Cramér–Rao bound — you can't learn faster). This is §9 made into a theorem and is the conceptual root of everything below.

Making it tractable — K-FAC and the modern optimizer zoo

Martens & Grosse, "Optimizing Neural Networks with Kronecker-factored Approximate Curvature" (K-FAC) 2015ICMLhigh >1000 · arXiv:1503.05671

The Fisher of a deep net is billions × billions — un-invertible. K-FAC's insight: each layer's Fisher block factorizes as a Kronecker product $\Fish_\ell \approx A_\ell \otimes G_\ell$ of two small matrices (one from input activations, one from back-propagated output gradients). Since $(A\otimes G)^{-1}=A^{-1}\otimes G^{-1}$, the natural-gradient step becomes cheap.

How K-FAC makes natural gradient runnable. A Kronecker product builds a huge structured matrix from two tiny ones — and its inverse is just the inverses of the tiny ones. That's the whole trick that turns Amari's theorem into something you can actually train with.

This is the canonical practical natural-gradient method and it reappears (as the curvature engine) in §11's Laplace approximation and influence functions.

The same "approximate the curvature/metric cheaply" idea drives a 2018–2026 lineage of optimizers, each choosing a different geometry: Shampoo (Gupta et al. 2018, 1802.09568) uses per-dimension Kronecker preconditioners; distributed Shampoo (Anil et al. 2020, 2002.09018; Shi et al. 2023, 2309.06497) scales it to production; Sophia (Liu et al. 2023, 2305.14342) uses a cheap diagonal-Hessian estimate; SOAP (Vyas et al. 2024, 2409.11321) proves Shampoo ≈ Adam in the preconditioner's eigenbasis; and Muon (Jordan et al. 2024; "Muon is Scalable," Liu et al. 2025, 2502.16982) orthogonalizes the update — which Bernstein & Newhouse (2409.20325) show is steepest descent under the spectral norm, a different metric than Fisher but the same "choose your geometry" philosophy.

Optimizer	Assumed geometry / what it approximates	Curvature structure	Cost
SGD	Euclidean (metric $=I$)	none	cheapest
Adam	diagonal (empirical) Fisher — per-coordinate rescale	diagonal	cheap
K-FAC	block-diagonal Fisher, Kronecker $A\otimes G$ per layer	block + Kronecker	medium
Shampoo	per-dimension Kronecker preconditioners (≈ full Gauss–Newton)	Kronecker, two-sided	medium-high
SOAP	Shampoo in Adam's eigenbasis (cheaper, ≈ Shampoo)	Kronecker eigenbasis	medium
Sophia	diagonal Hessian estimate (clipped)	diagonal Hessian	cheap
Muon	spectral-norm steepest descent (orthogonalized step)	spectral norm, not Fisher	medium
Natural gradient (ideal)	exact Fisher $\Fish^{-1}\grad L$	full $d\times d$	impossible at scale

Every row is a different answer to "what metric should steepest descent use?" Most approximate the Fisher (§9's warning: you can never afford the exact one); Muon instead swaps in the spectral norm. This is the "choose your geometry" idea as a menu.

The killer app — trust-region reinforcement learning

This is the cleanest place where "Fisher = Hessian of KL" (§8.1) becomes a billion-dollar algorithm. Here is the whole lineage as a pipeline — read each box as "what it adds / what it costs":

Fisher = Hessian of KL

the local ruler on policy space(just geometry, §8)

Kakade NPG (2001)

step = F⁻¹g, invariant to parameterizationmust invert F (huge)

TRPO (2015)

hard constraint KL(π_old‖π_new) ≤ δconjugate-grad + Fisher-vector products

PPO (2017)

clip ratio r=π_new/π_old → no F at allonly approximates the KL ball

GRPO / R1 (2024–25)

drop value net; KL penalty to frozen refgroup-relative advantage estimate

◀ the same idea threads all four: a KL trust region — how far the policy may move on the statistical manifold each update ▶

"Trust region" made concrete: the new policy (green) must stay inside a KL-ball (purple) around the old one (blue) — we only trust our reward estimate within that ball.

Term-glosses: trust region · policy gradient · probability ratio.

Kakade, "A Natural Policy Gradient" 2001NeurIPShigh — preconditions the policy gradient by $\Fish^{-1}$ of the policy distribution; the update measures improvement in distribution space, making it invariant to how the policy is parameterized. In the tabular case one natural step = one step of exact policy iteration.
Schulman et al., "Trust Region Policy Optimization" (TRPO) 2015ICMLvery-high >10000 · arXiv:1502.05477 — maximizes the policy's advantage subject to a KL constraint $\KL(\pi_\text{old}\,\|\,\pi_\text{new})\le\delta$. Because the KL's local quadratic form is the Fisher metric, the constrained step is the natural gradient $\Fish^{-1}g$, computed via conjugate gradients + Fisher-vector products so $\Fish$ is never formed. Geometry literally bounds how far the policy "moves" on the statistical manifold each update.
Schulman et al., "Proximal Policy Optimization" (PPO) 2017very-high >20000 · arXiv:1707.06347 — a cheap first-order surrogate for TRPO's trust region: clip the probability ratio (or add a KL penalty) to keep the new policy inside a KL ball without computing $\Fish$. The workhorse behind RLHF.
Tomar et al., "Mirror Descent Policy Optimization" 2020/ICLR'22 · arXiv:2005.09814 — recasts the whole family as mirror descent with a KL Bregman-divergence proximity term, the optimization-theory twin of natural gradient (Raskutti & Mukherjee 2015, 1310.7780, prove mirror descent = natural gradient in dual coordinates).
Shao et al., "DeepSeekMath / GRPO" 2024 · arXiv:2402.03300 and DeepSeek-R1 2025 · arXiv:2501.12948 — the modern LLM-reasoning RL algorithms. GRPO drops the value net (group-relative advantages) but keeps an explicit KL penalty to a frozen reference model. That KL anchor is the same Fisher/trust-region geometry, now constraining how far a reasoning model drifts from its starting distribution.

The through-line: Fisher is the Hessian of KL ⟶ Kakade preconditions by it ⟶ TRPO turns it into a hard KL trust region ⟶ PPO/GRPO approximate that trust region with clipping/penalties. Every time you read "KL penalty" in an RLHF or reasoning-model paper, you are reading information geometry.

11 · AI II — Fisher information across deep learning

Beyond optimization, the Fisher matrix is a Swiss-army knife: it doubles as parameter importance, posterior precision, and influence. The unifying fact (Martens 2020, 1412.1193): for exponential-family losses, Fisher = Generalized Gauss–Newton ≈ Hessian at a minimum, and (Jacot et al. 2018, NTK, 1806.07572) it's the parameter-space dual of the neural tangent kernel. So "use the Fisher" = "use the curvature" = "use the local KL metric." Five places this shows up:

Why Fisher ≈ Hessian (the Gauss–Newton split). This identity licenses the whole section, so here's the half-page. For a loss $L=-\log p(y\mid f_\vth(x))$, the chain rule splits the Hessian into two pieces: $$\grad^2_\vth L = \underbrace{J^\top\!\big(\grad^2_f L\big)J}_{\text{Gauss–Newton term, always PSD}} \;+\; \underbrace{\textstyle\sum_k (\grad_f L)_k\,\grad^2_\vth f_k}_{\text{residual term}},$$ where $J=\partial f/\partial\vth$. The residual term is weighted by $\grad_f L$ — the per-example "error signal" — whose expectation is zero at the true model (the same score-has-mean-zero fact from §7.1). So in expectation (or near a good optimum) the residual term washes out, leaving $\EE[\grad^2 L]=J^\top(\grad^2_f L)J$. For an exponential-family output head (softmax, Gaussian), $\grad^2_f L$ is exactly the output Fisher, so $\Fish=J^\top\Fish_{\text{out}}J$ — the Generalized Gauss–Newton. This is why §11 freely treats Fisher, Gauss–Newton, and the Hessian-at-the-optimum as interchangeable (and why they're guaranteed positive-semidefinite even when the true Hessian isn't); it's also the $A\otimes G$ that K-FAC factorizes in §10 — $A$ from the activation side of $J$, $G$ from the output-gradient side.

Use of Fisher	Paper(s)	How the geometry is used
Continual learning (don't forget old tasks)	Kirkpatrick et al., EWC 2017 PNAS · 1612.00796 very-high	Anchors each weight to its old value with a penalty weighted by the Fisher diagonal. High-Fisher weights are "stiff" (the loss is sharply curved there → important), low-Fisher weights stay plastic. Formally a Laplace approximation: Fisher = precision of a Gaussian posterior carried across tasks.
Bayesian uncertainty (post-hoc, cheap)	Ritter et al. 2018 ICLR; Daxberger et al., Laplace Redux 2021 NeurIPS · 2106.14806	Fit a Gaussian around a trained net's weights whose precision matrix is the Fisher/GGN (K-FAC-factored). Gives calibrated uncertainty + marginal-likelihood model selection with no retraining — "effortless Bayesian deep learning."
Model merging (combine fine-tunes)	Matena & Raffel, Fisher-weighted averaging 2022 NeurIPS · 2111.09832	Merge several models by a per-parameter Fisher-weighted average — each model dominates the merge on the coordinates it's most confident about (high Fisher). The principled cousin of Task Arithmetic (Ilharco et al. 2023, 2212.04089), which deliberately uses no curvature.
Pruning (remove parameters)	Molchanov et al. 2019 CVPR · 1906.10771; LLM-Pruner 2023 · 2305.11627	Score each weight/filter by a Taylor expansion of the loss; the squared-gradient term is an empirical-Fisher importance. Low-Fisher parameters are safe to delete (the Optimal-Brain-Damage lineage), now applied at LLM scale.
Data attribution (which example caused this?)	Koh & Liang, Influence Functions 2017 ICML best paper · 1703.04730; Grosse et al. 2023 (Anthropic, 52B) · 2308.03296	Trace a prediction to its most influential training points via an inverse-Hessian-vector product — and at the optimum the Hessian is the Fisher. Scaling to LLMs needs the EK-FAC Fisher approximation; pure §10 machinery reused for interpretability.

Step back and notice the economy of ideas: one matrix, the Fisher information, simultaneously tells you (a) the best direction to step (natural gradient), (b) how precisely each parameter is determined (Cramér–Rao, EWC stiffness, Laplace precision), (c) how to weight parameters when merging, (d) which parameters are disposable (pruning), and (e) how a training point sways a prediction (influence). They are all the same statement — "the local KL geometry of the model" — read five different ways. The theoretical anchor for why these approximations behave as they do is the Fisher eigenspectrum: Karakida et al. (2019, 1806.01316) show it's wildly anisotropic (a few huge eigenvalues, a sea of near-zero ones), which is exactly the ill-conditioning that makes natural-gradient preconditioning worthwhile.

One matrix, five readings. Natural gradient, parameter importance (EWC/Cramér–Rao), merge-weighting, pruning-disposability, and data influence are the same object — the local KL geometry of the model — read five different ways. Glosses: precision matrix · Laplace approximation.

12 · AI III — Hyperbolic & spherical deep learning

Here the non-Euclidean geometry of §2–§3 returns directly: instead of curving the space of distributions, we curve the embedding space where the model places its representations. The choice of curvature is the choice of inductive bias.

A decision tree for "which curved space do I reach for?" The recurring lesson of the whole document, in one branch point: match the geometry to the structure of your data. Mixed structure? Use a product manifold $\HH^a\times\mathbb S^b\times\RR^c$ with learned curvatures (Gu et al. 2019; HELM's mixture-of-curvature experts). And for moving probability mass between non-overlapping supports, reach for the Wasserstein/Otto metric (§13.3).

The one diagram that justifies hyperbolic AI. Recall (§2.3) that a disk of radius $r$ contains area $\sim r^2$ in flat space but $\sim e^{r}$ in hyperbolic space. A tree with branching factor $b$ has $\sim b^{r}$ nodes at depth $r$ — exponential. So a tree fits in hyperbolic space with almost no distortion in just 2 dimensions, while flat space crushes distant cousins together. Hierarchy ⟺ negative curvature. Conversely, normalizing vectors to a sphere (positive curvature) makes similarity directional (cosine) and bounded — the quiet geometry inside CLIP and every contrastive model.

The picture the whole section rests on. Node depth → distance from center; each subtree gets its own angular wedge. In flat space the leaves (depth grows like $b^r$) overflow the polynomial $r^2$ area, so unrelated "cousins" get crushed to near-zero distance (red). In the hyperbolic disk the area grows like $e^r$, matching the tree's exponential branching, so every node keeps its own room and cousins stay far apart (green). Hierarchy ⟺ negative curvature — this is why hyperbolic embeddings represent trees and taxonomies in just 2 dimensions with tiny distortion.

12.1 Hyperbolic space for hierarchy

The foundations — embed symbolic hierarchies in the Poincaré/Lorentz ball:

Nickel & Kiela, "Poincaré Embeddings for Learning Hierarchical Representations" 2017NeurIPSvery-high >5000 · arXiv:1705.08039 — the seminal paper. Places general concepts near the origin, specific ones near the boundary, so a single radius encodes both hierarchy (norm) and similarity (angle). Trained with Riemannian SGD ($\exp$/$\log$ maps from §6.3). Embeds WordNet with far lower distortion than 200-dim Euclidean, in 5 dims.
Nickel & Kiela, Lorentz model 2018 ICML · 1806.03417 — switches to the hyperboloid where geodesics/$\exp$ have cleaner closed forms and optimization avoids the Poincaré boundary singularity. Ganea et al., Entailment Cones 2018 ICML · 1804.01882 — represents the asymmetric "is-a" relation with nested geodesic cones (a child must lie in its parent's cone), capturing DAG direction, not just distance.

The architectures — make whole networks curved:

Ganea, Bécigneul & Hofmann, "Hyperbolic Neural Networks" 2018 NeurIPS · 1804.03329 — defines hyperbolic linear layers, bias, nonlinearity, softmax via Möbius gyrovector algebra. The recipe is pure §6.3: $\log$ into the tangent space, do a Euclidean op, $\exp$ back. Every later hyperbolic net builds on this.
Chami et al., "Hyperbolic GCN" 2019 NeurIPS · 1910.12933 — graph convolutions on the hyperboloid with trainable per-layer curvature; large gains on scale-free/tree-like graphs. Khrulkov et al., "Hyperbolic Image Embeddings" 2020 CVPR · 1904.02239 — images have latent hierarchy (a scene "entails" its crops); distance-to-origin becomes a natural uncertainty measure. Strong in few-shot/retrieval.

The 2023–2026 frontier — curvature reaches foundation models:

Desai et al., "MERU: Hyperbolic Image-Text Representations" 2023 ICML · 2304.09172 — a hyperbolic CLIP: trains in the Lorentz model with a contrastive + entailment-cone loss so that text geometrically contains the images it describes, recovering the visual-semantic hierarchy CLIP flattens away.
Yang et al., "HypLoRA: Hyperbolic Fine-Tuning for LLMs" 2024 / NeurIPS'25 · 2410.04010 — LoRA updates performed directly on the Lorentz manifold, motivated by measuring tree-like, power-law (negative-curvature) structure in LLM token embeddings; up to ~13–17% gains on hard reasoning. He et al., "HELM" 2025 · 2505.24722 — the first billion-scale fully-hyperbolic LLMs, with a Mixture-of-Curvature Experts that lets each token pick hyperbolic/Euclidean/spherical geometry.

12.2 The hypersphere for similarity & contrast

Radford et al., "CLIP" 2021 ICML · 2103.00020 very-high >5000 — usually called "Euclidean," but it L2-normalizes image and text embeddings onto the unit hypersphere and scores by cosine. The shared spherical space is what enables open-vocabulary zero-shot transfer.
Wang & Isola, "Alignment and Uniformity on the Hypersphere" 2020 ICML · 2005.10242 — the theory of why: contrastive (InfoNCE) loss optimizes two geometric quantities on the sphere — alignment (positives close) and uniformity (features spread out via a sphere energy). Optimizing these directly matches contrastive learning.
Davidson et al., "Hyperspherical VAEs" 2018 UAI · 1804.00891 — replaces the Gaussian latent with a von Mises–Fisher distribution on a sphere, avoiding origin-collapse and better modeling directional data. Mixed curvature: Gu et al., "Product Spaces" 2019 ICLR — embed in $\HH^a\times\mathbb S^b\times\RR^c$ with learned curvatures, matching each factor to the structure it suits. The conceptual ancestor of HELM's curvature experts.

The geometry inside contrastive learning. Wang & Isola proved the InfoNCE loss optimizes exactly these two things on the unit hypersphere: alignment (positive pairs land close) and uniformity (everything else spreads out to fill the sphere). Normalizing embeddings to the sphere isn't incidental — it's the geometry the loss is secretly shaping. (Failure mode: anisotropy.)

12.3 What geometry do LLM representations actually have?

A fast-growing 2023–2026 literature measures the geometry of transformer hidden states — and finds it is emphatically not flat Euclidean:

Manifold & intrinsic dimension: Valeriani et al. 2023 NeurIPS · 2302.00294 — hidden states lie on a low-dimensional curved manifold whose intrinsic dimension rises then falls with depth; the minimum-ID layer gives the most transferable features (building on Ansuini et al. 2019, 1905.12784). Viswanathan et al. 2025 · 2501.10573 — token-cloud intrinsic dimension correlates with next-token loss (harder prompts → higher-dimensional token geometry).
Curvature of trajectories: Hosseini & Fedorenko 2023 NeurIPS · 2311.04930 — LLMs progressively straighten a sentence's layer-by-layer trajectory; more straightening ⇒ better prediction (predict by linear extrapolation).
Anisotropy: Ethayarajh 2019 EMNLP · 1909.00512 — contextual embeddings occupy a narrow cone (highly anisotropic); Godey et al. 2024 EACL · 2401.12143 argue this is inherent to self-attention.
A learned non-Euclidean metric for concepts: Park, Choe & Veitch, "The Linear Representation Hypothesis and the Geometry of LLMs" 2023/ICML'24 · 2311.03658 — the right inner product on representation space is not Euclidean but a learned causal inner product (a Riemannian-style metric) under which unrelated concepts are orthogonal. Park et al. 2024/ICLR'25 · 2406.01506 show categorical concepts form simplices and hierarchies become orthogonal polytopes under that metric — directly the dually-flat / metric ideas of §8 inside a live LLM.
The skeptical counterpoint: Robinson et al., "Token embeddings violate the manifold hypothesis" 2025 · 2504.01002 — a statistical test rejects local flatness; token space has singularities (a stratified structure, not a clean manifold). A healthy reminder that the geometry is real but messier than any single model says.

13 · AI IV — Riemannian generative models & optimal transport

The final thread fuses both halves: generative models that either live on a Riemannian manifold (§4–§6) or that move probability mass under an optimal-transport geometry — itself a Riemannian metric on the space of distributions (§8).

Five words to own before the papers (the jargon that otherwise blocks this section):

score — the gradient of a log-density. Careful, two meanings: $\grad_\theta\log p$ (w.r.t. parameters → Fisher, §8) vs $\grad_x\log p$ (w.r.t. the data point → "which way is more-probable data?"). §13 means the second one — that's what diffusion models learn.
geodesic interpolation — instead of drawing a straight line between two points and reading off the midpoint, walk the shortest path that stays on the curved surface. On a sphere that's a great-circle arc, not the chord through the inside.
Wasserstein / earth-mover distance — the cost to turn one pile of probability mass into another by physically moving mass: cost = amount moved × distance moved. Unlike KL, it stays finite even when the two distributions don't overlap.
equivariant — rotate the input, and the output rotates the same way: $f(R\vx)=Rf(\vx)$. Stronger than invariant (output unchanged). A protein generator must be equivariant — rotate the molecule, the predicted forces rotate with it.
ELBO — Evidence Lower BOund: the trainable lower bound on log-likelihood that VAEs maximize, $=$ reconstruction $-\,\KL(\text{posterior}\,\|\,\text{prior})$.

13.1 Flow matching & diffusion on manifolds

Lipman et al., "Flow Matching for Generative Modeling" 2022/ICLR'23 · 2210.02747 very-high >5000 — trains a continuous normalizing flow by regressing a vector field onto fixed conditional probability paths (no ODE simulation). Its optimal-transport path variant uses straight-line interpolants — the displacement geodesics of Wasserstein geometry.
Chen & Lipman, "Flow Matching on General Geometries" (Riemannian Flow Matching) 2023/ICLR'24 · 2302.03660 — generalizes flow matching to arbitrary Riemannian manifolds (spheres, tori, hyperbolic, meshes). Conditional vector fields follow geodesics toward each data point; stays simulation-free where geodesics are closed-form. The manifold-native generative recipe.
De Bortoli et al., "Riemannian Score-Based Generative Modelling" 2022 NeurIPS · 2202.02763 and Huang et al., "Riemannian Diffusion Models" 2022 NeurIPS · 2208.07949 — diffusion's forward/reverse SDEs defined intrinsically on a manifold, learning the Riemannian score — here "score" means $\grad_x\log p$, the direction in data space toward higher probability (not the parameter-gradient of §8), measured using the manifold's metric. Used for climate data on spheres, $SO(n)$, hyperboloids.

13.2 SE(3)-equivariant generation for science — geometry that won Nobel-adjacent results

A protein backbone is a sequence of rigid frames — a point on the product manifold $SE(3)^N$ (rotations $\times$ translations). Generating molecules/proteins = running a diffusion or flow on that manifold, respecting its symmetry. This is the most consequential real-world payoff of Riemannian generative modeling.

Why protein generation is Riemannian. Each amino acid is a little coordinate triad (a rigid pose). The whole backbone is one point on $SE(3)^N$; a generative model diffuses or flows that point on the manifold. The model must be equivariant: rotate the molecule and every frame's prediction rotates identically. Glosses: SE(3) · SO(3) · torsion torus · IGSO(3).

Satorras et al., "E(n) Equivariant GNNs" (EGNN) 2021 ICML · 2102.09844 very-high >5000 — the equivariant backbone; updates coordinates along relative-position vectors so outputs rotate/translate with inputs, no expensive spherical harmonics. Hoogeboom et al., "Equivariant Diffusion (EDM)" 2022 ICML · 2203.17003 — diffusion directly on 3D atomic coordinates with an E(3)-equivariant denoiser.
Yim et al., "SE(3) Diffusion / FrameDiff" 2023 ICML · 2302.02277 — principled score-based diffusion on $SE(3)^N$, separate processes on $SO(3)$ (via the $\mathrm{IGSO}(3)$ distribution) and $\RR^3$. FrameFlow (2310.05297) & FoldFlow (2310.02391, ICLR'24 spotlight) recast this as SE(3) flow matching with Riemannian optimal-transport couplings for straighter, faster paths.
Watson et al., "RFdiffusion" 2023 Nature 620 — SE(3)-frame diffusion that designs real, experimentally-validated proteins (binders, scaffolds, oligomers). Abramson et al., "AlphaFold 3" 2024 Nature 630 — a diffusion module over all-atom coordinates for entire biomolecular complexes. Corso et al., "DiffDock" 2022/ICLR'23 · 2210.01776 — docking as diffusion on a product of non-Euclidean manifolds: $\RR^3\times SO(3)\times\mathbb T^m$ (translation × rotation × torsion torus).

13.3 Optimal transport: a Riemannian metric on the space of distributions

The Wasserstein distance equips the space of probability distributions with its own Riemannian structure (the Otto metric), where geodesics are mass-transport plans and gradient flows describe diffusion. This is information geometry's transport-flavored sibling.

Optimal transport, literally moving piles of dirt. The Wasserstein distance is the cheapest way to rearrange one probability "pile" into another (cost = mass × distance). Its geodesic is the smooth morph in the bottom row — and flow matching's straight-line paths are an approximation of exactly this.

What JKO / "diffusion = gradient flow" actually says. Picture entropy (disorder) as a hill, and the space of all distributions as the terrain. JKO (1998) proved that a diffusion process is literally gradient descent of free energy, where "downhill" is measured by the Wasserstein (dirt-moving) ruler instead of ordinary distance. That's the precise sense in which score-based diffusion models are "gradient flows on a probability manifold" — they roll downhill on the space of distributions, under the optimal-transport metric.

Arjovsky et al., "Wasserstein GAN" 2017 ICML · 1701.07875 very-high >20000 (+ WGAN-GP, Gulrajani et al. 1704.00028) — replaces the JS divergence with the Wasserstein-1 (earth-mover) distance, giving meaningful gradients even when model and data supports are disjoint, curing mode collapse. The breakthrough that made OT a first-class deep-learning tool.
Cuturi, "Sinkhorn Distances" 2013 NeurIPS · 1306.0895 — entropic-regularized OT solvable by fast GPU matrix-scaling; the computational backbone of nearly all OT-in-deep-learning (Feydy et al., Sinkhorn divergences, 1810.08278, interpolate OT↔MMD).
Jordan, Kinderlehrer & Otto (JKO) 1998 SIAM J. Math. Anal. — proves the Fokker–Planck equation is the Wasserstein gradient flow of free energy: diffusion = steepest descent of entropy in the $W_2$ metric. This "Otto calculus" is why score-based diffusion models can be read as gradient flows on a probability manifold (made explicit for deep models by Zhang et al. 2024, 2402.06162). OT-CFM (Tong et al. 2023, 2302.00482) & multisample flow matching (Pooladian et al. 2023, 2304.14772) use minibatch-OT couplings to straighten flows toward the Wasserstein geodesic, cutting sampling steps.

13.4 The hidden geometry of latent spaces

Even an ordinary VAE/GAN/diffusion latent space is secretly Riemannian: the decoder $g$ pulls the data-space metric back into the latent space, $\Met(\vz)=J_g(\vz)^\top J_g(\vz)$ (the decoder Jacobian), so "straight-line interpolation" in latent coordinates is geometrically wrong — you should follow geodesics.

Why latent interpolation needs geometry. The decoder warps distances, so a straight line in latent coordinates (red) cuts across regions the model never learned — producing blurry, invalid midpoints. The geodesic under the pulled-back metric (green) hugs the data manifold, so every interpolated point decodes to something realistic.

interpolation t:

See the decoded midpoint degrade. Slide $t$ from 0 to 1. Left: the curved data manifold with the two paths from A to B. Right: the "decoded sample" at the current $t$ for each path — its sharpness tracks how close the path is to the manifold (high data-density = crisp, off-manifold = blurry). The straight-line path wanders off the manifold near the middle and its sample smears; the geodesic hugs the manifold and stays crisp the whole way. The readout shows path length measured in the pulled-back metric — the geodesic is shorter there even though it looks longer on screen. That is the entire payoff of the exp/log/geodesic machinery from §6.3.

Arvanitidis et al., "Latent Space Oddity" 2018 ICLR · 1710.11379 — the founding paper: the decoder Jacobian induces a (stochastic) Riemannian metric; geodesics under it follow the data manifold instead of cutting through empty space, fixing interpolation, clustering, and random walks. Extended to non-Gaussian decoders by pulling back the Fisher–Rao metric (Arvanitidis et al. 2022, 2106.05367).
Diffusion-era (2025): Karczewski et al., "The Spacetime of Diffusion Models" · 2505.17517 equips the diffusion latent with the Fisher–Rao metric of the denoising distributions; Saito & Matsubara · 2510.05509 derive a training-free metric from the score Jacobian so guidance stays tangent to the data manifold. The §8 Fisher metric, now inside Stable-Diffusion-scale models.

14 · Geometry in production — where this actually ships

None of this is whiteboard-only. Every concept above is running inside products you've used this week. Here is the map from idea → shipped system, with the company and model named. The pattern to notice: a geometry paper takes 2–7 years to become an API knob or a frontier training run.

The four cleanest "geometry escaped the whiteboard" stories

Muon → Kimi K2. An orthogonalized-update optimizer (steepest descent under the spectral norm, §10) went from a hobbyist NanoGPT speed-run to training Moonshot AI's >1-trillion-parameter production model at ~2× the compute efficiency of AdamW (2502.16982).
Matryoshka embeddings → OpenAI's dimensions knob. A NeurIPS paper about nesting representations on the hypersphere (2205.13147) became, in under two years, a literal API parameter: text-embedding-3 lets you truncate a unit-sphere vector from 3072→256 dims and renormalize, trading accuracy for speed with no retraining.
GNNs → Google Maps ETA. A road network is a non-Euclidean graph; a graph neural net reasoning over it ships in Google Maps with up to ~50% better ETA accuracy, used by billions (2108.11482). A co-author also wrote the Geometric-Deep-Learning blueprint — a clean theory→deployment bridge.
Feature geometry → Golden Gate Claude. Anthropic extracted ~34M monosemantic feature directions from Claude 3 Sonnet's activation space, then publicly shipped a model with the "Golden Gate Bridge" direction clamped high — the clearest demonstration that representation geometry steers a live frontier LLM (Scaling Monosemanticity).

① Optimizers at frontier scale — Fisher & norm geometry (§9–§10)

Muon → Moonshot AI Kimi K2 / Moonlight (2502.16982) — spectral-norm steepest descent, ~2× AdamW efficiency at >1T params. Theory: modular duality (Bernstein & Newhouse, 2410.21265); Scion (2502.07529).
Distributed Shampoo → won MLCommons AlgoPerf 2024 (Meta, 2309.06497) — a Kronecker-factored second-order (Fisher/Gauss–Newton) preconditioner beat tuned Adam by ~28% on the neutral industry benchmark. SOAP (2409.11321) runs Shampoo in Adam's eigenbasis.
EK-FAC influence functions → Anthropic, 52B LLMs (2308.03296) — Fisher/Kronecker curvature deployed to trace which training data drives a model's behavior. Fisher-weighted merging (2111.09832) underpins model-soup pipelines.

② RL post-training — the KL trust-region geometry (§8.1, §10)

GRPO → DeepSeek-R1 (2402.03300) and the whole 2025 reasoning-model wave; DAPO → ByteDance (2503.14476, asymmetric "Clip-Higher" trust region); GSPO → Qwen3 (2507.18071, sequence-level ratio that stabilizes MoE RL); Dr.GRPO (2503.20783) fixes a normalization bias in the advantage geometry.
InstructGPT → ChatGPT (2203.02155) — the per-token KL penalty to a reference policy that all the above still argue about is a trust region in distribution space; the geometric ancestor of essentially all production RLHF.

③ Generative media — flow matching & optimal-transport straightening (§13.1, §13.3)

Stable Diffusion 3 / 3.5 → Stability AI (2403.03206) — built on rectified flow (2209.03003), the straight-line OT-displacement transport from noise to data. FLUX.1 → Black Forest Labs (12B rectified-flow transformer, the image engine behind several products). Movie Gen → Meta (2410.13720, 30B flow-matching model for video).
InstaFlow / Consistency Models / LCM (2309.06380, 2303.01469) — straighten-then-distill the transport path to a single step, powering real-time image generation in consumer apps.

④ Drug discovery & science — SE(3)/Riemannian generation (§13.2)

RFdiffusion → Xaira Therapeutics ($1B launch, 2024; Baker lab, Nature 620) — genuine $SE(3)^N$ frame diffusion (rotations on $SO(3)$ via IGSO(3)). RFantibody extends it to antibody design (Nature 2025).
AlphaFold 3 → Isomorphic Labs (Nature 630; Lilly + Novartis deals ~$3B) — diffusion over atomic coordinates. AlphaFold 2's Invariant Point Attention operates on residue frames as $SE(3)$ elements. Chroma → Generate Biomedicines; Boltz-2 → Recursion (open AF3-class model + binding affinity).

⑤ Robotics & embodied AI — SE(3) poses + diffusion/flow policies (§13.2)

Diffusion Policy → TRI Large Behavior Models (2303.04137) — diffusion over action trajectories, scaled by Toyota Research Institute to real two-armed robots. π₀ → Physical Intelligence (2410.24164) — a flow-matching action expert on a VLM, deployed folding laundry. GR00T → NVIDIA (flow-matching action head). Riemannian Motion Policies → NVIDIA Isaac (1801.02854).

⑥ Retrieval & embeddings — the unit hypersphere (§12.2)

Matryoshka → OpenAI text-embedding-3 (2205.13147), Nomic, Cohere — nested sphere-normalized embeddings as an API dimensions knob. E5 → Microsoft (2212.03533), the open-weight RAG workhorse, trained with alignment+uniformity on the sphere (2005.10242).
ScaNN → Google Vertex Vector Search (1908.10396, anisotropic quantization tuned to inner-product geometry); FAISS → Meta (1702.08734), the ANN engine under most vector databases. For unit vectors, inner product = cosine = spherical geometry.

⑦ Graphs & hyperbolic structure in products (§12.1)

GNN ETA → Google Maps (2108.11482, ~50% accuracy gain); PinSage → Pinterest (1806.01973, 3B nodes / 18B edges). The Geometric-Deep-Learning "5G" blueprint (2104.13478) is the unifying theory.
Hyperbolic KG embeddings → Google (RotH/AttH, 2005.00545; MuRP, 1905.09791) win at the low dimension production serving demands. HERec → Snap (2411.13865) — a fully-hyperbolic recommender from Snap's personalization team, matching the power-law user–item graph.

⑧ Interpretability & steering — the geometry of representations (§12.3)

Golden Gate Claude → Anthropic (Scaling Monosemanticity) — a sparse-autoencoder feature direction clamped in a live production model. Representation Engineering (2310.01405) and Contrastive Activation Addition (2312.06681) steer behavior by adding a direction in activation space — the dominant training-free control technique.
Linear Representation Hypothesis (2311.03658) shows the right metric on that space is a non-Euclidean causal inner product (§12.3) — the theoretical license for why steering vectors work at all.

15 · Master cheat-sheet & further reading

15.1 The whole story on one page

Concept	Symbol	One-line meaning	AI appearance
Manifold	$M$	locally-flat curved space	data manifold; embedding space; $SE(3)^N$ of a protein
Tangent space	$T_pM$	local linear space of directions	where gradients/velocities live
Metric tensor	$g_{ij}$	position-dependent Pythagoras, $ds^2=d\vx^\top \Met\,d\vx$	the central object — choosing geometry = choosing $g$
Geodesic	$\gamma$	straightest/shortest path	latent interpolation; flow-matching paths; hyperbolic distance
Curvature	$K, R^l_{ijk}$	failure of metric to be constant; holonomy	hyperbolic ($K\!<\!0$, hierarchy) vs sphere ($K\!>\!0$, similarity)
$\exp$/$\log$ maps	$\exp_p,\log_p$	hop between manifold & tangent space	every hyperbolic layer; Riemannian SGD
Score (params)	$\grad_\theta\log p$	likelihood sensitivity to parameters → builds Fisher	MLE, Fisher information (§7–8)
Score (data)	$\grad_x\log p$	direction toward more-probable data	score-based diffusion / score matching (§13)
Fisher information	$\Fish$	var of score = Hessian of KL = metric on distributions	natural gradient, EWC, Laplace, merging, influence
Fisher–Rao metric	$g=\Fish$	the metric of the statistical manifold	information geometry; reparam-invariant updates
KL divergence	$\KL(p\\|q)$	$\approx\tfrac12\delta^\top\Fish\delta$; canonical divergence	TRPO/PPO/GRPO trust region; VAE/variational loss
Natural gradient	$\Fish^{-1}\grad L$	steepest descent in Fisher metric	K-FAC, Shampoo, natural policy gradient
Wasserstein metric	$W_2$	Riemannian metric via mass transport	WGAN, OT flow matching, diffusion as gradient flow

15.2 The four "aha" equations to keep

1. The metric IS the geometry $$ds^2 = d\vx^\top \Met(\vx)\, d\vx$$ Constant $\Met$ → flat. Varying $\Met$ → curved. Everything (geodesics, curvature) is built from $\Met$.

2. Fisher = curvature of KL $$\KL(p_\vth\|p_{\vth+\delta})\approx \tfrac12\delta^\top\Fish\delta$$ The space of distributions is curved, and $\Fish$ is its metric tensor.

3. Natural gradient un-distorts $$\vth\leftarrow\vth-\eta\,\Fish^{-1}\grad L$$ Steepest descent in the right ruler; reparameterization-invariant; a KL trust region.

4. Curvature = inductive bias $$K<0 \leftrightarrow \text{hierarchy},\quad K>0 \leftrightarrow \text{similarity}$$ Pick the embedding curvature that matches your data's structure.

15.3 A learning path (free resources first)

Most accessible route:

Surfaces & curvature → Pressley, Elementary Differential Geometry (Ch. 10 Theorema Egregium, Ch. 13 Gauss–Bonnet).
Riemannian intuition (physicist style) → Sean Carroll's GR lecture notes (metric, Christoffels, geodesics, Riemann tensor with worked index gymnastics).
Information geometry → Nielsen, "An Elementary Introduction to Information Geometry" (arXiv:1808.08271) — the best free, self-contained tutorial.

Rigorous route: do Carmo, Differential Geometry of Curves and Surfaces → do Carmo / J.M. Lee, Riemannian Geometry → Lehmann & Casella, Theory of Point Estimation (Fisher information done right) → Amari & Nagaoka, Methods of Information Geometry (the canonical text) and Amari, Information Geometry and Its Applications (2016, more pedagogical).

Foundational primary sources: Riemann's 1854 habilitation lecture On the Hypotheses Which Lie at the Bases of Geometry; Gauss's Theorema Egregium (1827); Fisher (1925); Rao (1945, Fisher–Rao metric & Cramér–Rao); Chentsov (1972, uniqueness of the Fisher metric).

Go deeper — verified free resources by topic

Topic	Best next resource (free where ✓)
Geometric deep learning (the unifying frame)	✓ Bronstein, Bruna, Cohen, Veličković, GDL: Grids, Groups, Graphs, Geodesics, Gauges — arXiv:2104.13478 + lecture course
Differential geometry, CS-friendly + code	✓ Keenan Crane, Discrete Differential Geometry — CMU course + videos; ✓ Carroll's GR notes for index intuition
Optimal transport	✓ Peyré & Cuturi, Computational Optimal Transport — free book; libraries POT & OTT-JAX
Information geometry	✓ Nielsen, Elementary Introduction to Information Geometry — arXiv:1808.08271; Martens, Natural Gradient — 1412.1193
Flow matching (geometric generative)	✓ Lipman et al., Flow Matching Guide and Code — arXiv:2412.06264 + PyTorch library
Hyperbolic / geometry-in-foundation-models	✓ surveys: Peng et al. 2021 (2101.04562); Mettes et al. IJCV 2024; Hyperbolic DL for Foundation Models 2025 (2507.17787)
Intuition warm-up	✓ 3Blue1Brown — Essence of Linear Algebra / Calculus

Suggested order: 3Blue1Brown → GDL book+lectures → Crane (hands-on) → Nielsen (info-geo) → Martens (natural gradient) → Peyré–Cuturi (OT) → Lipman (flow matching) → hyperbolic surveys.

15.4 Timeline — pure math becomes AI

📁 Companion file in this folder

A full, organized bibliography of every paper referenced here — with authors, year, venue, arXiv ID, citation tier, and a one-line note on how each uses the geometry — is in REFERENCES.md (and grouped by theme). This HTML is the tutorial; that file is the map of the literature.

Built as a self-contained explainer · all papers verified via web search with arXiv IDs · math rendered with MathJax · diagrams hand-drawn in SVG · interactive demos in vanilla JS.
Geometry is just the rule for measuring small steps. Everything else — intelligence included — is what you build on top of that rule.

一份自洽、从零搭建的指南 · 写给 AI 研究者与好奇的本科生

智能的几何
面向 AI 的黎曼几何与信息几何

从平行公设出发，一路抵达自然梯度、双曲 transformer、流形上的流匹配——每个概念都从零定义，配以直觉、算例数字、图示，以及那些把数学变成最先进 AI 的确切论文。

非欧几里得几何黎曼流形 Fisher 信息信息几何自然梯度 · K-FAC · TRPO 双曲嵌入流匹配与最优传输

1 · 60 秒全景图

四个主题，一个思想：距离并不是你以为的那样。两点之间的最短路径、两个方向之间的"夹角"，乃至"这两样东西离得远吗？"这个问题——所有这些都依赖于一个隐藏的对象，叫做度量。改变度量，你就改变了几何。这四个领域，是对"什么才是正确的度量？"给出的四个答案。

本文的依赖关系图。非欧几里得几何表明，欧几里得那个平直的世界只是众多选项之一。黎曼几何是弯曲空间的一般理论。Fisher 信息发现：概率分布所构成的空间本身就是弯曲的。把黎曼几何对准这个空间，你得到的便是信息几何——它正是通往绝大多数 AI 应用的桥梁。

每个领域的一段话版本

① 非欧几里得几何——两千年来，人们一直试图用欧几里得的另外四条公理证出他的第五公理（平行公设）。他们失败了，因为它是独立的：你完全可以构造出自洽的几何，在其中它是假的。在球面上没有平行线，三角形内角和会鼓出超过 180°；在双曲空间里有无穷多条平行线，三角形则被压缩到 180° 以下。正是这一冲击教会了数学："平直"是一种选择。
② 黎曼几何——黎曼的飞跃：不要只挑一个弯曲空间，而要给出生成所有弯曲空间的配方。在每一点上，你都附着一把微型的尺与量角器（即度量张量 $\Met$），它告诉你微小步长的长度。仅凭这条局部规则，你就能推出全局长度、最短路径（测地线）以及曲率。这正是广义相对论的语言——也是现代几何深度学习的语言。
③ Fisher 信息——一个统计量：当某个参数发生变化时，你数据的似然反应得有多剧烈？Fisher 信息越高，意味着这个参数越容易从数据中被精确确定。它深藏的秘密是：它恰恰就是概率分布空间所对应的那个正确的度量张量。
④ 信息几何——把 ② 和 ③ 合在一起。一个模型中所有分布的集合（比如所有高斯分布，或一个神经网络 softmax 的所有取值设置）构成一个弯曲的流形，其度量就是 Fisher 信息。距离变成了"这两个模型有多可区分？"，直线变成了自然梯度更新，而勾股定理则以 KL 散度的形式重新现身。这便是自然梯度、TRPO、EWC 以及更多方法背后的引擎。

首先，是那个万物所依的词："度量"。本文余下内容都系于它，所以这里就具体地讲清楚，不打比方。一点处的度量是一个对称、正定的 $n\times n$ 矩阵 $\Met$。它推广了点积：微小步长 $d\vx$ 的长度平方为 $$ds^2 = d\vx^\top \Met\, d\vx \quad\text{instead of}\quad d\vx^\top d\vx.$$ 平直空间无非就是 $\Met = I$（单位矩阵）。"弯曲空间"意味着 $\Met$ 随你的移动而变化。这就是全部的思想——一把随位置而变的尺。

30 秒算例。取 $\Met=\begin{pmatrix}4&0\\0&1\end{pmatrix}$，步长 $d\vx=\begin{pmatrix}1\\1\end{pmatrix}$。普通的（平直的）长度本应是 $\sqrt{1^2+1^2}=\sqrt2\approx1.41$。但用这个度量： $$ds^2 = \begin{pmatrix}1&1\end{pmatrix}\begin{pmatrix}4&0\\0&1\end{pmatrix}\begin{pmatrix}1\\1\end{pmatrix}=4\cdot1^2+1\cdot1^2=5,\quad ds=\sqrt5\approx2.24.$$ $x$ 方向被"拉伸"了——它那把尺长了一倍（$4=2^2$）。正是这一个思想——一个对距离重新加权的矩阵——同时就是非欧几里得几何、黎曼几何、Fisher 信息和信息几何，四者皆然。它们的区别只在于用的是哪个矩阵 $\Met$，以及它从何而来。

2 · 非欧几里得几何：打破欧几里得

符号速查（故意保持精简——随时回看）：

$\vx$ —— 一个点（粗体 = 向量/位置）· $\|\vv\|$ —— $\vv$ 的普通长度 · $d\vx$ —— 一个微小步长

$ds$ —— 该微小步长的真实（度量）长度 · $dA$ —— 一小块面积 · $\Met$（或 $G$）—— 度量张量（随位置变化的尺子）· $K$ —— 曲率

这一切都始于一场围绕一句话、持续了 2300 年的争论。

2.1 欧几里得的五条公设与那个麻烦制造者

大约公元前 300 年，欧几里得用五条公设构建了整个平面几何。前四条简短而"显而易见"：

任意两点之间可以画一条直线。
线段可以无限延长。
可以以任意圆心和半径画一个圆。
所有直角彼此相等。

第五条则啰嗦冗长，感觉像是个伪装成公设的定理：

平行公设（Playfair 的简洁形式）。 给定一条直线 $\ell$ 和一个不在其上的点 $P$，过 $P$ 恰好有一条直线永不与 $\ell$ 相交。

在两千年里，数学家们都相信第 5 条必定能从第 1–4 条推出，并试图去证明它。可每一个"证明"都暗中假定了某个与第 5 条等价的命题。这一难题最终由高斯（未发表）、Bolyai（1832）和Lobachevsky（1829）各自独立地解决：这条公设是独立的。你可以否定它，从而得到一种全新的、完全自洽的几何。根据过 $P$ 的平行线数量不同，共有三种结果：

三种几何，只差一条公理。 过 $P$ 且与给定直线平行的直线数量为 $1$（平直）、$0$（球面——任意两条"大圆"都相交，就像两极处的子午线）或 $\infty$（双曲）。大圆是球面上最大的圆（赤道、任意一条子午线）——也就是球面版本的直线。三角形的内角和与曲率 $K$ 的符号，都被这条选择所绑定。

曲率 K:

一个三角形，一个旋钮。 这里不是三个彼此独立的宇宙，而是同一个三角形，随着你把曲率 $K$ 从 $-1$ 扫到 $+1$ 而变化。$K>0$（类球面）：边向外鼓出，内角和爬升超过 $180°$。$K=0$：边是直的，内角和恰为 $180°$。$K<0$（双曲）：边向内收缩，内角和降到 $180°$ 以下。实时读数展示了 Gauss–Bonnet 定律的实际效果：$(\text{角度和})-180° = K\times\text{面积}$ —— 曲率正是每单位面积上你所获得（或失去）的角度。

三种几何并列对照

性质	欧几里得（平直）	球面（椭圆型）	双曲
曲率 $K$	$0$	$>0$（例如 $+1/R^2$）	$<0$
过一点的平行线	恰好 1 条	0（所有测地线都相交）	无穷多条
三角形内角和	$=180°$	$>180°$（向外鼓）	$<180°$（向内收）
"直线"（测地线）	普通直线	大圆	与边界垂直的弧（庞加莱）
半径为 $r$ 的圆的周长	$2\pi r$	$2\pi\sin r\ (<2\pi r)$	$2\pi\sinh r\ (>2\pi r)$
圆盘面积增长方式	$r^2$（多项式）	有界（自我闭合）	$e^{r}$（指数）
能摊平到纸上吗？	能（本就平直）	不能（必须撕开——地图会失真）	不能（空间太多）
天然契合的数据	网格、表格	方向、相似度	树、层级结构
在 AI 中的出现	普通 SGD、$\ell_2$ 嵌入	CLIP、对比学习（§12.2）	庞加莱/Lorentz 嵌入（§12.1）

那一行面积指数增长，正是支撑全部双曲 AI（§12）的唯一事实：一棵树的节点数像 $e^{r}$ 一样增长，因此只有空间本身也像 $e^{r}$ 一样增长的几何，才能无失真地容纳它。

2.2 在曲面上，什么是"直线"？

整个把戏就是把"直线"重新定义为测地线——而那两个听起来自相矛盾的说法其实是同一回事。测地线是一条从旅行者视角看是笔直的路径：走的时候始终不向左或向右打方向盘，你画出的就是一条测地线。在平直平面上，那就是普通直线；在球面上，它是一条大圆（赤道、任意子午线）——这正是为什么长途航班的航线会朝两极弯曲。这样一条不转弯的路径，会自动成为邻近两点之间的最短路线；"局部最短"是指与它周围的小幅摆动相比是最短的，即便绕一圈可能还有更长的别的走法。两条不同的大圆总是相交（在一对对径点上），所以根本不存在平行线——而正是这一个事实，迫使三角形变"胖"。

为什么"最直" = "局部最短"。 绿色弧线在必须留在球面上的前提下尽可能少地弯曲；任何偏离它的小幅摆动都会增加长度。

一个有三个直角的三角形。 从北极出发。沿一条子午线笔直走到赤道。转 90°，沿赤道走四分之一圈。再转 90°，笔直走回北极。你画出的三角形有三个 90° 的角——内角和 $270°$，比欧几里得的 $180°$ 整整多出 $90°$。这个超出量不是误差；它就是曲率在整个三角形上的积分。

这正是 Gauss–Bonnet 定理的雏形：$(\text{角度和}) - 180° = \iint_{\triangle} K \, dA$（以弧度计，$180°=\pi$）。来验证一下：我们这个三直角三角形的超出量为 $270°-180°=90°=\tfrac\pi2$，在单位球面上它覆盖了面积为 $\tfrac\pi2$ 的一个八分之一球面，且 $K=1$ —— 于是 $\iint K\,dA = 1\cdot\tfrac\pi2 = \tfrac\pi2$。两边相等。曲率正是你绕行一圈所获得的角度。

2.3 庞加莱圆盘——你能握在手里的双曲空间

双曲空间更难想象，因为它有"太多空间"，塞不进平直的纸面。庞加莱圆盘模型是标准的"作弊"手法：把整个无限的双曲平面压进一个单位圆盘的内部。代价是：当你靠近边界时，尺子会缩小——在边缘附近，一个看起来很小的步长，实际上是巨大的。边界圆"无限远"。

左图：庞加莱圆盘中的直线（测地线）是与边界成直角相交的圆弧；绿色直径也是一条测地线。过一个点，你可以画出无穷多条永不与某条给定弧相交的弧——这就是 $\infty$ 条平行线。右图：度量 $ds = \tfrac{2\,\|d\vx\|}{1-\|\vx\|^2}$ 在 $\|\vx\|=1$ 附近发散，因此距离在边缘处爆炸式增长。

第一个真正的度量，用实际数字呈现。 解读这些符号：$\vx$ 是你当前的位置（到中心的距离 $\|\vx\|$，介于 $0$ 和 $1$ 之间）；$d\vx$ 是一个微小步长；$\|d\vx\|$ 是它普通的平直长度；$ds$ 是它真实的双曲长度。这个度量只是把平直长度乘上一个仅取决于你离中心多远的因子 $\dfrac{2}{1-\|\vx\|^2}$：

位置 $\\|\vx\\|$	因子 $\frac{2}{1-\\|\vx\\|^2}$	0.01 的平直步长变为…
0（中心）	2.0	0.02 —— 轻微
0.9	10.5	0.105 —— 约长 5 倍
0.99	100.5	~1.0 —— 约长 50 倍
→ 1（边缘）	→ ∞	→ ∞ —— 边界无限远

所以边缘只是看起来近：要真正到达它，你得对一个发散的因子做积分，也就是走无限远。这种空间爆炸式增长的性质，正是双曲空间成为 AI 中树与层级结构天然归宿的全部原因（§12）。

拖动两个白点 ·

亲手玩转双曲距离。 拖动点 A 和 B。绿色测地线是与边界成直角相交的圆弧——即双曲版的"直线"。注意观察两个读数：当你把一个点滑向边缘时，屏幕（欧几里得）距离几乎不变，但真实双曲距离 $d=\operatorname{arccosh}\!\big(1+\tfrac{2\|a-b\|^2}{(1-\|a\|^2)(1-\|b\|^2)}\big)$ 却爆炸式增长。沿测地线均匀分布的刻度点是相等的双曲步长——它们在边缘附近明显地挤在一起，让"相等的真实长度在边界附近看起来很小"这件事变成你能亲眼看见、而不只是读到的。

为什么一棵树"想要"双曲空间？数一数节点：一棵二叉树在深度 $d$ 处有 $2^d$ 个节点——事物的数量随着到根的距离指数级增长。在平直空间里，半径为 $r$ 的圆盘面积像 $r^2$（多项式）增长，于是你会用尽空间，邻居被挤成一团。在双曲空间里，半径为 $r$ 的圆盘面积像 $e^{r}$ 增长——指数级，恰好与树相匹配。空间"足够多"，能把每个节点放得远离它的表亲。把层级嵌入平直空间，距离会失真；把它嵌入双曲空间，仅用 2 维距离就几乎精确。请记住这一点，留待 §12。

3 · 曲率与绝妙定理

我们一直在说"曲率"。现在让它变成一个数字吧。最清晰的切入点是二维曲面的高斯曲率 $K$。

3.1 两个半径，一个数

站在曲面上某一点，观察它沿不同方向如何弯曲。总存在一个弯曲最厉害的方向和一个弯曲最平缓的方向（两者总是相互垂直）。每个方向都有一个带符号的曲率 $\kappa=1/R$——而符号正是关键所在。选定曲面的某一侧称作"上方"：朝着上方弯曲的方向取 $\kappa>0$，背离上方（向下）弯曲的方向取 $\kappa<0$。这两个极端值 $\kappa_1,\kappa_2$ 就是主曲率。高斯的定义是：

$$K = \kappa_1 \,\kappa_2 = \frac{1}{R_1 R_2}.$$

曲面	$\kappa_1,\kappa_2$	$K$	感觉像
平面 / 圆柱面	圆柱面：绕圈方向 $\kappa_1=1/R$，沿轴方向 $\kappa_2=0$	$K=\frac1R\cdot0=0$	平的（可以展开成纸）
球面，半径 $R$	两个方向朝同一侧弯曲	$K=+1/R^2 > 0$	穹顶——曲面处处朝你弯曲
鞍面 / 品客薯片	两个方向朝相反方向弯曲	$K<0$	双曲——一个方向向上、另一个向下

符号法则可视化。 在穹顶上，两个切片朝同一方向弯曲（都是 $+$），所以 $K=(+)(+)>0$。在鞍面上，一个切片向上微笑（$+$），垂直方向的切片向下皱眉（$-$），所以 $K=(+)(-)<0$。这个符号的翻转，正是球面几何与双曲几何之间的根本区别。

圆柱面的 $K=0$，尽管它看起来是弯的：你可以把一张平整的纸卷成圆柱而无需拉伸，所以"内蕴"地说它依然是平的。球面的 $K>0$，而你无法在不撕裂的情况下把它压平——这就是为什么每一张世界地图都会扭曲某些东西。

3.2 绝妙定理——"了不起的定理"

高斯绝妙定理（1827）。 高斯曲率 $K$ 是内蕴的：它完全可以由曲面之内所做的测量（微小三角形的边长与角度）算出，而无需参考周围的三维空间。

设想一只生活在曲面上的二维蚂蚁，它能测量距离，却永远无法"抬头"望向它世界之外。绝妙定理说，这只蚂蚁仍然能确定 $K$——办法是检查一个小三角形的内角和是大于、小于还是恰好等于 $180°$，或者一个小圆的周长是小于、大于还是恰好等于 $2\pi r$。曲率从内部就可以被探测到。 这正是让后续一切成为可能的概念性飞跃：我们将要计算概率分布所构成空间的曲率，而这个空间我们显然无法"走到外面"去看。我们所需要的，只是那把内蕴的尺子——度量。

蚂蚁的画圆测试——把"内蕴"变成一次真实的测量。 一只平面上的二维蚂蚁无法"抬头"望向曲面之外，但它仍然能探测到曲率。做法：朝所有方向走出距离 $r$（沿曲面测量 $r$），然后测量你所描出的这个圆的周长 $C$。

曲面	半径为 $r$ 的圆的周长	当 $r=1$ 时	蚂蚁的结论
平面	$C=2\pi r$	$6.283$	平的 ✓
圆柱面（半径 $R$）	$C=2\pi r$（能展平！）	$6.283$	平的——和平面一样！
球面（半径 1）	$C=2\pi\sin r < 2\pi r$	$2\pi\sin 1 = 5.29$	弯曲——有亏损！

球面上的蚂蚁测得 $C=5.29$，而它原本期待的是 $6.283$——这个周长亏损无论怎样"从内部观察"都无法解释，唯一的解释就是曲率。圆柱面上的蚂蚁测得恰好 $2\pi r$，于是正确地得出"平的"结论，尽管对站在外面的我们来说圆柱看起来是弯的。蚂蚁所测得的和我们所看到的之间的这道鸿沟，正是"内蕴 vs 外在"的全部内涵。

为什么地球地图总会撒谎。 圆柱面（$K=0$）用剪刀就能展平——没有任何距离改变。球面（$K=+1/R^2$）则无法在不撕裂或不拉伸的情况下压平，因为根据绝妙定理那将把 $K$ 从 $+$ 变成 $0$，而这是不可能的。每一张世界地图都被迫扭曲某些东西。（这也是为什么把比萨饼对折时它会变硬：把一个方向的曲率捏成非零，就迫使另一个方向变为 $0$ 以保持 $K=0$——这是你可以吃下去的几何学。）

这也是为什么地球的平面地图总会撒谎：压平会把 $K$ 从 $+1/R^2$ 变成 $0$，而根据该定理，这在不扭曲距离的前提下是不可能的。同一个定理也解释了比萨为什么会折弯：把一块比萨捏成 U 形，就在一个方向上迫使 $\kappa_1 \ne 0$，而既然 $K$ 必须保持为 $0$（像纸一样的面团），另一个方向的曲率就被迫为 $0$——这块比萨于是变硬、不再耷拉。你刚刚用绝妙定理吃了顿午餐。

现在我们已经拥有推广所需的两个想法：(a) 几何可以是弯曲的，以及 (b) 曲率是内蕴的，可以从一把局部的尺子推导出来。黎曼的框架把"局部的尺子"精确化，并在任意维度上都成立。

4 · 黎曼几何：统领一切的框架

黎曼 1854 年的演讲回答了一个激进的问题：与其研究嵌在三维空间中的曲面，我们能否抽象地定义一个弯曲空间——仅仅通过指定如何在每一点测量微小距离？答案是肯定的——而这套配方就是黎曼几何的全部。我们分四层来搭建它。

搭建顺序。 你真正需要选择的只有 ① 流形和 ③ 度量。切空间 ② 自动随之而来，而联络 ④——测地线、平行移动与曲率——则由度量唯一确定（黎曼几何基本定理）。所以在实践中，选择一种几何 = 选择一种度量。

4.1 第①层：流形——局部平坦的空间

一个 $n$ 维流形 $M$ 是这样一个空间：在每一点的一个小邻域内，它看起来都像普通的平坦空间 $\RR^n$。你可以铺设局部坐标（一张"图卡"），就像地球上的经纬度一样——但不必有任何单一图卡覆盖一切，而且全局形状可以是弯曲的或在拓扑上非平凡的。

地球表面是一个二维流形：局部上它是一张平坦的地图，全局上它是一个球面。关键词是局部：放大得足够近，曲率就变得不可见，这正是人们曾经以为地球是平的原因。这种"局部平坦"的性质，正是让我们能在弯曲空间上做微积分的根基。

把流形想象成一床由许多平坦坐标图卡拼成的百衲被，彼此光滑地缝合在一起。微积分是逐图卡进行的；几何则关乎这些图卡是如何缝合的。对 AI 而言："数据流形"假设认为自然图像 / 句子 / 蛋白质栖居在一个低维流形上，而这个流形蜷曲在一个巨大的环境空间（像素、token）之内。生成模型试图为这个流形学出一张图卡。

4.2 第②层：切空间——局部的线性世界

在一点 $p\in M$ 处，切空间 $T_pM$ 是所有经过 $p$ 的曲线的速度向量的集合。它是一个平坦的向量空间（$\RR^n$ 的一份副本），粘贴在流形上 $p$ 处——是 $M$ 在 $p$ 附近的最佳线性逼近，就像贴在地球仪上的切平面。

切空间。 每一点都有自己专属的一小块方向的线性空间。关键在于，不同点处的向量栖居在不同的切空间里——你无法直接把北极处的一个向量减去赤道处的一个向量。要比较它们，需要用到平行移动（第④层）。这个"不能相减不同点处的向量"的事实，正是曲率之所以微妙的全部缘由，而它在 AI 中重新现身，成为你不能在弯曲参数空间上简单粗暴地对梯度求平均的原因。

两个平面，两个世界。 弯曲空间的全部微妙之处尽在这一图中：绿色箭头栖居在不同的倾斜平面里，因此不能被直接相加或相减。这正是为什么在 AI 中，你不能天真地对栖居于弯曲模型空间不同点处的梯度 / 参数求平均或相减。

5 · 第③层：度量张量——压轴的主角

这是整篇文档中最重要的单一对象。其余一切都由它构建而成。

一个黎曼度量 $g$ 给每一点 $p$ 指派一个在切空间 $T_pM$ 上光滑变化的内积 $g_p(\cdot,\cdot)$。在坐标下它是一个对称正定矩阵 $\Met(p)=[g_{ij}(p)]$，而一个无穷小步长 $d\vx$ 的长度平方为 $$ ds^2 \;=\; \sum_{i,j} g_{ij}(\vx)\, dx^i\, dx^j \;=\; d\vx^\top \Met(\vx)\, d\vx. $$

把它读作一个依赖于位置的勾股定理。在平坦空间中 $\Met=I$（单位矩阵），你就还原出 $ds^2 = dx^2+dy^2$。让 $\Met$ 逐点变化，你就弯曲了这个空间——而自始至终都没离开过坐标。

每个分量 $g_{ij}$ 究竟意味着什么。 设 $e_1,e_2,\dots$ 是某点处的坐标方向。那么 $g_{ij}=g_p(e_i,e_j)$ 字面上就是方向 $i$ 与方向 $j$ 的内积。于是：

对角元 $g_{ii}=\|e_i\|^2$ 告诉你沿坐标轴 $i$ 走一步实际有多长；
非对角元 $g_{ij}$ 告诉你坐标轴 $i$ 和 $j$ 有多不垂直（为零 ⇒ 垂直）。

而一般的内积不过是这样的记账 $g_p(\vv,\vw)=\sum_{ij}g_{ij}v^iw^j=\vv^\top\Met\,\vw$——这就是为什么内积是一个矩阵。 正定条件（每个非零步长满足 $\vv^\top\Met\vv>0$，即所有特征值为正）只是保证这把尺子永远不会报出零长度或负长度。

$ds^2$ 的真实数字算例（含一个非零的非对角元）。 取 $$\Met=\begin{pmatrix}4&1\\1&9\end{pmatrix},\qquad d\vx=\begin{pmatrix}0.1\\0.2\end{pmatrix}.$$ 双重求和 $\sum_{i,j}g_{ij}\,dx^i dx^j$ 展开成四项；因为 $\Met$ 是对称的（$g_{12}=g_{21}=1$），两个交叉项合并成 $2g_{12}\,dx^1dx^2$： $$ds^2 = \underbrace{4(0.1)^2}_{0.04}+\underbrace{2(1)(0.1)(0.2)}_{0.04}+\underbrace{9(0.2)^2}_{0.36}=0.44,\qquad ds=\sqrt{0.44}\approx0.663.$$ 对比平坦空间（$\Met=I$）：$ds^2=0.1^2+0.2^2=0.05$，$ds\approx0.224$——在这个度量下，同样的坐标步长长了 3 倍。 非对角元 $g_{12}=1$ 正是让坐标轴不垂直的元凶：现在沿 $x$ 移动有一部分被算作沿 $y$ 移动。令 $g_{12}=0$，交叉项就消失了——干净的勾股定理，配上各自独立伸缩的坐标轴。

5.1 度量做三件事

量	由度量给出的公式	含义
向量 $\vv$ 的长度	$\\|\vv\\|_g=\sqrt{\vv^\top \Met\, \vv}$	一步有多"大"——而 $\Met$ 能让同一个坐标步长被算作极大或极小
$\vv,\vw$ 之间的夹角	$\cos\theta=\dfrac{\vv^\top \Met\,\vw}{\\|\vv\\|_g\\|\vw\\|_g}$	量角器
曲线 $\gamma$ 的长度	$L=\displaystyle\int_0^1\sqrt{\dot\gamma^\top \Met(\gamma)\,\dot\gamma}\,dt$	沿路径累加无穷小长度

具体的庞加莱圆盘度量。 它的度量矩阵是对单位矩阵做的一个依赖于位置的重缩放： $$\Met(\vx)=\left(\frac{2}{1-\|\vx\|^2}\right)^{2} I .$$ 在圆心 $\vx=0$ 处：该因子为 $(2/1)^2=4$。在半径 $\|\vx\|=0.9$ 处：该因子为 $\big(2/(1-0.81)\big)^2=\big(2/0.19\big)^2\approx 111$。所以靠近边缘的一个坐标步长所付出的"真实"长度，约是圆心处同样步长的 $\sqrt{111/4}\approx 5\times$ 倍——而且随着 $\|\vx\|\to 1$ 它会一直爆炸式增长。仅这一个标量场，就是双曲空间拥有指数级膨胀空间的全部缘由。 选择这个度量 = 选择双曲几何。

为什么叫"张量"？ 因为无论你用什么坐标（笛卡尔、极坐标，随便什么），算出来的几何长度都必须相同。当你改变坐标时，分量 $g_{ij}$ 会以恰好补偿的方式变换，从而让 $ds^2$ 保持不变。"张量"无非就是让一个量与坐标无关的那条记账规则。这种不变性正是 AI 圈子里赞美自然梯度"重参数化不变"时所指的东西——它是一种基于度量（张量性）的更新，所以它不在乎你是用这种还是那种方式参数化你的 softmax。

给机器学习读者的提醒： 几何张量跟 torch.Tensor 不是一回事。那一堆数字看起来一样，但"几何"这个词加上了一条约定：这些分量必须在坐标变换下做相应变换，从而让物理量（长度、角度）保持不变。torch.Tensor 只是存储；几何张量是存储 + 那条变换规则。

度量不是一个直接的距离函数。度量测量的是无穷小长度（$ds$）；要得到两个相距很远的点之间的距离，你必须找到连接它们的最短曲线，并沿其积分 $ds$。那条最短曲线就是测地线——下一节讲。把局部的尺子和全局的距离搞混，是初学者第一大错误。

6 · 测地线、联络与曲率

6.1 测地线——最直 / 最短的路径

测地线是一条局部长度最小的曲线：直线的推广。等价地，它是一个自由粒子"不转向"地沿之运动的路径——以内蕴方式测量时加速度为零。在坐标下它满足测地线方程 $$ \ddot\gamma^{\,k} + \sum_{i,j}\Gamma^{k}_{ij}\,\dot\gamma^{\,i}\dot\gamma^{\,j}=0, $$ 其中 $\Gamma^{k}_{ij}$ 是由度量的导数构建出来的 Christoffel 符号。

读懂这些符号（记号比想法吓人）：一条测地线是一条路径 $\gamma(t)$。这里的上标是分量标号，不是幂次：$\gamma^k$ 是该路径的第 $k$ 个坐标，$\dot\gamma^i=d\gamma^i/dt$ 是第 $i$ 个速度分量，$\ddot\gamma^k$ 是第 $k$ 个加速度。$\Gamma^k_{ij}$ 是一张数字表（每个三元组 $i,j,k$ 对应一个数），刻画弯曲坐标注入了多少"表观加速度"。

项 $\Gamma^k_{ij}\dot\gamma^i\dot\gamma^j$ 就是曲率施加的"虚拟力"。令 $\Gamma=0$（平坦空间），方程就变成 $\ddot\gamma=0$，即 $\gamma(t)=a+bt$——一条普通的直线。所以测地线是"被度量弯曲了的直线"。在地球仪上它们是大圆；在庞加莱圆盘里它们是那些与边界垂直的弧。

6.2 Christoffel 符号——尺子如何扭转

Christoffel 符号纯粹由度量及其一阶导数算出：

$$\Gamma^{k}_{ij}=\tfrac12\sum_{l} g^{kl}\Big(\partial_i g_{jl}+\partial_j g_{il}-\partial_l g_{ij}\Big),$$

解剖这条吓人的公式（它比看上去小）：

$\Gamma^{k}_{ij}$ —— 虚拟力表：当你沿 $i$ 和 $j$ 移动时，在方向 $k$ 上注入的表观加速度。每个 $(i,j,k)$ 三元组对应一个数。

$g^{kl}$ —— 逆度量（上指标 = 矩阵求逆）。它"除以尺子"，把原始的尺子变化换算回加速度。

$\partial_i g_{jl}$ —— 当你沿方向 $i$ 走一步时，度量分量 $g_{jl}$ 变化得有多快。 这就是整个引擎：如果尺子永不改变，每个 $\partial g=0$，于是 $\Gamma=0$，于是空间平坦。

那个特定的三项组合只是为了让 $\Gamma$ 关于 $i,j$ 对称且与度量相容——你不必去体会它，只需看出它仅仅由 $g$ 的一阶导数构建而成。

用一句话说： $\Gamma$ = (1 / 尺子) × (尺子的变化率)。尺子恒定 ⇒ $\Gamma=0$ ⇒ 直线保持笔直。

其中 $g^{kl}$ 是逆度量矩阵。别去背它——只要吸收这条信息：关于如何"笔直"移动的一切，都编码在度量从一点到另一点的变化方式之中。 如果 $\Met$ 是常数，所有导数消失，$\Gamma=0$，空间平坦。曲率就是度量未能保持恒定。

你其实已经懂了一半的算例：极坐标下的平坦平面。 用 $(r,\theta)$ 代替 $(x,y)$。度量为 $$\Met=\begin{pmatrix}1&0\\0&r^2\end{pmatrix}\quad\text{即}\quad ds^2=dr^2+r^2\,d\theta^2.$$ 它不是常数（依赖于 $r$），所以即便平面完全平坦，坐标也会扭转。代入 Christoffel 公式，只有两个符号幸存： $$\Gamma^r_{\theta\theta}=-r,\qquad \Gamma^\theta_{r\theta}=\Gamma^\theta_{\theta r}=\tfrac1r,$$ 测地线方程变成 $$\ddot r - r\dot\theta^2 = 0,\qquad \ddot\theta+\tfrac2r\dot r\dot\theta=0.$$ $\ddot r = r\dot\theta^2$ 你以前见过——它正是大一力学里的离心项。所以 Christoffel 符号无非就是你使用弯曲坐标时出现的那些"虚拟力"，并不可怕，而且这里空间仍然是平坦的（一条直线只是在极坐标里看起来弯曲）。一个真正弯曲的空间——球面——会多出一个 $\sin\theta\cos\theta$ 项，而任何坐标变换都无法消除它。那个无法消除的部分，才是真正的曲率。

在一个真正弯曲的空间上运转这套机器：大圆就是测地线。 我们在 §2.2 断言过这一点；这里是它唯一被真正证明的地方。单位球面的度量为 $ds^2=d\theta^2+\sin^2\theta\,d\varphi^2$（$\theta$ = 余纬，$\varphi$ = 经度）。其幸存的 Christoffel 符号是 $\Gamma^\theta_{\varphi\varphi}=-\sin\theta\cos\theta$ 和 $\Gamma^\varphi_{\theta\varphi}=\cot\theta$。检验赤道 $\theta=\tfrac\pi2$（常数），以匀速 $\dot\varphi$ 行进：$\theta$ 方程为 $$\ddot\theta-\sin\theta\cos\theta\,\dot\varphi^2 = 0-\underbrace{\sin\tfrac\pi2\cos\tfrac\pi2}_{=\,0}\dot\varphi^2=0.\ \checkmark$$ 所以赤道满足测地线方程——它是一条测地线。其余每一条大圆都只是赤道的一个旋转，而旋转是等距变换（它们保持度量不变），所以所有大圆都是测地线。此外还有一个三行的非微积分证明：任意两点都位于一个过球心的平面内；关于该平面的反射是一个对称变换，它固定这两点，并恰好固定那条大圆弧，所以最短路径（局部唯一）必定是它所固定的那部分——大圆。

平行移动与和乐。 联络（由 $\Gamma$ 编码）定义了如何"不转向"地移动一个切向量。在弯曲空间上把一个向量绕闭合回路移动一圈，它回来时已被旋转——那个亏角度量了曲率。黎曼曲率张量 $R^{l}_{ijk}$ 正是这个"每单位回路面积的旋转量"，它由 $\Gamma$ 及其导数构建而成（因而来自 $g$ 的二阶导数）。

回路大小（经度跨度）：

观察和乐的累积。 按下播放：一个橙色箭头被绕一个球面三角形携带（极点 → 赤道 → 沿赤道 → 返回），始终保持"尽可能平行"。一个淡淡的残影固定钉在起点的朝向上。当箭头返回时，它明显被旋转了——而读数显示这个旋转恰好等于所包围的面积：在单位球面上 和乐 $=\Delta\lambda = K\times\text{Area}$。拖动滑块以包围更多面积，亏角随之增大。这个每回路的旋转量就是黎曼曲率张量的可操作定义。

6.3 曲率阶梯与 exp / log 映射

完整的黎曼张量 $R^l_{ijk}$ 是个庞然大物，但它的四个指标有一种具体的可操作读法：$i,j$ 选定张成你所移动的那个小回路的两个方向；$k$ 是你携带的箭头的方向；$l$ 是返回时（被旋转的）箭头的分量。 所以 $R^l_{ijk}$ = "当一个指向 $k$ 的箭头被绕一个 $i$–$j$ 回路携带时，它向 $l$ 方向倾斜了多少"。四个指标 = (回路平面：2) + (输入方向：1) + (输出分量：1)。它被缩并（求和约化）成更友好的平均量：

对象	它是什么	一句话含义
黎曼张量 $R^l_{ijk}$	完整曲率，所有方向	每单位回路面积的和乐
截面曲率 $K(\vv,\vw)$	某个二维切片的曲率	该切片的高斯曲率 $K$
Ricci 曲率 $\mathrm{Ric}_{ij}$	对方向求平均	一个小球的体积比欧几里得的多还是少？（驱动广义相对论与"Ricci 流"）
标量曲率 $S$	每点一个数	局部总曲率

最后，有两个映射让我们能在弯曲流形和它的平坦切空间之间往返跳跃——它们是每一个几何深度学习库的主力：

指数映射 $\exp_p(\vv)$：从 $p$ 出发，沿方向 $\vv\in T_pM$ 的测地线射出单位时间，并报告你落在 $M$ 上的何处。它把一个平坦的切向量变成弯曲流形上的一个点。

对数映射 $\log_p(q)$：其逆——$p$ 处指向 $q$ 的那个切向量，其长度等于测地线距离。它把流形上的一个点变成一个平坦的切向量。

$\exp$ 和 $\log$ 正是实践者在弯曲空间上"伪装"做线性代数的方式：用 $\log$ 摊平到一个切空间，做你的欧几里得运算（相加、求平均、走一步梯度），然后用 $\exp$ 回到流形上。双曲神经网络（§12）字面上就是把每一层都裹在 $\log/\exp$ 里；黎曼优化器把 "$\theta \leftarrow \theta - \eta\,\vg$" 替换为 "$\theta\leftarrow \exp_\theta(-\eta\,\vg)$"。掌握这两个映射，大多数"几何深度学习"代码就不再神秘了。

这一来回，作为你能跑起来的代码：在弯曲流形上对两个点求平均。 你不能用 $\tfrac12(A+B)$ 对两个曲面上的点求平均——那个中点会飘离曲面（见图）。改为：

# Riemannian mean of points A, B, around a base point p
a     = log(p, A)        # log_p：流形点 -> T_pM 中的平坦切向量
b     = log(p, B)        # A 和 B 现在都是同一个平坦空间里的普通向量
m_tan = 0.5*(a + b)      # 普通的欧几里得平均——在这里完全合法
M     = exp(p, m_tan)    # exp_p：切向量 -> 回到流形上

类型： $\log_p:\,M\to T_pM$（点 → 平坦向量），$\exp_p:\,T_pM\to M$（平坦向量 → 点），并且它们互为逆：$\exp_p(\log_p(q))=q$。所以在 $\theta\leftarrow\exp_\theta(-\eta\vg)$ 中，对象 $-\eta\vg$ 是一个切向量，而它的 $\exp_\theta$ 是一个新的流形上的点。

这套机器为何存在。 两个流形点的朴素欧几里得中点通常不在流形上；而 $\log$→平均→$\exp$ 这套配方让每个结果都留在它该在的地方。

现在我们拥有了弯曲空间的全套机器：流形、切空间、度量、测地线、曲率、exp/log。接下来两节的魔力在于，概率分布构成的空间暗地里就是这些流形之一——而它的度量有一个你已经懂了一半的名字：Fisher 信息。

7 · Fisher 信息：给概率配上一把尺子

切换一下场景。暂时把曲面放到一边。考虑一个统计模型：一族由参数 $\vth$ 索引的概率分布 $p(x\mid\vth)$。例如：所有高斯分布（参数 = 均值和方差）、所有硬币（参数 = 偏置 $\theta$），或者——AI 的情形——当你改变神经网络的权重 $\vth$ 时它能产生的所有下一个 token 分布。

Fisher 在 1925 年提出的问题是：数据能告诉我们多少关于参数的信息？答案是一个矩阵，而这个矩阵最终会成为一个度量张量。

7.1 得分函数：似然有多敏感？

得分函数是对数似然关于参数的梯度： $$ s(\vth)=\grad_{\vth}\log p(x\mid\vth). $$ 我们使用对数是出于机器学习里常见的理由：它把独立数据点上的乘积变成了求和，于是整个数据集的得分函数就是每个样本得分函数之和（而它的离散程度将成为 Fisher 矩阵）。它指向参数空间中能最大程度提高你所观测数据概率的方向。一个关键事实：在模型自身生成的数据上取平均，得分函数为零，即 $\EE_{x\sim p_\vth}[s(\vth)]=0$。

具体算一算硬币的得分函数（以及为什么"均值为零"不是一句口号）。由 $\log p(x\mid\theta)=x\log\theta+(1-x)\log(1-\theta)$，对 $\theta$ 求导： $$ s(\theta)=\frac{x}{\theta}-\frac{1-x}{1-\theta}. $$

正面（$x=1$）：$s=+1/\theta$——为正，"数据想要一个更大的 $\theta$。"
反面（$x=0$）：$s=-1/(1-\theta)$——为负，"数据想要一个更小的 $\theta$。"

现在在硬币实际产生的数据上取平均（正面概率为 $\theta$，反面概率为 $1-\theta$）： $$\EE[s]=\theta\cdot\frac{1}{\theta}+(1-\theta)\cdot\frac{-1}{1-\theta}=1-1=0.$$ 这是作为算术结果的均值为零，而非一句口号：正面把 $\theta$ 往上拉，反面把它往下拉，如果硬币真的有偏置 $\theta$，这两股拉力会恰好抵消。只有当数据来自不同的 $\theta$ 时它们才无法抵消——而这恰恰是学习获得信号的方式。

7.2 Fisher 信息 = 得分函数的方差

Fisher 信息矩阵是得分函数的协方差（等价地，是对数似然期望曲率的相反数）： $$ \Fish(\vth)\;=\;\EE_{x\sim p_\vth}\!\big[\,s(\vth)\,s(\vth)^\top\big]\;=\;-\,\EE_{x\sim p_\vth}\!\big[\grad^2_\vth \log p(x\mid\vth)\big]. $$ 在温和的正则性条件下这两种形式相等（"Fisher 恒等式"）。

为什么这两种形式一致（一行话，复用"得分函数均值为零"）。从 $\EE[s]=\int p\cdot\frac{\grad p}{p}=\grad\!\int p=\grad 1=0$ 出发。把这个恒等式再对 $\theta$ 求一次导；乘积法则恰好把它拆成 $$\EE[\grad^2\log p]+\EE[(\grad\log p)(\grad\log p)^\top]=0,\quad\text{即}\quad -\EE[\text{Hessian}]=\EE[ss^\top].$$ 所以"得分函数的方差 = 期望曲率的相反数"无非就是把"均值得分 = 0"再求一次导。一个方差（一阶导数，恒 $\ge0$）和一个曲率（二阶导数）原来是同一个矩阵。

从直觉上理解这两种形式：

作为得分函数方差（第一种形式）：如果得分函数随数据变化而剧烈跳动，那么数据对 $\vth$ 就非常有信息量——$\vth$ 的微小变化会明显改变哪些数据更可能出现。得分函数方差高 ⇒ 信息量高。
作为对数似然的负曲率（第二种形式）：一个尖锐峰化的对数似然（二阶导数大）意味着数据把 $\vth$ 钉得很紧。似然平坦 ⇒ 数据几乎不约束 $\vth$ ⇒ 信息量低。

硬币的 Fisher 信息。一枚硬币以概率 $\theta$ 落为正面。观察一次抛掷 $x\in\{0,1\}$。则 $\log p = x\log\theta+(1-x)\log(1-\theta)$，简短计算给出（标量的）Fisher 信息 $$ \Fish(\theta)=\frac{1}{\theta(1-\theta)}. $$ 在 $\theta=0.5$（公平硬币）时：$\Fish = 1/0.25 = 4$——信息量最小；一枚 50/50 的硬币，其抛掷结果对 $\theta$ 的微小变化最为模糊。在 $\theta=0.99$ 时：$\Fish=1/(0.99\cdot0.01)\approx 101$——信息量巨大；一枚近乎确定的硬币能很快暴露其偏置，因为哪怕只看到一次反面都非常意外，会强烈地更新 $\theta$。硬币的参数空间在 0 和 1 两端被"拉伸"、在中间被"压缩"——它是弯曲的，而 $\Fish(\theta)$ 就是它的度量。

7.3 Cramér–Rao 下界——估计的速度极限

对于由 $n$ 个 i.i.d. 样本得到的 $\vth$ 的任意无偏估计量 $\hat\vth$， $$ \mathrm{Cov}(\hat\vth)\;\underbrace{\succeq}_{\text{matrix }\ge}\;\frac{1}{n}\,\Fish(\vth)^{-1}. $$ 符号 $\succeq$ 是"$\ge$"的矩阵版本（差为半正定）。换句话说：你对一个参数的估计精度不可能超过逆 Fisher 信息所允许的程度——Fisher 信息高 ⇒ 下界低 ⇒ 可被精确学习。

这就是 Fisher 信息配得上"信息"这个名字的原因：它实实在在地是统计精度的通货。Fisher 信息越多 ⇒ 可达到的方差越小 ⇒ 参数越可学习。最大似然估计量在渐近意义下达到这个下界，这正是 MLE（也因此是深度学习大多数损失函数）之所以合理的深层原因。

从几何上看，Cramér–Rao 下界说的是：那些分布容易区分的方向（高 Fisher）是你可以精确估计的方向；那些分布看起来几乎相同的方向（低 Fisher）是你注定只能含糊其辞的方向。Fisher 矩阵是"参数距离"与"统计可区分性"之间的汇率。而这个汇率恰恰就是度量张量。记住这个念头——§8 会让它变得名副其实。

8 · 信息几何：宏大的统一

这就是整篇文档一直在铺垫的那个点睛之笔。

核心定理（一口气说完）

一个模型中所有分布构成的集合 $\{p(x\mid\vth)\}$ 是一个 黎曼流形 —— 点是分布，坐标是参数 $\vth$ —— 而它的 度量张量就是 Fisher 信息： $$ \boxed{\;g_{ij}(\vth)=\Fish_{ij}(\vth)\;}\qquad\text{即 \emph{Fisher–Rao 度量}。} $$ 这个流形上的距离 = 统计可区分性。直线、曲率、勾股定理统统重新登场 —— 只不过披上了 KL 散度、指数族和自然梯度的外衣。

为什么它和曲面度量是同一类对象？把这两个事实并排放在一起看：

$$\underbrace{ds^2=\delta^\top \Met\,\delta}_{\text{弯曲曲面上的长度（§5）}}\qquad\Longleftrightarrow\qquad \underbrace{2\,\KL\approx\delta^\top \Fish\,\delta}_{\text{邻近模型的可区分性（§8.1）}}$$

度量就是任何把一个微小坐标步长变成平方长度的东西。在曲面上这个长度用米来量；在统计流形上它用「有多可区分」来量，途径就是 KL。同一台机器，不同的尺子 —— 所以把 $\Fish$ 称作度量并不是类比，它确确实实满足 $g_{ij}$ 的定义。

这座桥 —— 全文最重要的一张图。 左边是一张弯曲曲面，度量 $g$ 使橙色步长「长」、蓝色步长「短」。右边是同一块抽象的区域，但现在每个点都实实在在是一个概率分布（一条钟形曲线）。橙色邻居的分布与中心截然不同（容易区分 ⇒ 「远」 ⇒ 高 Fisher）；蓝色邻居的分布几乎一模一样（难以区分 ⇒ 「近」 ⇒ 低 Fisher）。同一个步长 $\delta$，同一个矩阵在做重新缩放 —— 这正是 $g_{ij}=\Fish_{ij}$ 的原因。一旦你看清这些点就是分布，「分布的空间是一个弯曲流形」就不再只是一句口号。

8.1 为什么 Fisher 是那个对的度量？KL 散度暗地里是二次的

KL 散度 —— 先给定义，因为下面（以及整个 §9–§10）都要用到它。 给定同一组结果 $x$ 上的两个分布 $p$ 和 $q$， $$\KL(p\,\|\,q)=\EE_{x\sim p}\!\left[\log\frac{p(x)}{q(x)}\right]=\sum_x p(x)\log\frac{p(x)}{q(x)}.$$ 用大白话说： 平均对数似然比 —— 如果你相信的是 $q$ 而数据其实来自 $p$，你平均会有多惊讶（等价地，用为 $q$ 设计的编码去压缩 $p$ 的数据所浪费的额外比特）。记住三个事实：

$\KL(p\|q)\ge0$ 恒成立，且 $=0$ 当且仅当 $p=q$ —— 所以它的行为像一个从 $p$ 到 $q$ 的「距离平方」。
它不对称：一般有 $\KL(p\|q)\ne\KL(q\|p)$。这就是为什么它是散度，而非真正的距离。
当 $p,q$ 难以分辨时它很小；当一方在另一方几乎不放概率的地方放了概率时它很大。

抛硬币验算： $p=(\text{正面 }0.5)$，$q=(\text{正面 }0.9)$。则 $\KL(p\|q)=0.5\log\frac{0.5}{0.9}+0.5\log\frac{0.5}{0.1}\approx0.51$；交换后 $\KL(q\|p)\approx0.37$ —— 数值不同，证实了非对称性。（单位：$\log_2\!\to$ 比特，$\ln\!\to$ 奈特。）

Kullback–Leibler 散度 $\KL(p_\vth\,\|\,p_{\vth+\delta})$ 衡量两个邻近分布有多可区分。对一个微小步长 $\delta$ 把它泰勒展开。常数项为 0（一个分布与自身完全相同），线性项也为 0（KL 在 $\delta=0$ 处取最小，所以它的梯度在那里消失），第一个存活下来的项是二次的：

$$ \KL\big(p_\vth \,\big\|\, p_{\vth+\delta}\big)\;=\;\tfrac12\,\delta^\top\,\Fish(\vth)\,\delta\;+\;O(\|\delta\|^3). $$

看 Fisher 是怎么「蹦出来」的 —— 两枚邻近硬币之间的 KL。 硬币 $\theta$ 对硬币 $\theta+\delta$： $$\KL=\theta\log\frac{\theta}{\theta+\delta}+(1-\theta)\log\frac{1-\theta}{1-(\theta+\delta)}.$$ 把对数按 $\delta$ 泰勒展开。$\delta^0$ 项相消。$\delta^1$ 项也相消 —— 它们求和为 $\big(\tfrac{-\theta}{\theta}+\tfrac{1-\theta}{1-\theta}\big)\delta=0$，这正是得分函数均值为零（§7.1），如今表现为线性项消亡。$\delta^2$ 项存活下来： $$\KL\approx\tfrac12\Big(\tfrac1\theta+\tfrac{1}{1-\theta}\Big)\delta^2=\tfrac12\cdot\frac{1}{\theta(1-\theta)}\,\delta^2.$$ 那个系数 $\frac{1}{\theta(1-\theta)}$ 恰好就是 §7.2 中硬币的 Fisher 信息 —— 同一个数，用两条截然不同的路径得到。在 $\theta=0.5,\ \delta=0.01$ 处数值上：$\KL\approx\tfrac12\cdot4\cdot0.0001=2.0\times10^{-4}$，而精确 KL 是 $2.0\times10^{-4}$。两者吻合。这一个例子把 §7.1（得分均值为零 ⇒ 线性项消亡）、§7.2（硬币的 Fisher）和 §8.1（二次系数）串成了一条算术链条。

现在来看矩阵版本 —— Fisher 矩阵 $\Fish=\diag(1/\sigma^2,\,2/\sigma^2)$ 究竟从何而来。 硬币是标量的；这里两个参数 $(\mu,\sigma)$ 表明非对角项也会消亡。两个高斯分布之间的精确 KL 有闭式解（记 $\sigma'=\sigma+\delta\sigma$）： $$\KL\big(\mathcal N(\mu,\sigma^2)\,\big\|\,\mathcal N(\mu+\delta\mu,\sigma'^2)\big)=\log\frac{\sigma'}{\sigma}+\frac{\sigma^2+\delta\mu^2}{2\sigma'^2}-\tfrac12.$$ 按 $(\delta\mu,\delta\sigma)$ 展开到二阶：

$\delta\mu$ 只以 $\dfrac{\delta\mu^2}{2\sigma^2}$ 的形式出现 → 系数 $1/\sigma^2 = \Fish_{\mu\mu}$。
$\delta\sigma^0$ 和 $\delta\sigma^1$ 项相消（KL 在 $\delta\sigma=0$ 处取最小），留下 $\dfrac{\delta\sigma^2}{\sigma^2}$ → $\Fish_{\sigma\sigma}=2/\sigma^2$。
交叉项 $\delta\mu\,\delta\sigma$ 消失 → $\Fish_{\mu\sigma}=0$。

$$\Rightarrow\quad \KL\approx\tfrac12\Big(\tfrac{\delta\mu^2}{\sigma^2}+\tfrac{2\,\delta\sigma^2}{\sigma^2}\Big)=\tfrac12\,\delta^\top\!\diag\!\big(\tfrac1{\sigma^2},\tfrac2{\sigma^2}\big)\delta.$$ 这恰好就是 §8.2 椭圆演示所绘制的那个矩阵 —— 而非对角项为零正是为什么那个椭圆保持轴对齐。数值验算：$\sigma=1,\delta\mu=0.1,\delta\sigma=0$ 给出 $\tfrac12(0.01)=0.005$，与精确 KL $\approx0.005$ 吻合。

硬币偏向 θ:

二次近似在哪里成立 —— 又在哪里失效。 红色曲线是随步长 $\delta$（横轴）变化的精确 $\KL(\theta\,\|\,\theta+\delta)$；蓝色虚线抛物线是 Fisher 近似 $\tfrac12\frac{1}{\theta(1-\theta)}\delta^2$。它们在 $\delta=0$ 处相切（Fisher = KL 的局部曲率），并随着 $|\delta|$ 增大而分离 —— 那道缝隙就是 $O(\delta^3)$ 截断项。拖动 $\theta$ 时要注意两件事：当 $\theta\to0$ 或 $1$ 时抛物线变陡（高 Fisher = 这枚硬币容易被钉死），而精确的红色 KL 明显是偏斜的（非对称），蓝色度量却始终对称 —— 这正是 KL 是「散度」而非真正距离的精确含义。

所以 Fisher 信息就是 KL 散度在零间隔处的二阶导数（Hessian）。KL 的行为像二分之一个平方距离，而定义那个平方距离的矩阵恰好就是 $\Fish$。这是「Fisher = 分布空间上的局部尺子」的严格版本，也是驱动 TRPO、自然梯度、EWC 以及 RLHF 中 KL 惩罚项的那个方程 —— 全都在 §10–§11。

Chentsov 唯一性定理给这件事盖棺定论：Fisher–Rao 度量在统计流形上，至多相差一个整体缩放，是唯一在「充分统计量」下不变的黎曼度量 —— 也就是任何不损失信息的数据再加工。换句话说，它是唯一不在乎你如何编码或重参数化数据与模型的几何。这个不变性正是 AI 研究者说自然梯度「重参数化不变」时所珍视的性质。这套几何是自然强加给我们的，不是我们挑选出来的。

8.2 可视化 Fisher 度量：可区分性椭圆

族均值 μ: 标准差 σ:

高斯族的 Fisher 度量 $\mathcal N(\mu,\sigma^2)$，以 $(\mu,\sigma)$ 参数化。它的 Fisher 矩阵是 $\Fish=\diag(1/\sigma^2,\,2/\sigma^2)$。我们画出等 KL 椭圆 $\{\delta:\delta^\top\Fish\delta=c\}$ —— 即与中心模型同等可区分的所有参数步长的集合。它的 $\mu$ 半轴是 $\sqrt{c}\,\sigma$，所以椭圆随着 $\sigma$ 增大而增大：当一个高斯已经又宽又模糊（大 $\sigma$）时，你必须把 $\mu$ 或 $\sigma$ 移动很多才能产生统计上可察觉的变化 —— 在 $\sigma=2$ 时，要达到相同的 KL，你必须把 $\mu$ 移动到 $\sigma=0.5$ 时的四倍远。当 $\sigma$ 很小（一个尖峰）时，对 $\mu$ 轻轻一推就立刻可被察觉 —— 椭圆很紧，空间被「拉伸」了。这种各向异性、随位置变化的拉伸正是自然梯度的全部要义：普通梯度下降无视它，于是迈出大小失当的步子。

8.3 对偶平坦空间、指数族 & 广义勾股定理

本小节是全文最抽象的部分。初读时你可以直接跳到 §9，从操作层面不会损失任何东西 —— 等你好奇为什么 KL 的行为像平方距离时再回来。

信息几何还有一份礼物，它正是 KL「感觉像」平方距离的结构性原因。Amari 证明，统计流形携带的不是一个、而是一对对偶平坦联络 —— e-联络（指数/自然坐标）和 m-联络（混合/期望坐标）—— 由一个勒让德变换相连。在两者下都平坦的流形称为对偶平坦，而指数族（高斯、伯努利、类别/softmax、泊松……）恰好就是那些对偶平坦的族。

在对偶平坦流形上，典范散度是某个凸势函数的 Bregman 散度，而对概率分布而言这个 Bregman 散度就是 KL 散度。一个广义勾股定理成立： $$ \KL(p\,\|\,r)=\KL(p\,\|\,q)+\KL(q\,\|\,r) $$ 只要 m-测地线 $p\!\to\!q$ 与 e-测地线 $q\!\to\!r$ 以「直角」相交（在 Fisher 度量下）。

广义勾股定理。 极大似然、EM 算法、变分推断和信息投影的整座大厦都建立在此之上：把 KL 最小化到一个指数族上，就是 Fisher 几何中的一次正交投影，并对「误差」做出勾股式分解。当你拟合一个 softmax、运行 EM 或做变分推断时，不管你知不知道，你都在一个对偶平坦流形上做几何。

衡量两个分布之间距离的三种方式

本文档在不同章节定义了这三者；这里说明你为何会用到其中的每一个。它们是同一个分布空间上的三把尺子。

	KL 散度 $\KL(p\\|q)$	Wasserstein $W_2$（推土机）	Fisher–Rao
衡量什么	用 $q$ 的编码去编 $p$ 所需的额外比特	最廉价的质量搬运（质量 × 距离）	邻近模型的无穷小可区分性
对称吗？	否（$\KL(p\\|q)\ne\KL(q\\|p)$）	是（真正的度量）	是（真正的黎曼距离）
三角不等式？	否	是	是
支撑集不重叠时仍有限吗？	否（→ ∞）	是（保持有限）	不适用（局部的）
与其他两者的关系	当 $\delta$ 小时 $\approx\tfrac12\delta^\top\Fish\delta$	Otto 度量下的测地距离	度量张量 = KL 在 $\delta\!=\!0$ 处的 Hessian
尺度	全局（比特/奈特）	全局（地面距离 × 质量）	局部 / 无穷小
在 AI 中的出场	TRPO/PPO/GRPO、VAE ELBO、RLHF	WGAN、OT 流匹配、扩散	自然梯度、EWC、Laplace

KL 是局部视角（§8），Fisher 是它的二阶导数，Wasserstein 是传输视角（§13）—— 关于「两个分布相距多远？」的三种视角。

9 · 自然梯度：几何与优化的交汇

现在几何作为一种算法开始回报我们了。这一个核心思想，正是后续大多数 AI 应用赖以分叉生长的主干。

9.1 普通梯度的问题

梯度下降执行 $\vth \leftarrow \vth - \eta\,\grad L$。但 $\grad L$ 是最速下降方向，仅当你用朴素的欧几里得尺子 $\|\delta\vth\|^2$ 来度量距离时才成立。在统计流形上这把尺子是错的——衡量“模型移动了多远”的正确概念是 Fisher 长度 $\delta\vth^\top \Fish\,\delta\vth$（≈ KL 变化），而不是参数空间的长度。用错误的尺子意味着梯度指向一个被扭曲的方向，而且各坐标方向上的步长彼此不一致。

自然梯度（Amari, 1998）是在 Fisher 度量下度量的最速下降方向： $$ \tilde\grad L \;=\; \Fish(\vth)^{-1}\,\grad L,\qquad \vth \leftarrow \vth - \eta\,\Fish(\vth)^{-1}\grad L. $$ 它就是被逆 Fisher 信息预条件化（pre-conditioned）后的梯度。

$\Fish^{-1}$ 是从哪来的？两行推导出来。别把这个公式当作信条照单全收——它其实是从一个优化问题中自然落出来的。“最速下降”真正的含义是：在移动距离固定预算的约束下，取使损失下降最多的那一步 $\delta$。用 KL（分布上的自然距离）来度量“移动多远”，它 $\approx\tfrac12\delta^\top\Fish\delta$（§8.1）。于是： $$\min_{\delta}\ \grad L^\top\delta \quad\text{subject to}\quad \tfrac12\delta^\top\Fish\delta\le\varepsilon.$$ 构造拉格朗日量 $\mathcal L=\grad L^\top\delta+\lambda\big(\tfrac12\delta^\top\Fish\delta-\varepsilon\big)$ 并令 $\partial\mathcal L/\partial\delta=0$： $$\grad L+\lambda\Fish\delta=0\ \Longrightarrow\ \delta=-\tfrac1\lambda\,\Fish^{-1}\grad L.$$ 最优方向就是 $-\Fish^{-1}\grad L$——即自然梯度——而 $\lambda$（学习率）只是把步长调到刚好填满 KL 预算。所以“在 Fisher 度量下的最速下降”和“在 KL 信赖域内取得最大损失下降”字面上就是同一个优化问题。求解这个预算给出 $\eta=\sqrt{2\varepsilon/(\grad L^\top\Fish^{-1}\grad L)}$——这恰恰就是 TRPO 的步长（§10）。下面那两项“超能力”如今成了推论，而非空口断言：

两项超能力立刻随之而来：

重参数化不变性。因为 $\Fish$ 像度量张量那样变换，更新量 $\Fish^{-1}\grad L$ 无论你怎样参数化模型，都会在底层分布上产生相同的变化。普通 SGD 做不到这一点——重新缩放你的权重，SGD 就会走一条不同的路；而自然梯度走的还是同一条。
它是 KL 中的信赖域。取一个固定大小的自然梯度步，等价于“在输出分布的变化不超过固定 KL 量的前提下，尽可能多地改进损失”。这正是 TRPO 背后的原理（§10）。

9.2 直观看：在被拉伸的地形上比较自然梯度与普通梯度

各向异性（度量被拉伸的程度）：

普通梯度（橙色）与自然梯度（蓝色）同向一个病态二次型的最小值（★）下降——这个二次型代表一个其 Fisher 度量把某一方向拉伸得远比另一方向厉害的损失。普通梯度垂直于等高线，在狭长的山谷里来回锯齿状摆动，浪费步数。自然梯度先左乘 $\Fish^{-1}$，把空间“拉回不被拉伸”的状态，于是它几乎笔直指向最小值，所需步数大大减少。这幅图本质上就是为什么二阶 / 曲率感知优化器（K-FAC、Shampoo、自然策略梯度）在病态问题上胜过朴素 SGD——把滑块往上拖，可以夸大这一效果。这里被拉伸的等高线，与你在 §8.2 遇到的可区分性椭圆是同一个各向异性——那个椭圆就是 $\delta^\top\Fish\delta$ 的一条等值集，而 $\Fish^{-1}$ 正是把它变回圆的那个变换。

为什么 $\Fish^{-1}$ 能“拉回不被拉伸”。$\Fish$ 拉伸空间；$\Fish^{-1}$ 字面上就是那个拉伸的逆。自然梯度 =（1）暂时变换到度量恰好是一个普通圆的坐标系（乘以 $\Fish^{-1/2}$），（2）走那一步显而易见的步——在圆整的空间里，最速下降确实指向最小值——（3）再把方向映射回去。这一复合恰好就是 $\Fish^{-1}\grad L$。

症结所在——也是下文半数 AI 论文之所以存在的原因——是 $\Fish$ 是一个 $d\times d$ 矩阵，其中 $d$ = 参数个数。对于一个现代网络，$d$ 是数十亿量级，所以构造、存储或求逆 $\Fish$ 都彻底不可能。每一种实用的“自然梯度”方法，实际上都是对 $\Fish^{-1}$ 的某种巧妙近似：对角化（类 Adam）、块对角的 Kronecker 因子分解（K-FAC）、低秩，或者在根本不显式构造出 $\Fish$ 的情况下求解一个 KL 信赖域（TRPO 的共轭梯度法）。在我们巡览这些应用时，请记住这一点。

整篇文档浓缩在一页上。你真正要选择的，永远只有 ① 度量。第 ② 列中的一切——测地线、曲率、指数/对数映射、分布上的 Fisher 度量、Wasserstein 度量——都是从它推导出来的。而第 ③ 列中的每一种 AI 方法（§10–§13）都挂靠在这些推导出的对象之一上。“选择一种几何 = 选择一个度量”，正是这张图所编码的那一句话。

至此，数学核心部分就完成了。从这里开始，每一节都是同样的四个思想——度量、测地线、曲率、自然梯度——只是披上某种当下最先进 AI 方法的外衣。每个关联框都会点出确切的论文（已核实，附 arXiv ID），方便你深入研读。

10 · AI I —— 自然梯度、K-FAC 与信赖域 RL

最直接的应用：用 Fisher 度量来优化。这条线索从 1998 年的一个定理一路通向当今每个推理型 LLM（DeepSeek-R1 等）的训练方式。

奠基性结果 —— 自然梯度

Amari, "Natural Gradient Works Efficiently in Learning" 1998Neural Computation 10(2)very-high >5000

证明了：当参数生活在一个统计（黎曼）流形上时，最速下降方向是 $\Fish^{-1}\grad L$，而不是 $\grad L$；并且在线自然梯度学习是渐近 Fisher 高效的（它达到了 Cramér–Rao 下界 —— 你不可能学得更快）。这就是把 §9 写成了一个定理，也是下面一切内容的概念性根源。

让它可计算 —— K-FAC 与现代优化器全家桶

Martens & Grosse, "Optimizing Neural Networks with Kronecker-factored Approximate Curvature" (K-FAC) 2015ICMLhigh >1000 · arXiv:1503.05671

一个深度网络的 Fisher 是数十亿 × 数十亿 —— 无法求逆。K-FAC 的洞见是：每一层的 Fisher 块都分解为两个小矩阵的 Kronecker 积 $\Fish_\ell \approx A_\ell \otimes G_\ell$（一个来自输入激活，一个来自反向传播的输出梯度）。由于 $(A\otimes G)^{-1}=A^{-1}\otimes G^{-1}$，自然梯度步就变得很便宜了。

K-FAC 如何让自然梯度可运行。 Kronecker 积从两个极小矩阵构建出一个巨大的结构化矩阵 —— 而它的逆就只是那两个小矩阵的逆。这就是把 Amari 的定理变成你真正能拿来训练的东西的全部诀窍。

这是自然梯度的经典实用方法，它（作为曲率引擎）会在 §11 的 Laplace 近似和影响函数中再次出现。

同一个"廉价地近似曲率/度量"的思想，驱动了 2018–2026 年间的一系列优化器，每个都选择了不同的几何：Shampoo（Gupta et al. 2018, 1802.09568）使用逐维度的 Kronecker 预条件子；distributed Shampoo（Anil et al. 2020, 2002.09018；Shi et al. 2023, 2309.06497）把它扩展到生产规模；Sophia（Liu et al. 2023, 2305.14342）使用一个廉价的对角 Hessian 估计；SOAP（Vyas et al. 2024, 2409.11321）证明了在预条件子的特征基下 Shampoo ≈ Adam；而 Muon（Jordan et al. 2024；"Muon is Scalable," Liu et al. 2025, 2502.16982）对更新做正交化 —— Bernstein & Newhouse（2409.20325）证明这是谱范数下的最速下降，一种不同于 Fisher 的度量，但秉持同样的"选择你的几何"哲学。

优化器	假设的几何 / 它近似的对象	曲率结构	开销
SGD	欧几里得（度量 $=I$）	无	最便宜
Adam	对角（经验）Fisher —— 逐坐标重缩放	对角	便宜
K-FAC	块对角 Fisher，每层用 Kronecker $A\otimes G$	块 + Kronecker	中等
Shampoo	逐维度 Kronecker 预条件子（≈ 完整高斯-牛顿）	Kronecker，双侧	中-高
SOAP	Adam 特征基下的 Shampoo（更便宜，≈ Shampoo）	Kronecker 特征基	中等
Sophia	对角 Hessian 估计（裁剪过的）	对角 Hessian	便宜
Muon	谱范数最速下降（正交化的步）	谱范数，而非 Fisher	中等
自然梯度（理想）	精确 Fisher $\Fish^{-1}\grad L$	完整 $d\times d$	大规模下不可行

每一行都是对"最速下降应该用什么度量？"的一个不同回答。大多数都近似 Fisher（§9 的警告：你永远负担不起精确的那个）；Muon 则换成了谱范数。这就是把"选择你的几何"这一思想做成了一份菜单。

杀手级应用 —— 信赖域强化学习

这是"Fisher = KL 的 Hessian"（§8.1）变成一个价值十亿美元算法的最干净的地方。下面把整条谱系作为一条流水线呈现 —— 把每个方框读作"它增添了什么 / 它的代价是什么"：

Fisher = KL 的 Hessian

策略空间上的局部标尺（只是几何，§8）

Kakade NPG (2001)

步 = F⁻¹g，对参数化不变必须对 F 求逆（巨大）

TRPO (2015)

硬约束 KL(π_old‖π_new) ≤ δ共轭梯度 + Fisher-向量积

PPO (2017)

裁剪比率 r=π_new/π_old → 完全无需 F只是近似那个 KL 球

GRPO / R1 (2024–25)

丢弃价值网络；对冻结参考施加 KL 惩罚组内相对优势估计

◀ 同一个思想贯穿这四者：一个 KL 信赖域 —— 每次更新中策略在统计流形上能移动多远 ▶

把"信赖域"具体化：新策略（绿）必须停留在围绕旧策略（蓝）的一个 KL 球（紫）之内 —— 我们只在那个球内信任自己对奖励的估计。

术语注解：信赖域 · 策略梯度 · 概率比率。

Kakade, "A Natural Policy Gradient" 2001NeurIPShigh —— 用策略分布的 $\Fish^{-1}$ 对策略梯度做预条件；这个更新在分布空间中度量改进，使它对策略如何参数化保持不变。在表格情形下，一个自然步 = 一步精确的策略迭代。
Schulman et al., "Trust Region Policy Optimization" (TRPO) 2015ICMLvery-high >10000 · arXiv:1502.05477 —— 在 KL 约束 $\KL(\pi_\text{old}\,\|\,\pi_\text{new})\le\delta$ 下最大化策略的优势。因为 KL 的局部二次型就是 Fisher 度量，这个受约束的步就是自然梯度 $\Fish^{-1}g$，通过共轭梯度 + Fisher-向量积来计算，所以 $\Fish$ 从不被显式构造。几何确实地限制了每次更新中策略在统计流形上"移动"的距离。
Schulman et al., "Proximal Policy Optimization" (PPO) 2017very-high >20000 · arXiv:1707.06347 —— TRPO 信赖域的一个廉价的一阶替代：裁剪概率比率（或加一个 KL 惩罚），从而在不计算 $\Fish$ 的情况下把新策略保持在一个 KL 球内。它是 RLHF 背后的主力。
Tomar et al., "Mirror Descent Policy Optimization" 2020/ICLR'22 · arXiv:2005.09814 —— 把整个家族重新表述为带有 KL Bregman 散度邻近项的镜像下降，即自然梯度的优化理论孪生体（Raskutti & Mukherjee 2015, 1310.7780 证明在对偶坐标下镜像下降 = 自然梯度）。
Shao et al., "DeepSeekMath / GRPO" 2024 · arXiv:2402.03300 以及 DeepSeek-R1 2025 · arXiv:2501.12948 —— 现代 LLM 推理 RL 算法。GRPO 丢弃了价值网络（用组内相对优势），但保留了对一个冻结参考模型的显式 KL 惩罚。那个 KL 锚就是同一套 Fisher/信赖域几何，如今用来约束一个推理模型偏离其起始分布的程度。

贯穿主线：Fisher 是 KL 的 Hessian ⟶ Kakade 用它做预条件 ⟶ TRPO 把它变成一个硬性的 KL 信赖域 ⟶ PPO/GRPO 用裁剪/惩罚来近似那个信赖域。每当你在一篇 RLHF 或推理模型论文里读到"KL 惩罚"，你读到的就是信息几何。

11 · AI II——深度学习中无处不在的 Fisher 信息

除了优化之外，Fisher 矩阵还是一把瑞士军刀：它同时身兼参数重要性、后验精度与影响力三职。统一这一切的事实是（Martens 2020，1412.1193）：对于指数族损失，在极小点处 Fisher = 广义高斯–牛顿 ≈ Hessian，并且（Jacot et al. 2018，NTK，1806.07572）它正是神经正切核在参数空间中的对偶。因此"使用 Fisher" =" 使用曲率" = "使用局部 KL 度量"。它会在以下五处现身：

为什么 Fisher ≈ Hessian（高斯–牛顿拆分）。 这一恒等式为整节内容背书，所以这里给出半页推导。对于损失 $L=-\log p(y\mid f_\vth(x))$，链式法则把 Hessian 拆成两块： $$\grad^2_\vth L = \underbrace{J^\top\!\big(\grad^2_f L\big)J}_{\text{Gauss–Newton term, always PSD}} \;+\; \underbrace{\textstyle\sum_k (\grad_f L)_k\,\grad^2_\vth f_k}_{\text{residual term}},$$ 其中 $J=\partial f/\partial\vth$。残差项以 $\grad_f L$ 加权——即逐样本的"误差信号"——而在真实模型处它的期望为零（正是 §7.1 中那条得分函数均值为零的事实）。所以在期望意义下（或在良好的最优点附近）残差项被冲刷掉，只留下 $\EE[\grad^2 L]=J^\top(\grad^2_f L)J$。对于指数族输出头（softmax、高斯），$\grad^2_f L$ 恰好就是输出端的 Fisher，于是 $\Fish=J^\top\Fish_{\text{out}}J$——这就是广义高斯–牛顿。这正是为什么 §11 可以自由地把 Fisher、高斯–牛顿和最优点处的 Hessian 视为可互换的（也是为什么即便真实 Hessian 并非半正定，它们也保证半正定）；它同时还是 §10 中 K-FAC 所分解的那个 $A\otimes G$——$A$ 来自 $J$ 的激活侧，$G$ 来自输出梯度侧。

Fisher 的用途	论文	几何如何被使用
持续学习（别忘了旧任务）	Kirkpatrick et al., EWC 2017 PNAS · 1612.00796 very-high	用一个以 Fisher 对角元加权的惩罚项，把每个权重锚定到它的旧值。高 Fisher 的权重是"刚性的"（损失在那里曲率陡峭 → 重要），低 Fisher 的权重则保持可塑。形式上这是一个拉普拉斯近似：Fisher = 跨任务传递的高斯后验的精度。
贝叶斯不确定性（事后、廉价）	Ritter et al. 2018 ICLR；Daxberger et al., Laplace Redux 2021 NeurIPS · 2106.14806	在训练好的网络权重周围拟合一个高斯，其精度矩阵就是 Fisher/GGN（经 K-FAC 分解）。无需重新训练即可给出经过校准的不确定性 + 基于边际似然的模型选择——"毫不费力的贝叶斯深度学习"。
模型合并（融合多个微调）	Matena & Raffel, Fisher-weighted averaging 2022 NeurIPS · 2111.09832	通过逐参数的 Fisher 加权平均来合并多个模型——每个模型在它最有把握的坐标上（高 Fisher）主导合并结果。它是 Task Arithmetic（Ilharco et al. 2023，2212.04089，该方法刻意不使用任何曲率）有原则的近亲。
剪枝（移除参数）	Molchanov et al. 2019 CVPR · 1906.10771；LLM-Pruner 2023 · 2305.11627	用损失的泰勒展开给每个权重/滤波器打分；其中的梯度平方项就是一种经验 Fisher 重要性。低 Fisher 的参数可以安全删除（Optimal-Brain-Damage 一脉），如今已应用到 LLM 规模。
数据归因（是哪个样本导致的？）	Koh & Liang, Influence Functions 2017 ICML best paper · 1703.04730；Grosse et al. 2023（Anthropic, 52B）· 2308.03296	通过逆 Hessian 与向量的乘积把一个预测追溯到对它影响最大的训练点——而在最优点处 Hessian 就是 Fisher。扩展到 LLM 需要 EK-FAC 的 Fisher 近似；纯粹是把 §10 的机器复用于可解释性。

退一步，留意这套思想的经济性：同一个矩阵，即 Fisher 信息，同时告诉你 (a) 最佳的步进方向（自然梯度），(b) 每个参数被确定得有多精确（Cramér–Rao、EWC 刚性、拉普拉斯精度），(c) 合并时如何给参数加权，(d) 哪些参数可有可无（剪枝），以及 (e) 一个训练点如何左右某个预测（影响力）。它们全都是同一句话——"模型的局部 KL 几何"——的五种不同读法。这些近似之所以会如此表现，其理论锚点是 Fisher 的特征谱：Karakida et al.（2019，1806.01316）表明它高度各向异性（少数几个巨大的特征值，加上一大片接近零的特征值），而这正是使自然梯度预条件值得一用的那种病态性。

一个矩阵，五种读法。 自然梯度、参数重要性（EWC/Cramér–Rao）、合并加权、剪枝可弃性，以及数据影响力，都是同一个对象——模型的局部 KL 几何——的五种不同读法。术语注释：精度矩阵 · 拉普拉斯近似。

12 · AI III —— 双曲与球面深度学习

在这里，§2–§3 中的非欧几里得几何直接回归：我们不再去弯曲分布的空间，而是弯曲模型放置其表示的嵌入空间。曲率的选择，就是归纳偏置的选择。

“我该取用哪种弯曲空间？”的决策树。整篇文档反复强调的那一课，浓缩在一个分叉点上：让几何匹配你数据的结构。结构混合？就用一个带可学习曲率的乘积流形 $\HH^a\times\mathbb S^b\times\RR^c$（Gu et al. 2019；HELM 的混合曲率专家）。而要在互不重叠的支撑集之间搬运概率质量，则取用 Wasserstein/Otto 度量（§13.3）。

那张为双曲 AI 正名的图。回想（§2.3）半径为 $r$ 的圆盘，在平坦空间中面积为 $\sim r^2$，而在双曲空间中为 $\sim e^{r}$。一棵分支因子为 $b$ 的树，在深度 $r$ 处有 $\sim b^{r}$ 个节点——指数级增长。所以一棵树只需 2 维就能几乎无失真地嵌入双曲空间，而平坦空间会把远房表亲挤到一起。层级 ⟺ 负曲率。反过来，把向量归一化到球面（正曲率）上，会让相似度变得有方向（余弦）且有界——这正是 CLIP 和每个对比模型内部那悄无声息的几何。

整节内容所依托的那张图。节点深度 → 到圆心的距离；每棵子树各占一个角度楔形。在平坦空间中，叶子（深度按 $b^r$ 增长）溢出了多项式级的 $r^2$ 面积，于是不相关的“表亲”被挤压到近乎零的距离（红色）。在双曲圆盘中，面积按 $e^r$ 增长，与树的指数级分支相匹配，于是每个节点都保有自己的空间，表亲彼此远离（绿色）。层级 ⟺ 负曲率——这正是双曲嵌入能仅用 2 维就以极小失真表示树与分类法的原因。

12.1 用双曲空间表示层级

奠基工作——把符号化层级嵌入庞加莱/洛伦兹球：

Nickel & Kiela, "Poincaré Embeddings for Learning Hierarchical Representations" 2017NeurIPSvery-high >5000 · arXiv:1705.08039 —— 开山之作。把一般概念放在靠近原点处，把具体概念放在靠近边界处，于是单个半径就同时编码了层级（范数）与相似度（角度）。用黎曼 SGD 训练（§6.3 的 $\exp$/$\log$ 映射）。仅用 5 维嵌入 WordNet，失真就远低于 200 维的欧几里得嵌入。
Nickel & Kiela, Lorentz model 2018 ICML · 1806.03417 —— 改用双曲面（hyperboloid），那里测地线/$\exp$ 有更简洁的闭式解，且优化能避开庞加莱边界的奇异性。Ganea et al., Entailment Cones 2018 ICML · 1804.01882 —— 用嵌套的测地锥表示非对称的“is-a”关系（子节点必须落在父节点的锥内），从而刻画 DAG 的方向，而不仅是距离。

架构工作——让整个网络弯曲：

Ganea, Bécigneul & Hofmann, "Hyperbolic Neural Networks" 2018 NeurIPS · 1804.03329 —— 借助 Möbius 陀螺向量代数定义了双曲的线性层、偏置、非线性和 softmax。其配方纯属 §6.3：$\log$ 进切空间，做一次欧几里得运算，再 $\exp$ 回去。之后所有的双曲网络都建立在此之上。
Chami et al., "Hyperbolic GCN" 2019 NeurIPS · 1910.12933 —— 在双曲面上做图卷积，并带逐层可训练曲率；在无标度/树状图上大幅提升。Khrulkov et al., "Hyperbolic Image Embeddings" 2020 CVPR · 1904.02239 —— 图像具有潜在层级（一个场景“蕴含”它的各个裁剪块）；到原点的距离成了一种天然的不确定性度量。在小样本/检索中表现强劲。

2023–2026 前沿——曲率进入基础模型：

Desai et al., "MERU: Hyperbolic Image-Text Representations" 2023 ICML · 2304.09172 —— 一个双曲版 CLIP：在洛伦兹模型中以对比 + 蕴含锥损失训练，使文本在几何上包含它所描述的图像，从而找回 CLIP 抹平掉的视觉-语义层级。
Yang et al., "HypLoRA: Hyperbolic Fine-Tuning for LLMs" 2024 / NeurIPS'25 · 2410.04010 —— 在 LLM 词元嵌入中测出树状的、幂律的（负曲率）结构，由此动机出发，将 LoRA 更新直接在洛伦兹流形上进行；在困难推理上提升高达约 13–17%。He et al., "HELM" 2025 · 2505.24722 —— 首个十亿规模的全双曲 LLM，配有混合曲率专家（Mixture-of-Curvature Experts），让每个词元自行选择双曲/欧几里得/球面几何。

12.2 用超球面表示相似度与对比

Radford et al., "CLIP" 2021 ICML · 2103.00020 very-high >5000 —— 它通常被称为“欧几里得”，但其实它把图像和文本嵌入做 L2 归一化到单位超球面上，并按余弦打分。正是这个共享的球面空间，使开放词表的零样本迁移成为可能。
Wang & Isola, "Alignment and Uniformity on the Hypersphere" 2020 ICML · 2005.10242 —— 给出了“为什么”的理论：对比（InfoNCE）损失优化的是球面上的两个几何量——对齐性（正样本靠近）与均匀性（特征借助一种球面能量铺展开）。直接优化这二者就等同于对比学习。
Davidson et al., "Hyperspherical VAEs" 2018 UAI · 1804.00891 —— 用球面上的 von Mises–Fisher 分布替代高斯潜变量，避免了原点坍缩，并能更好地建模方向性数据。混合曲率：Gu et al., "Product Spaces" 2019 ICLR —— 在 $\HH^a\times\mathbb S^b\times\RR^c$ 中嵌入，曲率是学得的，让每个因子匹配它所擅长的结构。这是 HELM 曲率专家在概念上的先驱。

对比学习内部的几何。Wang & Isola 证明了 InfoNCE 损失在单位超球面上优化的恰是这两件事：对齐性（正样本对落得很近）与均匀性（其余一切铺展开以填满球面）。把嵌入归一化到球面并非偶然之举——它正是损失暗中塑造的那种几何。（失败模式：各向异性。）

12.3 LLM 的表示实际上具有什么几何？

2023–2026 间迅速增长的一批文献开始测量 transformer 隐状态的几何——并发现它绝非平坦的欧几里得：

流形与内蕴维度：Valeriani et al. 2023 NeurIPS · 2302.00294 —— 隐状态落在一个低维的弯曲流形上，其内蕴维度随深度先升后降；内蕴维度最小的那一层给出最可迁移的特征（建立在 Ansuini et al. 2019 之上，1905.12784）。Viswanathan et al. 2025 · 2501.10573 —— 词元云的内蕴维度与下一词元损失相关（越难的提示 → 越高维的词元几何）。
轨迹的曲率：Hosseini & Fedorenko 2023 NeurIPS · 2311.04930 —— LLM 会逐层把一个句子的逐层轨迹渐渐拉直；拉得越直 ⇒ 预测越好（通过线性外推来预测）。
各向异性：Ethayarajh 2019 EMNLP · 1909.00512 —— 上下文嵌入占据一个狭窄的锥（高度各向异性）；Godey et al. 2024 EACL · 2401.12143 论证这是自注意力固有的。
为概念学得的非欧几里得度量：Park, Choe & Veitch, "The Linear Representation Hypothesis and the Geometry of LLMs" 2023/ICML'24 · 2311.03658 —— 表示空间上正确的内积不是欧几里得的，而是一个学得的因果内积（一种黎曼式的度量），在它之下不相关的概念彼此正交。Park et al. 2024/ICLR'25 · 2406.01506 表明，在该度量下，范畴性概念构成单纯形，层级则成为正交的多胞形——这恰是 §8 的对偶平坦/度量思想活生生地出现在一个真实 LLM 内部。
持怀疑态度的对照：Robinson et al., "Token embeddings violate the manifold hypothesis" 2025 · 2504.01002 —— 一项统计检验否定了局部平坦性；词元空间存在奇异点（一种分层结构，而非干净的流形）。这是个有益的提醒：几何确实存在，但比任何单一模型所述都更杂乱。

13 · AI IV——黎曼生成模型与最优传输

最后这条主线把前面两半融为一体：要么栖居于黎曼流形之上的生成模型（§4–§6），要么在最优传输几何下搬运概率质量的生成模型——而后者本身就是分布空间上的一个黎曼度量（§8）。

读论文前要先拿下的五个词（否则这些行话会卡住本节）：

得分函数（score）——对数密度的梯度。当心，它有两个含义：$\grad_\theta\log p$（对参数求导 → Fisher，§8）与 $\grad_x\log p$（对数据点求导 →“哪个方向是更可能的数据？”）。§13 指的是后者——这正是扩散模型所学习的对象。
测地线插值——不要在两点之间画一条直线再读取中点，而要沿着停留在弯曲曲面上的最短路径行走。在球面上那是一段大圆弧，而不是穿过内部的弦。
Wasserstein 距离 / 推土机距离——把一堆概率质量物理地搬运成另一堆所付出的代价：代价 = 搬运的质量 × 搬运的距离。与 KL 散度不同，即使两个分布毫不重叠，它也保持有限。
等变（equivariant）——旋转输入，输出会以同样的方式旋转：$f(R\vx)=Rf(\vx)$。这比不变（invariant）（输出不变）更强。蛋白质生成器必须是等变的——旋转分子，预测出的力会随之一同旋转。
ELBO——证据下界（Evidence Lower BOund）：VAE 所最大化的对数似然的可训练下界，$=$ 重构项 $-\,\KL(\text{后验}\,\|\,\text{先验})$。

13.1 流形上的流匹配与扩散

Lipman et al., "Flow Matching for Generative Modeling" 2022/ICLR'23 · 2210.02747 very-high >5000 —— 通过把一个向量场回归到固定的条件概率路径上来训练连续归一化流（无需 ODE 仿真）。其最优传输路径变体使用直线插值——即 Wasserstein 几何的位移测地线。
Chen & Lipman, "Flow Matching on General Geometries" (Riemannian Flow Matching) 2023/ICLR'24 · 2302.03660 —— 把流匹配推广到任意黎曼流形（球面、环面、双曲空间、网格）。条件向量场沿测地线朝每个数据点前进；在测地线有闭式解处保持免仿真。这是流形原生的生成范式。
De Bortoli et al., "Riemannian Score-Based Generative Modelling" 2022 NeurIPS · 2202.02763 以及 Huang et al., "Riemannian Diffusion Models" 2022 NeurIPS · 2208.07949 —— 把扩散的前向/反向 SDE 内蕴地定义在流形上，学习黎曼得分函数——这里的“得分函数”指 $\grad_x\log p$，即数据空间中朝向更高概率的方向（不是 §8 的参数梯度），并用流形的度量来度量。被用于球面上的气候数据、$SO(n)$、双曲面。

13.2 面向科学的 SE(3)-等变生成——赢得近诺奖级成果的几何

蛋白质主链是一串刚性坐标系——乘积流形 $SE(3)^N$（旋转 $\times$ 平移）上的一个点。生成分子/蛋白质 = 在该流形上运行一个扩散或流，并尊重其对称性。这是黎曼生成建模在现实世界中最具影响力的回报。

为什么蛋白质生成是黎曼的。每个氨基酸都是一个小小的坐标三元组（一个刚性位姿）。整条主链是 $SE(3)^N$ 上的一个点；生成模型在该流形上扩散或流动这个点。模型必须是等变的：旋转分子，则每个坐标系的预测都以完全相同的方式旋转。词条释义：SE(3) · SO(3) · 扭转环面 · IGSO(3)。

Satorras et al., "E(n) Equivariant GNNs" (EGNN) 2021 ICML · 2102.09844 very-high >5000 —— 等变骨干网络；沿相对位置向量更新坐标，使输出随输入一同旋转/平移，无需昂贵的球谐函数。Hoogeboom et al., "Equivariant Diffusion (EDM)" 2022 ICML · 2203.17003 —— 用 E(3)-等变去噪器直接在三维原子坐标上做扩散。
Yim et al., "SE(3) Diffusion / FrameDiff" 2023 ICML · 2302.02277 —— 在 $SE(3)^N$ 上原理化的基于得分的扩散，在 $SO(3)$（借助 $\mathrm{IGSO}(3)$ 分布）与 $\RR^3$ 上分别设置独立过程。FrameFlow（2310.05297）与 FoldFlow（2310.02391，ICLR'24 spotlight）把它重新表述为带黎曼最优传输耦合的 SE(3) 流匹配，以获得更直、更快的路径。
Watson et al., "RFdiffusion" 2023 Nature 620 —— SE(3)-坐标系扩散，设计出真实、经实验验证的蛋白质（结合体、支架、寡聚体）。Abramson et al., "AlphaFold 3" 2024 Nature 630 —— 一个在全原子坐标上、面向整个生物分子复合物的扩散模块。Corso et al., "DiffDock" 2022/ICLR'23 · 2210.01776 —— 把对接建模为在非欧几里得流形之积上的扩散：$\RR^3\times SO(3)\times\mathbb T^m$（平移 × 旋转 × 扭转环面）。

13.3 最优传输：分布空间上的一个黎曼度量

Wasserstein 距离为概率分布空间赋予了它自己的黎曼结构（即 Otto 度量），其中测地线是质量传输方案，而梯度流描述扩散。这是信息几何带有传输风味的姊妹。

最优传输，字面意义上的搬运泥堆。Wasserstein 距离是把一个概率“泥堆”重新排布成另一个泥堆的最便宜方式（代价 = 质量 × 距离）。它的测地线就是底行那种平滑形变——而流匹配的直线路径正是它的近似。

JKO /“扩散 = 梯度流”到底在说什么。把熵（无序度）想象成一座山丘，把所有分布构成的空间想象成地形。JKO（1998）证明了扩散过程字面上就是自由能的梯度下降，其中“下坡”是用 Wasserstein（搬土）这把尺子来度量的，而非普通距离。这正是基于得分的扩散模型被称为“概率流形上的梯度流”的精确含义——它们在分布空间上、在最优传输度量之下顺坡滚下。

Arjovsky et al., "Wasserstein GAN" 2017 ICML · 1701.07875 very-high >20000（+ WGAN-GP，Gulrajani et al. 1704.00028）—— 用 Wasserstein-1（推土机）距离取代 JS 散度，即使模型与数据的支撑集不相交也能给出有意义的梯度，治愈了模式坍缩。这是让最优传输成为一等深度学习工具的突破。
Cuturi, "Sinkhorn Distances" 2013 NeurIPS · 1306.0895 —— 熵正则化的最优传输，可用快速的 GPU 矩阵缩放求解；几乎所有“深度学习中的最优传输”的计算骨干（Feydy et al.，Sinkhorn 散度，1810.08278，在 OT↔MMD 之间插值）。
Jordan, Kinderlehrer & Otto (JKO) 1998 SIAM J. Math. Anal. —— 证明 Fokker–Planck 方程是自由能的 Wasserstein 梯度流：扩散 = 在 $W_2$ 度量下熵的最陡下降。这套“Otto 微积分”正是基于得分的扩散模型可被解读为概率流形上梯度流的原因（Zhang et al. 2024 将其对深度模型显式化，2402.06162）。OT-CFM（Tong et al. 2023，2302.00482）与 multisample flow matching（Pooladian et al. 2023，2304.14772）利用小批量最优传输耦合，把流朝 Wasserstein 测地线拉直，从而削减采样步数。

13.4 潜空间隐藏的几何

即便是一个普通的 VAE/GAN/扩散潜空间，暗地里也是黎曼的：解码器 $g$ 把数据空间的度量拉回到潜空间，$\Met(\vz)=J_g(\vz)^\top J_g(\vz)$（解码器 Jacobian），因此在潜坐标里做“直线插值”在几何上是错的——你应当沿测地线走。

为什么潜空间插值需要几何。解码器扭曲了距离，所以潜坐标中的直线（红色）会横穿模型从未学过的区域——产生模糊、无效的中点。在拉回度量下的测地线（绿色）紧贴数据流形，于是每个插值点解码出的都是真实的东西。

插值 t：

看着解码出的中点逐渐退化。把 $t$ 从 0 滑到 1。左：弯曲的数据流形，以及从 A 到 B 的两条路径。右：每条路径在当前 $t$ 处“解码出的样本”——其清晰度跟踪该路径与流形的接近程度（高数据密度 = 清晰，离开流形 = 模糊）。直线路径在中段附近偏离流形，其样本随之糊掉；测地线紧贴流形，全程保持清晰。读数显示在拉回度量下测得的路径长度——尽管在屏幕上看起来更长，测地线在那里其实更短。这就是 §6.3 那套指数/对数/测地线机制的全部回报。

Arvanitidis et al., "Latent Space Oddity" 2018 ICLR · 1710.11379 —— 奠基之作：解码器 Jacobian 诱导出一个（随机的）黎曼度量；在它之下的测地线沿数据流形行进，而不是横穿空白区域，从而修正了插值、聚类和随机游走。通过拉回 Fisher–Rao 度量推广到非高斯解码器（Arvanitidis et al. 2022，2106.05367）。
扩散时代（2025）：Karczewski et al., "The Spacetime of Diffusion Models" · 2505.17517 用去噪分布的 Fisher–Rao 度量为扩散潜空间赋予结构；Saito & Matsubara · 2510.05509 从得分 Jacobian 推导出一个免训练的度量，使引导保持与数据流形相切。§8 的 Fisher 度量，如今进入了 Stable-Diffusion 规模的模型内部。

14 · 工业落地中的几何 —— 它真正落地在哪里

这些都不只是黑板上的推演。上面的每个概念，本周你用过的产品里都在运行。这里给出从想法 → 落地系统的对照图，并指明背后的公司与模型。值得注意的规律是：一篇几何论文要变成某个 API 旋钮或一次前沿训练实验，往往需要 2–7 年。

四个最干净的"几何逃出黑板"的故事

Muon → Kimi K2。 一个正交化更新的优化器（谱范数下的最速下降，§10）从一个业余爱好者的 NanoGPT 刷速项目，一路成长为训练 Moonshot AI 那个 >1 万亿参数生产模型的方案，计算效率约为 AdamW 的 2 倍（2502.16982）。
Matryoshka embeddings → OpenAI 的 dimensions 旋钮。 一篇关于在超球面上嵌套表示的 NeurIPS 论文（2205.13147），在不到两年内变成了一个实打实的 API 参数：text-embedding-3 让你把一个单位球面向量从 3072→256 维截断后重新归一化，用精度换速度，无需重新训练。
GNNs → Google Maps ETA。 道路网络本身就是一个非欧几里得图；一个在其上推理的图神经网络已落地于 Google Maps，将 ETA 准确率提升最多约 50%，服务数十亿用户（2108.11482）。其中一位合著者还撰写了 Geometric-Deep-Learning 蓝图 —— 一座干净的"理论→部署"桥梁。
特征几何 → Golden Gate Claude。 Anthropic 从 Claude 3 Sonnet 的激活空间中提取出约 3400 万个单义特征方向，随后公开发布了一个把"金门大桥"方向钳制到高值的模型 —— 这是表示几何能操控一个在线前沿 LLM 的最清晰演示（Scaling Monosemanticity）。

① 前沿规模的优化器 —— Fisher 与范数几何（§9–§10）

Muon → Moonshot AI Kimi K2 / Moonlight（2502.16982）—— 谱范数最速下降，在 >1T 参数上达到约 2 倍于 AdamW 的效率。理论：modular duality（Bernstein & Newhouse，2410.21265）；Scion（2502.07529）。
Distributed Shampoo → 赢得 MLCommons AlgoPerf 2024（Meta，2309.06497）—— 一个 Kronecker 分解的二阶（Fisher/高斯-牛顿）预条件器，在这个中立的工业基准上比精调过的 Adam 快约 28%。SOAP（2409.11321）在 Adam 的特征基中运行 Shampoo。
EK-FAC 影响函数 → Anthropic，52B LLMs（2308.03296）—— 将 Fisher/Kronecker 曲率部署用来追踪是哪些训练数据驱动了模型的行为。Fisher-weighted merging（2111.09832）支撑了 model-soup 流水线。

② RL 后训练 —— KL 信赖域几何（§8.1、§10）

GRPO → DeepSeek-R1（2402.03300）以及整个 2025 年的推理模型浪潮；DAPO → ByteDance（2503.14476，非对称的 "Clip-Higher" 信赖域）；GSPO → Qwen3（2507.18071，序列级比率，能稳定 MoE RL）；Dr.GRPO（2503.20783）修正了优势几何中的一个归一化偏差。
InstructGPT → ChatGPT（2203.02155）—— 上述方法至今仍在争论的那个对参考策略逐 token 的 KL 惩罚，本质上是分布空间中的一个信赖域；它是几乎所有生产级 RLHF 的几何祖先。

③ 生成式媒体 —— 流匹配与最优传输拉直（§13.1、§13.3）

Stable Diffusion 3 / 3.5 → Stability AI（2403.03206）—— 建立在整流流（2209.03003）之上，即从噪声到数据的直线 OT-位移传输。FLUX.1 → Black Forest Labs（120 亿参数的整流流 transformer，多款产品背后的图像引擎）。Movie Gen → Meta（2410.13720，用于视频的 300 亿参数流匹配模型）。
InstaFlow / Consistency Models / LCM（2309.06380、2303.01469）—— 先拉直再蒸馏，把传输路径压缩到单步，为消费级应用中的实时图像生成提供动力。

④ 药物发现与科学 —— SE(3)/黎曼生成（§13.2）

RFdiffusion → Xaira Therapeutics（10 亿美元启动，2024；Baker lab，Nature 620）—— 货真价实的 $SE(3)^N$ 坐标系扩散（通过 IGSO(3) 在 $SO(3)$ 上做旋转扩散）。RFantibody 将其扩展到抗体设计（Nature 2025）。
AlphaFold 3 → Isomorphic Labs（Nature 630；与 Lilly + Novartis 的合作约 30 亿美元）—— 在原子坐标上的扩散。AlphaFold 2 的 Invariant Point Attention 把残基坐标系当作 $SE(3)$ 元素来运算。Chroma → Generate Biomedicines；Boltz-2 → Recursion（开源的 AF3 级模型 + 结合亲和力）。

⑤ 机器人与具身智能 —— SE(3) 位姿 + 扩散/流策略（§13.2）

Diffusion Policy → TRI Large Behavior Models（2303.04137）—— 在动作轨迹上的扩散，由 Toyota Research Institute 扩展到真实的双臂机器人。π₀ → Physical Intelligence（2410.24164）—— 一个建在 VLM 之上的流匹配动作专家，已部署用于叠衣服。GR00T → NVIDIA（流匹配动作头）。Riemannian Motion Policies → NVIDIA Isaac（1801.02854）。

⑥ 检索与嵌入 —— 单位超球面（§12.2）

Matryoshka → OpenAI text-embedding-3（2205.13147）、Nomic、Cohere —— 嵌套的球面归一化嵌入，作为一个 API dimensions 旋钮。E5 → Microsoft（2212.03533），开源权重的 RAG 主力，用球面上的对齐+均匀性目标训练（2005.10242）。
ScaNN → Google Vertex Vector Search（1908.10396，针对内积几何调优的各向异性量化）；FAISS → Meta（1702.08734），大多数向量数据库底层的 ANN 引擎。对单位向量而言，内积 = 余弦 = 球面几何。

⑦ 产品中的图与双曲结构（§12.1）

GNN ETA → Google Maps（2108.11482，准确率提升约 50%）；PinSage → Pinterest（1806.01973，30 亿节点 / 180 亿边）。Geometric-Deep-Learning 的 "5G" 蓝图（2104.13478）是其统一理论。
双曲 KG 嵌入 → Google（RotH/AttH，2005.00545；MuRP，1905.09791）在生产服务所要求的低维度下取胜。HERec → Snap（2411.13865）—— 来自 Snap 个性化团队的一个全双曲推荐系统，与幂律的用户–物品图相契合。

⑧ 可解释性与操控 —— 表示的几何（§12.3）

Golden Gate Claude → Anthropic（Scaling Monosemanticity）—— 在一个在线生产模型中钳制一个稀疏自编码器的特征方向。Representation Engineering（2310.01405）与 Contrastive Activation Addition（2312.06681）通过在激活空间中加上一个方向来操控行为 —— 当前主导的免训练控制技术。
Linear Representation Hypothesis（2311.03658）表明，那个空间上正确的度量是一个非欧几里得的因果内积（§12.3）—— 这为操控向量为何能奏效提供了理论依据。

15 · 总览速查表 & 延伸阅读

15.1 一页讲完整个故事

概念	符号	一句话含义	在 AI 中的出现
流形	$M$	局部平坦的弯曲空间	数据流形；嵌入空间；蛋白质的 $SE(3)^N$
切空间	$T_pM$	方向所构成的局部线性空间	梯度/速度所在的地方
度量张量	$g_{ij}$	依赖位置的勾股定理，$ds^2=d\vx^\top \Met\,d\vx$	核心对象——选择几何 = 选择 $g$
测地线	$\gamma$	最直/最短路径	潜空间插值；流匹配路径；双曲距离
曲率	$K, R^l_{ijk}$	度量无法保持常数的程度；和乐	双曲（$K\!<\!0$，层级）vs 球面（$K\!>\!0$，相似性）
$\exp$/$\log$ 映射	$\exp_p,\log_p$	在流形与切空间之间往返	每一个双曲层；黎曼 SGD
得分函数（参数）	$\grad_\theta\log p$	似然对参数的敏感度 → 构建 Fisher	MLE、Fisher 信息（§7–8）
得分函数（数据）	$\grad_x\log p$	指向概率更高数据的方向	基于得分的扩散 / 得分匹配（§13）
Fisher 信息	$\Fish$	得分函数的方差 = KL 的 Hessian = 分布上的度量	自然梯度、EWC、Laplace、模型合并、影响函数
Fisher–Rao 度量	$g=\Fish$	统计流形的度量	信息几何；重参数化不变的更新
KL 散度	$\KL(p\\|q)$	$\approx\tfrac12\delta^\top\Fish\delta$；典范散度	TRPO/PPO/GRPO 信赖域；VAE/变分损失
自然梯度	$\Fish^{-1}\grad L$	Fisher 度量下的最速下降	K-FAC、Shampoo、自然策略梯度
Wasserstein 度量	$W_2$	通过质量传输定义的黎曼度量	WGAN、OT 流匹配、作为梯度流的扩散

15.2 值得牢记的四个"顿悟"方程

1. 度量就是几何本身 $$ds^2 = d\vx^\top \Met(\vx)\, d\vx$$ $\Met$ 为常数 → 平坦。$\Met$ 变化 → 弯曲。一切（测地线、曲率）都由 $\Met$ 构建而来。

2. Fisher = KL 的曲率 $$\KL(p_\vth\|p_{\vth+\delta})\approx \tfrac12\delta^\top\Fish\delta$$ 分布所构成的空间是弯曲的，而 $\Fish$ 就是它的度量张量。

3. 自然梯度消除扭曲 $$\vth\leftarrow\vth-\eta\,\Fish^{-1}\grad L$$ 在正确的尺子下做最速下降；重参数化不变；是一个 KL 信赖域。

4. 曲率 = 归纳偏置 $$K<0 \leftrightarrow \text{层级},\quad K>0 \leftrightarrow \text{相似性}$$ 选择与你的数据结构相匹配的嵌入曲率。

15.3 一条学习路径（免费资源优先）

最易上手的路线：

曲面与曲率 → Pressley, Elementary Differential Geometry（第 10 章 Theorema Egregium，第 13 章 Gauss–Bonnet）。
黎曼直觉（物理学家风格）→ Sean Carroll's GR lecture notes（度量、Christoffel 符号、测地线、Riemann 张量，配有详尽的指标演算算例）。
信息几何 → Nielsen, "An Elementary Introduction to Information Geometry"（arXiv:1808.08271）——最好的免费、自洽的入门教程。

严谨路线：do Carmo, Differential Geometry of Curves and Surfaces → do Carmo / J.M. Lee, Riemannian Geometry → Lehmann & Casella, Theory of Point Estimation（把 Fisher 信息讲透）→ Amari & Nagaoka, Methods of Information Geometry（典范著作）以及 Amari, Information Geometry and Its Applications（2016，更注重教学）。

奠基性的一手文献：Riemann 1854 年的就职演讲 On the Hypotheses Which Lie at the Bases of Geometry；Gauss 的 Theorema Egregium（1827）；Fisher（1925）；Rao（1945，Fisher–Rao 度量与 Cramér–Rao）；Chentsov（1972，Fisher 度量的唯一性）。

深入一步——按主题整理的、经过验证的免费资源

主题	最佳进阶资源（标 ✓ 者免费）
几何深度学习（统一性框架）	✓ Bronstein, Bruna, Cohen, Veličković, GDL: Grids, Groups, Graphs, Geodesics, Gauges — arXiv:2104.13478 + 讲座课程
微分几何，对 CS 友好 + 含代码	✓ Keenan Crane, Discrete Differential Geometry — CMU 课程 + 视频；✓ Carroll 的 GR notes 培养指标直觉
最优传输	✓ Peyré & Cuturi, Computational Optimal Transport — 免费书；库 POT 与 OTT-JAX
信息几何	✓ Nielsen, Elementary Introduction to Information Geometry — arXiv:1808.08271；Martens, Natural Gradient — 1412.1193
流匹配（几何式生成）	✓ Lipman et al., Flow Matching Guide and Code — arXiv:2412.06264 + PyTorch 库
双曲 / 基础模型中的几何	✓ 综述：Peng et al. 2021（2101.04562）；Mettes et al. IJCV 2024；Hyperbolic DL for Foundation Models 2025（2507.17787）
直觉热身	✓ 3Blue1Brown — Essence of Linear Algebra / Calculus

建议顺序：3Blue1Brown → GDL 书+讲座 → Crane（动手实践）→ Nielsen（信息几何）→ Martens（自然梯度）→ Peyré–Cuturi（最优传输）→ Lipman（流匹配）→ 双曲综述。

15.4 时间线——纯数学如何变成 AI

📁 本文件夹中的配套文件

这里引用的每一篇论文都有一份完整、有组织的参考文献——包含作者、年份、发表场所、arXiv ID、引用层级，以及一句话说明每篇如何运用几何——见 REFERENCES.md（并按主题分组）。这份 HTML 是教程；那份文件是文献地图。

作为一份自洽的讲解材料而构建 · 所有论文均通过网络搜索结合 arXiv ID 验证 · 数学由 MathJax 渲染 · 图示用 SVG 手工绘制 · 交互演示用原生 JS 实现。
几何不过是度量小步长的规则。其余的一切——包括智能在内——都是你在这条规则之上构建出来的东西。