Interactive figures

Every interactive figure in the book, grouped by part. Each one illustrates a single idea from its chapter; follow the link to see it in context.

Computer science

Huffman codeData structures · heapsLetter counts of the text become the leaf weights. Hash probingData structures · Hash tablesThe same keys inserted into a table of size 31 with hash(k) = k mod 31. Knapsack tableAlgorithms · dynamic programmingThe table \(K[i][w]\) from the code below, for four items. Sort stepsAlgorithms · comparison-basedScrub through a sort one comparison at a time. Allreduce stepsLanguages and parallel computing · computationsFour ranks, each with its own vector cut into four slices. Float bitsArchitecture · numbersThe 32 bits of a float, starting at 42.125 (the note's union example, 0x42288000). Cache localityArchitecture · memoryReading a 16×16 int array through a cache with 16-byte lines. Page replacementOperating systems · virtual memoryEach column is one memory reference; a highlighted cell is a page loaded on a fault. Nfa runTheory of computation · ch 1-3 - finite automata, regular expressionsAn example NFA that accepts binary strings ending in 01. Two slit amplitudesQuantum computing · introEach slit contributes an amplitude at every point of the screen, drawn as an arrow on the right.

Mathematics

Complex polarMath basics · imaginary numbersSet the magnitude and angle of \(z_1\) and \(z_2\). Taylor approxCalculus · taylor expansionThe dashed curves are the 0th- (blue), 1st- (orange) and 2nd-order (green) approximations, colored like the terms in the expansion below; purple is order \(N\). Eigen directionsLinear algebra · eigenvalues intro - Strang 5.1Drag \(v\) and watch \(Av\). Svd ellipseLinear algebra · Strang 6.3 - singular value decompositionSet the entries of a 2×2 matrix \(X\) and follow the unit circle through \(V^T\), \(\Sigma\) and \(U\). Power iterationLinear algebra · Strang 5.3 - difference eqs and power $A^k$Iterating \(u_{k+1} = A u_k\) with eigenvectors \(q_1, q_2\) and eigenvalues \(\lambda_1, \lambda_2\). Char rootsDifferential equations · Differential EquationsThe constant-coefficient case \(y'' + py' + qy = 0\), starting from \(y(0) = 1\), \(y'(0) = 0\). Epsilon nReal analysis · ch 2 - sequences and seriesThe definition of convergence. Uniform convergenceReal analysis · 6.2 uniform convergence of a sequence of functionsEach \(f_n\) converges pointwise to \(f\) (orange). Lagrange geometryOptimization · optimality conditionsMinimize \(f_0\) on the circle \(h(x) = 0\) (illustrative \(f_0\)). Gradient descentOptimization · gd methodGradient descent on \(f = (\theta_1^2 + \kappa\,\theta_2^2)/2\), whose Hessian has condition number \(\kappa\). Gradient descentOptimization · why is it hard?A winding canyon: the Rosenbrock function has its minimum at \((1, 1)\) at the end of a curved, flat-bottomed valley. Sampling aliasSignal processing · introA 3 Hz sine sampled 10 times a second. Fourier partial sumsSignal processing · fourier analysisA Fourier series rebuilt one harmonic at a time. Logistic cobwebNonlinear dynamics and chaos · Discrete Nonlinear Dynamical SystemsThe cobweb plot iterates the map: go up to the curve, across to the diagonal, and repeat.

Statistics

Central limitStat basics · central limit thmEach histogram shows 4,000 sample means. Bias varianceStat basics · bias and point estimationPolynomials fit to many noisy samples of the same sine curve (one sample's points are shown). Bayes updateStat basics · overview - Jordan ch 5Estimating a coin's probability of heads. Kernel smootherData science · visual summariesSixty noisy points from a smooth curve. Missing dataData science · missing-data imputation300 rows where \(X\) is always observed and about 40% of the \(Y\) values are missing (hollow circles are the values that were never seen). Multiple testingTesting · basics50 one-sided tests, \(k\) of them with a true effect (illustrative effect size of 3 standard errors). Test powerTesting · tests on hypothesesAn upper-tailed test of \(\theta = 0\) against \(\theta > 0\) from the mean of \(n\) draws. Hat influenceLinear models · introductionLeast squares on twelve fixed points (gray) and one point you move (pink). Reg pathLinear models · regularizationCoefficient paths on one dataset with six standardized features, where \(x_1\) and \(x_2\) have correlation about 0.9 and \(\beta_4 = \beta_6 = 0\). Ar processTime series · ch 3 - linear time seriesAn AR(1) series driven by the same white noise for every \(\phi\). Kl divergenceInformation theory · relative entropy / mutual infoA binary variable with \(P(X = 1) = p\). Mutual infoInformation theory · relative entropy / mutual infoTwo binary variables. Burglary netGraphical models · exact inferenceEnumeration on the network in the figure below: each number is a sum of the joint over all 32 assignments that agree with the evidence. Kalman 1dGraphical models · kalman filteringA 1d random walk tracked by a Kalman filter. Simpson paradoxCausal inference · basicsThe note's example as a binary treatment, with illustrative numbers: within each age group exercise lowers cholesterol by 10, and old people have higher cholesterol. Backdoor pathsCausal inference · back-door criterion: capture parents of TClick a node to add it to the adjustment set \(S\) (shaded); \(U_1\) and \(U_2\) are unmeasured. Ipw confoundingCausal inference · inverse propensity weightingA confounder \(X\) raises both the chance of treatment and the outcome, and the true effect is \(\tau = 1\).

Machine learning

Perceptron trainClassification · binary classificationThe perceptron cycles through the points, and for each misclassified one (\(y_i \theta^T x_i \leq 0\), the points that add to its loss) it updates \(\theta \leftarrow \theta + y_i x_i\), with \(x = (1, x_1, x_2)\) so ... Lda qdaClassification · gaussian class-conditioned classifiersTwo classes, each fit with a Gaussian by maximum likelihood; shading is the posterior \(p(Y=1\vert x)\). Adaboost stumpsClassification · boostingAdaBoost with decision stumps (one split on one feature) on toy data. Roc thresholdEvaluation · measuresScores for negatives and positives as two overlapping normal curves, with equal numbers of each. Kfold cvEvaluation · cvK-fold cross-validation choosing the degree of a polynomial fit to 24 noisy points from a sine curve. Kmeans stepsUnsupervised · statistical clustering (j 10)Step through K-means by hand. Gmm emUnsupervised · statistical clustering (j 10)EM for a mixture of \(K\) Gaussians on made-up 2-d data. Gram psdKernels · kernel basicsThe Gram matrix \(K_n\) of a function \(K(x, z)\) on \(n\) points of a line (left, points sorted so neighbors sit next to each other) and its eigenvalues (right). Spectral ringsKernels · spectral clusteringTwo noisy rings, an RBF kernel \(K(x, z) = \exp(-r\vert\vert x-z\vert\vert^2)\), and Shi and Malik's normalized cut: points are split by the sign of the second smallest generalized eigenvector of \(L_n\). Backprop graphDeep learning · basicsThe graph from the figure below, f = (x + y) z, with forward values in green above each wire and gradients df/d(·) in pink below it. Grad flowDeep learning · empirical notesThe size of the gradient at each layer of a randomly initialized 20-layer network, relative to its size at the output (log scale). Aliasing samplerComputer vision · image preprocessingA cosine sampled once per time step. Image filtersComputer vision · image preprocessingPick a kernel and hover over (or tap) an output pixel: the pink square on the input is the patch whose weighted sum gives that pixel. Bpe mergesNLP · tokenizationBPE on a small corpus, with each word's count on the right and "_" marking the end of a word. Ngram smoothingNLP · language modelingA bigram model trained on five short sentences, with add-\(k\) smoothing \(P(w\vert v) = (c(v, w) + k)/(c(v) + kV)\). Rectangle gameLearning theory · rectangle learning game - KearnsLeft: one game with \(m\) uniform examples. Vc shatterLearning theory · VC dimensionEvery labeling of the points on the left, positives green and negatives pink, with a hypothesis that produces it in blue or a × if none does.

Artificial intelligence

Astar gridSearch and logic · informed searchSearch from S to G on a grid where a step costs 1 (a diagonal step \(\sqrt 2\)) and \(h\) is the cost of the path to G if there were no walls, so it never overestimates. Csp forward checkSearch and logic · backtrackingBacktracking search for a 3-coloring: neighbors must get different colors. Alpha betaRL and decisions · game trees - Russell & Norvig 5.2-5.5Minimax with alpha-beta pruning on a 3-ply tree with random leaf values. Value iterationRL and decisions · value iterationRussell & Norvig's 4×3 grid world. Q learning gridRL and decisions · learning action-utility function $Q(s, a)$Q-learning with ε-greedy exploration on the 4×3 grid world from the value iteration figure, without a model of the transitions. Ngram backoffLLMs · ngram-based modelsAn n-gram model with backoff on a corpus of five short sentences. Attention scoresLLMs · visual explanation of self-attentionAttention weights \(\text{softmax}(QK^T/\sqrt{d})\) for five tokens with random queries and keys. Hawk dovePsychology · social strifeMaynard Smith's hawk-dove game. Base rateCognitive science · lesswrongBase rate neglect with 1000 people (the numbers are illustrative). Fairness thresholdsFairness and STS · metricsPost-processing with one threshold per group.

Neuroscience

Nernst potentialBrain basics · 2 - electrical signals of nerve cellsThe Nernst potential \(V_{ion} = 58/z \cdot \log(X_{out}/X_{in})\) in mV. Quantal releaseBrain basics · synaptic transmissionA toy model of quantal release. Canal adaptationSensory input · 14 - vestibular systemThe horizontal canal as the note describes it: when the head starts turning, the fluid lags and bends the cupula, so the canal fires. Sta cone mosaicVision · electrode arraysA toy ganglion cell whose receptive field is eight cones (right), shown Gaussian white noise and firing Poisson spikes at a rate that grows exponentially with its filtered input. Center surroundVision · horizontal cells - outer retinal signaling and lateral inhibitionAn on-center receptive field as a narrow positive center minus a wide negative surround (left), and its response to a centered light spot (right). Gabor orientationVision · v1 (primary)The oriented linear filter of the standard model, drawn as a Gabor receptive field (top left) and tested with gratings at its preferred spatial frequency (bottom left). Basal ganglia gateMotor system · 18 basal ganglia (choose what you want to do)A toy rate model of the body movement loop. Axon competitionDevelopment · 23 circuit formationSeveral motor axons start out innervating one muscle fiber and share its fixed supply of neurotrophin; an axon that takes up more grows and takes up more still. Rater kappaDisease · prostate cancerCohen's κ, the agreement measure in these benchmarks, corrects the raw agreement \(p_o\) for the agreement two raters would reach by chance, \(p_e\), given how often each calls each grade: \(\kappa = (p_o - p_e)/(1 - p_e)\). Hodgkin huxleyComputational neuroscience · modeling neuronsThe Hodgkin-Huxley equation with the classic squid-axon parameters, driven by current pulses (pink). Lif neuronComputational neuroscience · simplified model neuronsA leaky integrate-and-fire neuron with \(E_L = -65\) mV, \(V_{th} = -50\) mV and a 2 ms refractory period. Population vectorComputational neuroscience · neural decodingEach neuron has a rectified cosine tuning curve with a peak rate of 50 Hz (illustrative) and adds its preferred direction weighted by its firing rate; the sum is the population vector. Hopfield memoryComputational neuroscience · hopfield netsA net of \(d = 100\) units stores \(P\) patterns with the Hebbian rule; pattern 1 is the smiley, the rest are random. Oja ruleComputational neuroscience · synaptic plasticity, hebb's rule, and statistical learningA neuron \(v = \mathbf{w} \cdot \mathbf{x}\) sees one 2-d input at a time and updates \(\mathbf{w}\).

Research overviews

Infonce temperatureEmbeddings · lossesThe softmax weights inside InfoNCE for one query with a positive pair and \(N\) negatives. Self consistencyLLMs · chain-of-thoughtSelf-consistency in an idealized model: each sampled reasoning path is independently right with probability \(p\), otherwise it gives one of four wrong answers, and the most common final answer wins. Covariate shiftTransfer learning · domain invarianceCovariate shift with a misspecified model (illustrative): a line fit to \(y = \sin x\) + noise, with training \(x\) from \(p_\text{train}\) and test \(x\) from \(p_\text{test}\). Platt calibrationUncertainty · basicsAn overconfident classifier on simulated data: its raw probabilities are too extreme, so in the reliability diagram the observed frequencies (blue) sit closer to 0.5 than the predictions, and the pink gaps are large. Conformal intervalsUncertainty · conformal inferenceSplit conformal on illustrative data whose noise grows with \(x\): fit \(\hat f\) on a training split, take the \(\lceil (n+1)(1-\alpha) \rceil\)-th smallest calibration residual as \(\hat q\), and predict $\hat f(x) \pm \hat ... Shapley valuesInterpretability · model-agnosticExact Shapley values for an illustrative model with an interaction, \(f(x) = 1 + 2x_1 + x_2 - 1.5x_3 + \gamma x_1 x_2\), explained against the baseline \(b = 0\). Beta vae tradeoffDisentanglement · disentangled vae lossesA toy where the beta-VAE optimum has a closed form: six independent Gaussian factors with variances \(\lambda_i\), a linear encoder and decoder, and decoder noise \(\sigma^2\). Propensity overlapCausal inference · heterogenous treatment effectsTreatment probability \(e(x) = \sigma(\gamma x)\) for one covariate \(x \sim N(0, 1)\). Ridgeless riskComplexity · double descentThe limiting risk R(γ) from the hastie et al. Min norm gdDeep learning theory · inductive bias=implicit regularization: gradient descent finds good minimaLeast squares with one data point and two weights: every point on the blue line fits exactly.

\(\,\)