Notes on ML, AI, and neuroscience#
Working notes on machine learning, statistics, AI, and neuroscience, kept since 2015 and organized into a book.
Each chapter is one of the notes from csinva.io: mostly compact bullet points, equations and figures, with a short overview at the top and interactive figures where moving a slider says more than a paragraph would. The foundations come first (computer science, math, statistics), then machine learning, AI and neuroscience, and finally research overviews that collect papers on open problems.
Where to start#
A few routes through the book, each ordered so that earlier chapters set up later ones. The parts further down list every chapter.
Machine learning from the ground up
The linear algebra, probability and optimization that ML uses, then the standard methods, then why they generalize.
Statistics and causal inference
From estimators and tests to regression and graphical models, then reading causal effects from data.
Deep learning, LLMs and interpretability
Neural networks and language models, then the research on how they work and what they learn.
Brains and minds
How neurons signal, how the senses and movement work, and the models and cognitive science built on them.
Classic AI
Search and logic, decisions under uncertainty and reinforcement learning, then the questions AI raises for society.
Computer systems
The data structures and algorithms everything else assumes, then how the machine and the operating system run programs.
How the chapters connect#
Each chapter is linked to the chapters that share the most vocabulary with it, which is a quick way to see how the topics connect. Hover over a chapter to see its neighbors, or click it to open it.
Computer science#
Data structures, algorithms, architecture and operating systems, programming languages and parallel training, information retrieval, the theory of computation, and a little quantum computing.
Linked lists and stacks, balanced search trees, heaps and Huffman codes, hash tables, and the standard graph algorithms.
Big-O notation, a catalog of sorting algorithms, lower bounds for finding the min and the median, and convex hulls and Voronoi diagrams.
Syntax references for python, c/c++, java and R, and the collective operations behind multi-GPU training.
Integers and floats in bits, assembly and calling conventions, a pipelined processor, caches, exceptions and virtual memory.
Processes and threads, synchronization and deadlock, CPU scheduling, paging and virtual memory, and file systems.
How a search engine is put together, how retrieval differs from exact database lookup, and the main kinds of databases.
Finite automata, context-free grammars and pushdown automata, Turing machines, undecidability, and NP-completeness.
Qubits and amplitudes, which problems quantum computers might speed up, Maxwell’s demon, and a matrix view of probability.
Mathematics#
Reference chapters on calculus, linear algebra and optimization, plus real analysis, differential equations, nonlinear dynamics and signal processing.
Handy bounds, common functions, sets and logic, proof techniques, log and trig identities, and complex numbers in polar form.
Taylor expansions, tables of derivatives, integrals and series, and Lagrange multipliers for constrained minimization.
Matrix properties, matrix calculus, vector and matrix norms, and the eigenvalue and singular value decompositions.
Recipes for ordinary differential equations, from separable and first-order linear equations to constant coefficients, variation of parameters and Laplace transforms.
Completeness of the reals, sequences and series, open and compact sets, continuity, derivatives, uniform convergence and the Riemann integral.
Convex sets and functions, duality and the KKT conditions, descent methods, the optimizers used for neural nets, EM and branch-and-bound.
Sampling, filters, the Fourier transform and DFT, and wavelets, which are localized in both time and frequency.
Bifurcations, stability of fixed points, maps that period-double into chaos, conservative systems and the Lorenz equations.
Statistics#
Probability basics, data analysis, testing, linear models, time series, information theory, graphical models and causal inference.
Expectations and variances, the usual distributions, the law of large numbers and CLT, and the bias-variance trade-off.
Sampling, clear plots, smoothing, imbalanced and missing data, feature selection, and tools and principles for reproducible analysis.
Null and alternative hypotheses, p-values, type 1 and 2 errors and power, t- and F-tests, confidence intervals, and ANOVA.
Least squares and its geometry, ridge, lasso and elastic net, GLS and GLMs, and nonparametric extensions like kernel regression and GAMs.
Trend, seasonality and stationarity, AR, MA and ARMA models, autocovariance and Yule-Walker estimation, and similarity measures like DTW.
Entropy, KL divergence and mutual information, their chain rules and inequalities, and minimum description length.
Bayesian networks and undirected models, exact inference by elimination and message passing, sampling and variational inference, and HMMs and Kalman filters.
Potential outcomes and causal graphs, randomized experiments, quasi-experiments like instruments and regression discontinuity, and effect estimation from observational data.
Machine learning#
Classifiers and how to evaluate them, unsupervised and generative models, kernels, deep learning, computer vision, NLP, and learning theory.
Linear classifiers as regularizer plus loss, SVMs, trees and forests, naive Bayes and Gaussian classifiers, nearest neighbors, and boosting.
Losses and risk, precision, recall and ROC curves, cross-validation, and resampling for stability.
Clustering, dimensionality reduction, deep generative models (VAEs, GANs, diffusion, flows), and self-supervised and contrastive learning.
Valid kernels and their RKHS, and how the eigenvectors of a kernel matrix reveal clusters.
Backpropagation, the main architectures (CNNs, RNNs, GNNs), normalization layers, and training tricks.
How images form, how to filter them and find features, stereo and motion, human vision, and deep networks for vision.
NLP tasks and benchmarks, tokenization, word embeddings, n-gram language models, and parse trees.
How many examples a learner needs, from PAC bounds and VC dimension to version spaces.
Artificial intelligence#
Search, logic, decisions and reinforcement learning, transformers and LLMs, and the psychology, philosophy and ethics around AI.
Uninformed and A* search, local search, constraint satisfaction, propositional and first-order logic, planning, and rule learning.
Game trees, utility theory, the value of information, MDPs, reinforcement learning, deep RL, and RL for LLMs.
The shapes inside a decoder-only transformer, KV caching, self-attention worked by hand, and a reading list of influential transformer models.
How people explain things (Miller, Lombrozo), plus book notes on evolutionary and moral psychology, the evolution of intelligence, and therapy.
How minds might form representations (nativism, empiricism, Bayesian models, deep learning), language, theory of mind, symbolic AI, and cognitive biases.
What makes something a scientific explanation (DN model, statistical relevance, causal mechanisms, unification, pragmatics), plus Deutsch, Kuhn and Durant.
Fairness definitions and fixes, harms of computer vision, a legal test for proxy variables, ethics and moral trade, STS, and effective altruism.
Short thoughts on AGI, then notes on Stuart Russell’s Human Compatible and the essays in Possible Minds.
Neuroscience#
How neurons signal, how the senses, motor system and memory work, and the models and data used to study them.
The parts of the brain, then how a neuron makes a voltage, fires, and talks to the next neuron through synapses and transmitters.
How touch, pain, sight, hearing, balance and taste get from receptors through the thalamus to cortex.
The visual system from photoreceptors and retinal circuits to V1 and beyond, with the methods used to record it and the diseases that damage it.
From spinal reflexes up through motor cortex, basal ganglia and cerebellum, plus eye movements and the autonomic system.
A history of memory research, from Cajal’s neurons to the molecules that turn a short-term memory into a long-term one.
How the nervous system builds itself, from the neural tube and morphogen gradients to axon guidance, competition, critical periods and repair.
Alzheimer’s and Parkinson’s, then how pathologists grade cancer from tissue slides and where deep learning fits in.
Models of neurons, neural codes and learning rules, then data-driven neuroscience, brain encoding models and neuro-inspired AI.
Research overviews#
Annotated reading lists on open research topics, from LLMs, embeddings and interpretability to uncertainty, causal inference and deep learning theory. Most papers come with a short note on what they do.
A grab bag of surprising results from neuroscience, biology, statistics and machine learning.
Clinical datasets, bias in clinical decision instruments, and how doctors reason about and use models.
Training embedding models, contrastive losses, approximate nearest-neighbor search, RAG and interpretable embeddings.
Prompting, architectures and reasoning, mechanistic interpretability, and LLMs that explain data and do science.
Training models that hold up when the test data shift, from domain adaptation and test-time training to adversarial attacks.
Calibration, outlier detection, ensembles, Bayesian neural nets and conformal prediction.
Evaluating interpretations, transparent models like rules, trees and GAMs, and post-hoc explanations of fitted models.
Generalization error as risk minus empirical risk, bounds from uniform stability, and a lower bound for trees on additive data.
Disentangled VAE losses and their variants, GAN latent spaces edited during or after training, weak supervision, and how disentanglement is measured.
Classic causal studies, mediation, heterogeneous effects, causal discovery, invariant prediction, and neural nets for treatment effects.
Predicting RNA and protein structure from sequence, from dynamic programming to AlphaFold, plus engineered viruses as treatment.
Fixed wavelet filters in place of learned ones, from Mallat’s scattering transform to hybrid networks and adaptive wavelets.
What complexity means for a model, from philosophy and Kolmogorov complexity to VC dimension, double descent and minimum description length.
Why overparameterized nets generalize, seen through implicit regularization, kernels, expressiveness, statistical physics and empirical studies.