01

CLASSICAL FOUNDATIONS

Statistics & optimization — the load-bearing math under everything newer.

FOUNDATIONALCLASSICAL FOUNDATIONS

Bayes Theorem

P(H∣E) = P(E∣H) · P(H)P(E)

How belief must update when evidence arrives. The prior walks in, the data speaks, the posterior walks out.

◈ 1763RUN SIM ▸
01
SOLVEDCLASSICAL FOUNDATIONS

Gradient Descent

θt+1 ← θt − η ∇θ ℒ(θt)

Walk downhill on the loss surface, one small step at a time. Oldest trick in optimization — still the engine of every deep net.

◈ 1847RUN SIM ▸
02
SOLVEDCLASSICAL FOUNDATIONS

Logistic Regression

p = 11 + e−(w·x + b)

Squash any score into a probability with the sigmoid. The humble baseline that beats deep nets more often than anyone admits.

◈ 1838RUN SIM ▸
03
SOLVEDCLASSICAL FOUNDATIONS

Support Vector Machine

minw,b ½‖w‖² + C Σ max(0, 1 − yi(w·xi+b))

Not just any separator — the one with the widest margin. The hinge loss only charges points that get too close.

◈ 1963RUN SIM ▸
04
SOLVEDCLASSICAL FOUNDATIONS

K-Means

argminS Σi=1k Σx∈Si ‖x − μi‖²

Assign every point to its nearest centroid, move centroids to the mean, repeat. The ping-pong that finds clusters.

◈ 1957RUN SIM ▸
05
SOLVEDCLASSICAL FOUNDATIONS

Decision Trees · CART

Gini(S) = 1 − Σc pc²

Ask the single question that purifies the data most, then ask again. Greedy axis-aligned splits, scored by impurity.

◈ 1984RUN SIM ▸
06
IN-USECLASSICAL FOUNDATIONS

Random Forests

f̂(x) = 1B Σb=1B Tb(x)

A parliament of noisy trees, each trained on a bootstrap sample, votes away individual stupidity. Bagging works.

◈ 2001RUN SIM ▸
07
SOLVEDCLASSICAL FOUNDATIONS

Principal Component Analysis

w* = argmax‖w‖=1 wT Σ w

Find the direction the data stretches along most, project onto it, keep the variance, discard the noise.

◈ 1901RUN SIM ▸
08
02

THE DEEP LEARNING CORE

Mechanisms that made multilayer networks trainable at scale.

SOLVEDTHE DEEP LEARNING CORE

Backpropagation

∂ℒ∂W(ℓ) = δ(ℓ) (a(ℓ−1))T

The chain rule, industrialized. Error flows backward layer by layer, telling every weight exactly how much it is to blame.

◈ 1970RUN SIM ▸
09
IN-USETHE DEEP LEARNING CORE

Softmax

softmax(z)i = eziΣj ezj

Turn a vector of scores into a probability distribution. Temperature decides whether it whispers or shouts.

◈ 1986RUN SIM ▸
10
IN-USETHE DEEP LEARNING CORE

Adam Optimizer

θ ← θ − η m̂t√v̂t + ε

Momentum plus per-parameter learning rates, bias-corrected. The default engine of the deep learning era.

◈ 2014RUN SIM ▸
11
FOUNDATIONALTHE DEEP LEARNING CORE

Convolution

(I ∗ K)i,j = ΣmΣn Ii−m, j−n · Km,n

Slide a small kernel across a signal; multiply, sum, repeat. Two centuries old — and still how machines see.

◈ 1754RUN SIM ▸
12
IN-USETHE DEEP LEARNING CORE

Scaled Dot-Product Attention

Attention(Q,K,V) = softmaxQKT√dk V

Every token asks every other token: how relevant are you to me? — then reads a weighted blend of their values. The heart of every LLM.

◈ 2017RUN SIM ▸
13
03

GENERATIVE MODELS

Adversarial games, latent codes and denoising: machines that synthesize.

IN-USEGENERATIVE MODELS

Generative Adversarial Networks

minG maxD 𝔼x[log D(x)] + 𝔼z[log(1 − D(G(z)))]

A forger and a detective locked in a zero-sum game. The forger gets good because the detective keeps score.

◈ 2014RUN SIM ▸
14
IN-USEGENERATIVE MODELS

Variational Autoencoder

ℒ = 𝔼q(z∣x)[log p(x∣z)] − β · DKL(q(z∣x) ‖ 𝒩(0,I))

Encode into a probability cloud, decode from anywhere inside it. Reconstruction quality versus latent smoothness — one dial, β.

◈ 2013RUN SIM ▸
15
IN-USEGENERATIVE MODELS

Denoising Diffusion · DDPM

q(xt∣x0) = 𝒩(√ᾱt x0, (1−ᾱt)I)

Destroy data with noise, then train a net to undo it one whisper at a time. Generation is just denoising from pure static.

◈ 2020RUN SIM ▸
16
IN-USEGENERATIVE MODELS

Flow Matching

d xtd t = vθ(xt, t)

Learn the velocity field that carries noise to data — and straighten the roads while you are at it. The faster, cleaner cousin of diffusion.

◈ 2022RUN SIM ▸
17
04

REINFORCEMENT LEARNING

Learning from reward — and aligning models with intent.

SOLVEDREINFORCEMENT LEARNING

Q-Learning · Bellman Update

Q(s,a) ← Q(s,a) + α[ r + γ maxa′ Q(s′,a′) − Q(s,a) ]

Bootstrapping value from value: each step teaches the current guess using the best guess next door. Off-policy, model-free, provably convergent.

◈ 1989RUN SIM ▸
18
IN-USEREINFORCEMENT LEARNING

Proximal Policy Optimization

ℒ = 𝔼t[ min( rt Ât, clip(rt, 1−ε, 1+ε) Ât )]

Improve the policy, but clip how far any single update may wander. The workhorse behind RLHF and aligned chat models.

◈ 2017RUN SIM ▸
19
05

THE 2020s FRONTIER

Contrastive pairs, sparse experts, selective scan, low-rank patches.

IN-USETHE 2020s FRONTIER

Contrastive Learning · InfoNCE / CLIP

ℒ = −log exp(sim(zi, zj)/τ)Σk exp(sim(zi, zk)/τ)

Push matching pairs together, shove everything else apart. 400 million image–text pairs taught embeddings to understand language.

◈ 2018RUN SIM ▸
20
EXPERIMENTALTHE 2020s FRONTIER

KAN — Kolmogorov–Arnold Networks

f(x) = Σq Φq( Σp φq,p(xp) )

Fire the fixed activations: put learnable univariate functions on the edges instead. Promising, contested, very 2024.

◈ 2024RUN SIM ▸
21
EMERGINGTHE 2020s FRONTIER

Mixture of Experts

y = Σi G(x)i · Ei(x), G = softmax(Wgx)

A router picks the few experts each token needs. Billions of parameters exist — only a fraction ever fire at once.

◈ 1991RUN SIM ▸
22
EMERGINGTHE 2020s FRONTIER

Mamba · Selective State Spaces

ht = Āt ht−1 + B̄t xt, yt = Ct ht

A recurrent scan that chooses what to remember and what to forget, per token — the O(n) challenger to attention.

◈ 2023RUN SIM ▸
23
IN-USETHE 2020s FRONTIER

LoRA — Low-Rank Adaptation

W = W0 + ΔW = W0 + BA, B∈ℝd×r, r ≪ d

Freeze the giant. Train two skinny matrices whose product nudges it. Fine-tuning went from impossible to laptop-sized.

◈ 2021RUN SIM ▸
24