1.5 Machine Learning Fundamentals¶

Machine learning is the art of extracting structure from data automatically, rather than writing explicit rules. Before we can understand how large language models are trained, fine-tuned, and evaluated, we need a rock-solid foundation in the concepts that govern all supervised learning systems: the bias-variance tradeoff, regularization, evaluation protocols, and the metrics that tell us whether a model is actually working. This chapter is deliberately broad — it is the “interview breadth core” that interviewers at Google and elsewhere probe before diving into LLM-specific depth.

We will build every idea from first principles and anchor each concept in code you can run. The chapter connects forward to Neural Networks From Scratch: MLPs & Backprop, Calculus, Optimization & Convexity, and eventually to how these ideas manifest in LLM training (see The Pretraining Objective & Loss).

Learning Paradigms: Supervised, Unsupervised, and Self-Supervised¶

Every machine learning algorithm lives in one of three paradigms, distinguished by the type of feedback signal available during training.

Supervised learning is the most classical setting. We have a dataset \(\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^{N}\) of input-output pairs and we want to learn a function \(f_\theta: \mathcal{X} \to \mathcal{Y}\) that minimizes some loss \(\mathcal{L}(f_\theta(\mathbf{x}), y)\) over unseen inputs. Classification (discrete \(y\)) and regression (continuous \(y\)) are the canonical sub-cases. A neural network trained with cross-entropy on ImageNet labels is supervised learning.

Unsupervised learning receives only inputs \(\{x_i\}\) with no labels. The goal is to discover structure — clusters (k-means, DBSCAN), latent representations (PCA, autoencoders, VAEs), or density models (GMMs, normalizing flows). The signal comes from the data geometry itself.

Self-supervised learning (SSL) sits at the intersection. Labels are automatically derived from the data itself, without human annotation. A model predicts a masked or shifted portion of the input. This is the paradigm that makes LLMs possible: the next-token prediction objective creates billions of (context, next-token) pairs from raw text for free. BERT’s masked-language-modeling and contrastive objectives like SimCLR are also SSL. See The Pretraining Objective & Loss for the language-model variant in depth.

The Bias-Variance Tradeoff¶

Why does a model that fits training data perfectly often perform worse on new data? The answer lies in the bias-variance decomposition of expected prediction error.

For a regression problem with true function \(f^*(x)\) and additive noise \(\epsilon \sim \mathcal{N}(0, \sigma^2)\), the expected mean squared error of an estimator \(\hat{f}\) at a point \(x\) decomposes as:

• Bias measures systematic error — how far off is the average prediction from the truth? A linear model fit to non-linear data has high bias.
• Variance measures sensitivity to the specific training set — does the model change dramatically when trained on a different draw from the same distribution? A degree-20 polynomial has high variance.
• Irreducible noise \(\sigma^2\) is the floor: even the perfect model cannot escape it.

The classic tradeoff: as model complexity increases, bias falls but variance rises. The optimal model complexity minimizes their sum.

import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

rng = np.random.default_rng(42)

# True function: f*(x) = sin(x), with Gaussian noise sigma=0.3
def true_fn(x):
    return np.sin(x)

N_train = 15
N_repeats = 200  # number of training-set draws to empirically estimate variance

x_test = np.linspace(0, 2 * np.pi, 100).reshape(-1, 1)
y_test_true = true_fn(x_test)

results = {}  # degree -> (bias^2, variance, avg_test_mse)

for degree in [1, 3, 10]:
    preds_matrix = []
    for _ in range(N_repeats):
        # Fresh noisy training set each time
        x_train = rng.uniform(0, 2 * np.pi, N_train).reshape(-1, 1)
        y_train = true_fn(x_train) + rng.normal(0, 0.3, (N_train, 1))

        model = make_pipeline(
            PolynomialFeatures(degree, include_bias=False),
            LinearRegression()
        )
        model.fit(x_train, y_train.ravel())
        preds_matrix.append(model.predict(x_test))

    preds_matrix = np.array(preds_matrix)        # (N_repeats, N_test)
    mean_pred = preds_matrix.mean(axis=0)        # average prediction across draws

    bias_sq   = np.mean((mean_pred - y_test_true.ravel()) ** 2)
    variance  = np.mean(np.var(preds_matrix, axis=0))
    avg_mse   = np.mean((preds_matrix - y_test_true.ravel()) ** 2)

    results[degree] = (bias_sq, variance, avg_mse)
    print(f"Degree {degree:2d}:  Bias²={bias_sq:.4f}  Var={variance:.4f}  "
          f"MSE≈{avg_mse:.4f}  (Bias²+Var={bias_sq+variance:.4f})")

Running this typically produces something like (magnitudes depend on the RNG seed):

Degree  1:  Bias²=0.4810  Var=0.0062  MSE≈0.5783  (Bias²+Var=0.4872)
Degree  3:  Bias²=0.0041  Var=0.0101  MSE≈0.1041  (Bias²+Var=0.0142)
Degree 10:  Bias²=0.0015  Var=0.2341  MSE≈0.3256  (Bias²+Var=0.2356)

Degree-3 wins: low enough bias to capture the sinusoid, low enough variance because 15 training points constrain the fit well.

Overfitting, Underfitting, and the Generalization Puzzle¶

Underfitting (high bias): the model is too simple to capture the signal. Training loss and test loss are both high. Fix: more capacity, more features, less regularization, longer training.

Overfitting (high variance): the model memorizes training data idiosyncrasies. Training loss is low but test loss is high. The gap is called the generalization gap. Fix: more data, regularization, or a simpler model.

The fundamental measure of generalization is the population risk:

We cannot compute this directly; we estimate it with the empirical risk on held-out data. Classical VC theory bounds the generalization gap as roughly \(O(\sqrt{d_{\text{VC}} / N})\) where \(d_{\text{VC}}\) is the Vapnik–Chervonenkis dimension of the model class. For neural networks, VC theory is loose — modern understanding relies on PAC-Bayes, neural tangent kernel theory, and empirical scaling laws (see Scaling Laws: Kaplan, Chinchilla & Beyond).

Why LLMs seem to avoid classical overfitting. The mechanical reason is the single-epoch regime. A pretraining run consumes each token roughly once, so the empirical risk being minimized is a fresh, unbiased sample of the population risk at every step — train loss and held-out loss track each other almost exactly, and the classical U-curve never appears. It is not that the model memorizes less; it is that there is nothing to memorize twice.

This breaks the moment you repeat data, which you will at 100M scale if your corpus is smaller than your compute budget wants. Muennighoff et al., Scaling Data-Constrained Language Models (2023), found empirically that repeating a corpus for up to roughly four epochs is nearly as valuable as fresh tokens, after which returns decay quickly and the run starts to overfit in the classical sense. That is the number to keep in mind when sizing the capstone corpus (see Data: Sourcing, Filtering, Dedup, Tokenize & Pack ~20B Tokens).

Regularization¶

Regularization refers to any technique that reduces generalization gap, usually by adding a penalty to the training objective or by constraining the model during training.

L2 Regularization (Weight Decay)¶

Add a penalty proportional to the squared norm of the parameters:

The gradient update becomes:

The \((1 - \eta\lambda)\) factor is why this is also called weight decay: parameters are shrunk toward zero at every step. L2 has a Bayesian interpretation as a Gaussian prior \(\theta \sim \mathcal{N}(0, \lambda^{-1}I)\) — MAP estimation with this prior yields L2 regularization.

L1 Regularization (Lasso)¶

L1 promotes sparsity: many parameters are driven to exactly zero, performing automatic feature selection. Bayesian interpretation: Laplace prior on weights. In practice, L1 is rarely used for neural networks (subgradients complicate optimization) but is standard for linear models and feature selection.

Dropout¶

Dropout (Srivastava et al., 2014) randomly zeroes each activation with probability \(p\) during training:

The \(1/(1-p)\) scale factor keeps the expected activation unchanged (inverted dropout). At inference, no masking is applied. Intuition: each forward pass uses a different subnetwork, forcing the model to learn redundant representations that cannot co-adapt. An equivalent view is that dropout approximates an ensemble of \(2^d\) subnetworks (where \(d\) is the number of units).

import torch
import torch.nn as nn

class SimpleMLPWithDropout(nn.Module):
    def __init__(self, input_dim: int, hidden_dim: int, output_dim: int,
                 dropout_p: float = 0.3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Dropout(p=dropout_p),   # Applied only during model.train()
            nn.Linear(hidden_dim, hidden_dim),
            nn.ReLU(),
            nn.Dropout(p=dropout_p),
            nn.Linear(hidden_dim, output_dim),
        )

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.net(x)

model = SimpleMLPWithDropout(64, 256, 10, dropout_p=0.3)

# Training mode: dropout active
model.train()
x = torch.randn(32, 64)
out_train = model(x)   # some activations zeroed

# Eval mode: dropout disabled, full network used
model.eval()
with torch.no_grad():
    out_eval = model(x)  # deterministic, all activations present

print(f"Train output shape: {out_train.shape}")  # (32, 10)
print(f"Eval  output shape: {out_eval.shape}")   # (32, 10)

Early Stopping¶

Rather than adding a penalty to the loss, early stopping monitors validation loss and halts training when it begins to increase. In practice, keep a running best checkpoint and restore it at the end.

class EarlyStopping:
    """Stop training when validation loss has not improved for `patience` steps."""
    def __init__(self, patience: int = 5, min_delta: float = 1e-4):
        self.patience   = patience
        self.min_delta  = min_delta
        self.best_loss  = float('inf')
        self.counter    = 0
        self.best_state = None

    def step(self, val_loss: float, model: nn.Module) -> bool:
        """Returns True if training should stop."""
        if val_loss < self.best_loss - self.min_delta:
            self.best_loss  = val_loss
            self.counter    = 0
            # Deep-copy only the state dict — cheap on GPU
            import copy
            self.best_state = copy.deepcopy(model.state_dict())
        else:
            self.counter += 1

        return self.counter >= self.patience

    def restore_best(self, model: nn.Module) -> None:
        if self.best_state is not None:
            model.load_state_dict(self.best_state)

# Usage in a training loop:
# stopper = EarlyStopping(patience=5)
# for epoch in range(max_epochs):
#     train_one_epoch(model, train_loader, optimizer)
#     val_loss = evaluate(model, val_loader)
#     if stopper.step(val_loss, model):
#         print("Early stopping triggered")
#         break
# stopper.restore_best(model)

Early stopping is, in a sense, equivalent to L2 regularization for gradient descent (see Goodfellow et al., 2016, Chapter 7 for the proof sketch — the equivalence is exact only for a quadratic loss with small learning rate). It has essentially zero computational overhead and is the default in the multi-epoch supervised regime.

Other Regularization Techniques¶

Which of These Survive in LLM Pretraining¶

The regularization toolkit above was built for the multi-epoch, small-data regime. Pretraining a ~100M-parameter model on ~20B tokens is the opposite: roughly a single pass over the corpus, so the model never sees a training token twice and classical overfitting is not the binding constraint — compute is. Accordingly, modern pretraining recipes keep only two of the levers:

• Dropout is set to 0. GPT-2 used dropout 0.1; from GPT-3 onward, and in Llama, OLMo and the SmolLM family, pretraining dropout is disabled. It costs throughput and buys nothing when data is effectively infinite. Dropout returns for SFT and LoRA fine-tuning, where you do run several epochs over a small dataset.
• Decoupled weight decay (AdamW), typically \(\lambda \approx 0.1\), applied only to matrix-shaped parameters. Biases and LayerNorm/RMSNorm gains are excluded: they carry no capacity to overfit with, and decaying a norm gain silently rescales an entire layer’s output. This exclusion is implemented with optimizer parameter groups, and is one of the small details that is easy to get wrong when writing a training loop from scratch.
• Global-norm gradient clipping at 1.0, which is really a stability device rather than a regularizer (see Training Stability, Loss Spikes & Debugging Large Runs).

import torch
import torch.nn as nn

def configure_optimizer(model: nn.Module, lr: float = 3e-4,
                        weight_decay: float = 0.1, betas=(0.9, 0.95)):
    """AdamW with the standard LLM parameter-group split.

    Decay only tensors with ndim >= 2 (weight matrices, embeddings);
    never decay biases or norm gains (ndim == 1).
    """
    decay, no_decay = [], []
    for name, p in model.named_parameters():
        if not p.requires_grad:
            continue
        (decay if p.ndim >= 2 else no_decay).append(p)

    groups = [
        {"params": decay,    "weight_decay": weight_decay},
        {"params": no_decay, "weight_decay": 0.0},
    ]
    print(f"decayed tensors: {len(decay)}  "
          f"({sum(p.numel() for p in decay):,} params)")
    print(f"non-decayed    : {len(no_decay)} "
          f"({sum(p.numel() for p in no_decay):,} params)")
    # betas=(0.9, 0.95) is the LLM convention; 0.999 is too slow-moving here
    return torch.optim.AdamW(groups, lr=lr, betas=betas)

# A toy block with a matrix, a bias, and a norm gain
block = nn.Sequential(nn.Linear(128, 512), nn.GELU(),
                      nn.Linear(512, 128), nn.LayerNorm(128))
opt = configure_optimizer(block)

# In the training loop, clip before stepping:
# loss.backward()
# torch.nn.utils.clip_grad_norm_(block.parameters(), max_norm=1.0)
# opt.step(); opt.zero_grad(set_to_none=True)

This exact parameter-group split is what the capstone’s optimizer uses — see Optimizer & Schedule: Muon + MuonClip and Warmup-Stable-Decay and Optimizers: SGD, Adam, Adafactor, Lion, Muon & Shampoo for why AdamW’s decoupled decay differs from adding \(\frac{\lambda}{2}\|\theta\|^2\) to the loss under an adaptive optimizer.

Train / Validation / Test Splits and Cross-Validation¶

The Three-Way Split¶

The gold standard protocol separates data into three non-overlapping subsets:

• Training set — used to compute gradients and update parameters.
• Validation set — used to tune hyperparameters, select checkpoints, and estimate generalization during development. No gradient flows here.
• Test set — used once at the end to report final performance. Treating test data as if it were validation data leads to test set contamination, a subtle form of data leakage.

A common split for medium-sized datasets is 70/15/15 or 80/10/10. For very large datasets (millions of examples) even 1% is often enough for validation and test.

Cross-Validation¶

When data is scarce, \(k\)-fold cross-validation makes better use of every example:

1. Partition data into \(k\) equal folds.
2. Train \(k\) times, each time holding out one fold as validation.
3. Average the \(k\) validation scores as the generalization estimate.

For \(k = N\) (leave-one-out CV, LOOCV) the estimate is nearly unbiased but expensive. Stratified \(k\)-fold preserves class ratios in each fold.

from sklearn.model_selection import StratifiedKFold
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
import numpy as np

X, y = make_classification(n_samples=500, n_features=20,
                            n_informative=10, random_state=0)

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
fold_accuracies = []

for fold_idx, (train_idx, val_idx) in enumerate(skf.split(X, y)):
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]

    clf = LogisticRegression(max_iter=200)
    clf.fit(X_train, y_train)

    acc = accuracy_score(y_val, clf.predict(X_val))
    fold_accuracies.append(acc)
    print(f"  Fold {fold_idx+1}: accuracy={acc:.4f}")

print(f"\n5-fold CV accuracy: {np.mean(fold_accuracies):.4f} "
      f"± {np.std(fold_accuracies):.4f}")

Hyperparameter Search: What the Validation Set Is For¶

The validation set exists so you can choose things: learning rate, weight decay, width, dropout. Three facts govern how:

• Random search beats grid search (Bergstra & Bengio, 2012). If only a few hyperparameters actually matter, a grid wastes most of its budget re-evaluating irrelevant axes at the same values of the important ones; random sampling gives every trial a distinct value of every hyperparameter. Sample learning rates and weight decays log-uniformly, not uniformly.
• Kill bad trials early. Successive halving and its asynchronous variant ASHA run many configurations for a short budget, keep the top fraction, and repeat — an order of magnitude more configurations explored for the same compute.
• You cannot tune at target scale. For anything larger than a toy, the hyperparameters are chosen on small proxy models and transferred, either empirically via a scaling-law sweep or analytically via \(\mu\)P width-scaling rules. See Mini Scaling Laws: Fit Your Own Law Before Spending the Budget and Learning Rate Schedules, Warmup, Batch Size & Hyperparameters.

The open-source tooling at this layer is Optuna (define-by-run search with a TPE sampler and pruners), Ray Tune (distributed scheduling, including ASHA and PBT), and Weights & Biases Sweeps or MLflow for tracking and comparing runs. A minimal Optuna study:

import optuna                       # pip install optuna
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=20,
                           n_informative=10, random_state=0)

def objective(trial):
    # Log-uniform over regularization strength; C = 1/lambda in sklearn
    C = trial.suggest_float("C", 1e-4, 1e2, log=True)
    penalty = trial.suggest_categorical("penalty", ["l1", "l2"])
    clf = LogisticRegression(C=C, penalty=penalty, solver="liblinear", max_iter=500)
    # Objective = mean validation accuracy; never touches the test set
    return cross_val_score(clf, X, y, cv=3, scoring="accuracy").mean()

study = optuna.create_study(direction="maximize",
                            sampler=optuna.samplers.TPESampler(seed=0))
study.optimize(objective, n_trials=25, show_progress_bar=False)
print(study.best_params, f"{study.best_value:.4f}")

Data Leakage — The Silent Killer¶

Data leakage occurs when information from the validation or test set contaminates the model during training or preprocessing. It causes the model to appear better than it truly is on unseen data.

Common leakage sources:

1. Preprocessing with full-dataset statistics. Fitting a StandardScaler on all data before splitting; the test set’s statistics influence the training normalization.
2. Feature engineering that looks ahead. A time-series feature like “average sales next week” leaks future labels into training features.
3. Duplicates across splits. Near-duplicate samples in both train and test; especially common with web-scraped LLM pretraining data (see Data Cleaning, Deduplication & Quality Filtering).
4. Label leakage. A feature that is a direct proxy for the label (e.g., including the diagnosis code when predicting disease).

The Split Protocol for a Pretraining Run¶

Cross-validation is never used for LLM pretraining — retraining \(k\) times is unaffordable, and with \(10^{10}\)+ tokens the variance of a single held-out estimate is already negligible. The protocol instead looks like this, and it is worth getting right before you spend a GPU-week:

1. Hold out whole documents, not random token windows. If half of a document lands in train and half in validation, the model has effectively seen the validation text’s context and your val loss is optimistic. Split at the document level, deterministically, by hashing a stable document id.
2. Hold out after deduplication. A near-duplicate of a validation document sitting in the training set is exactly the “duplicates across splits” leak from above, and web crawls are full of them (see Data Cleaning, Deduplication & Quality Filtering).
3. Match the training mixture, and also keep per-domain shards. A single validation loss over the same mixture as training is your primary curve; separate per-domain shards (web / code / math) tell you which domain a regression came from. A few million tokens per shard is plenty.
4. Decontaminate against your evaluation benchmarks. Strip training documents that n-gram-overlap with the test splits of the benchmarks you plan to report. This is the LLM version of test-set contamination and the single most common cause of over-reported results (see The Evaluation Problem & Benchmark Landscape).

import hashlib

def split_of(doc_id: str, val_frac: float = 0.001) -> str:
    """Deterministic, order-independent document-level train/val assignment.

    Hashing (rather than shuffling) means the same document always lands in
    the same split, even across re-runs, added shards, or parallel workers.
    """
    h = hashlib.blake2b(doc_id.encode("utf-8"), digest_size=8).digest()
    bucket = int.from_bytes(h, "big") / 2**64        # uniform in [0, 1)
    return "val" if bucket < val_frac else "train"

docs = [f"cc-2026-08/{i:07d}" for i in range(100_000)]
counts = {"train": 0, "val": 0}
for d in docs:
    counts[split_of(d)] += 1
print(counts)   # {'train': 99894, 'val': 106} — ~0.1% held out

In practice you do not write the surrounding pipeline by hand: HuggingFace datasets handles sharded streaming reads and train_test_split, and datatrove provides distributed dedup and filtering blocks that run the same code locally or on a Slurm cluster. The capstone applies exactly this protocol to ~20B tokens in Data: Sourcing, Filtering, Dedup, Tokenize & Pack ~20B Tokens, and the resulting validation shard is what the training loop in The Pretraining Run: A Complete Single-GPU Training Loop evaluates against.

Evaluation Metrics¶

Choosing the right metric is at least as important as choosing the right model. The loss function guides training; evaluation metrics guide decisions.

Classification Metrics¶

For binary classification with a threshold at 0.5:

The \(F_\beta\) score generalizes this:

• \(\beta > 1\): weight recall higher (e.g., medical screening — false negatives are costly).
• \(\beta < 1\): weight precision higher (e.g., spam filter — false positives annoy users).

The confusion matrix for binary classification:

For multi-class problems, we report per-class precision/recall and average them:

• Macro average: unweighted mean across classes — treats all classes equally regardless of size.
• Weighted average: weighted by class support — appropriate for imbalanced datasets.

ROC and AUC¶

The Receiver Operating Characteristic (ROC) curve plots True Positive Rate (TPR = recall) vs. False Positive Rate (FPR = \(FP / (FP + TN)\)) as the classification threshold varies from 1 to 0. A perfect classifier passes through \((0, 1)\); a random classifier lies on the diagonal.

AUC (Area Under the ROC Curve) summarizes the entire curve in one number. AUC = 0.5 is random; AUC = 1.0 is perfect. AUC has a useful probabilistic interpretation: it equals the probability that the model ranks a randomly chosen positive example higher than a randomly chosen negative.

PR-AUC (Precision-Recall AUC) is preferred for highly imbalanced datasets, where even a near-trivial model can achieve high ROC-AUC by virtue of the many true negatives.

import numpy as np
from sklearn.metrics import (
    classification_report, roc_auc_score, average_precision_score,
    confusion_matrix
)

# Simulate true labels and a probabilistic classifier
rng = np.random.default_rng(0)
y_true  = rng.integers(0, 2, size=1000)  # 50% positive
y_score = rng.beta(2, 2, size=1000)      # random probs, slightly informed

# Nudge scores toward correct labels to create a non-trivial model
y_score = np.where(y_true == 1,
                   np.clip(y_score + 0.2, 0, 1),
                   np.clip(y_score - 0.2, 0, 1))

y_pred  = (y_score >= 0.5).astype(int)

print(classification_report(y_true, y_pred, digits=4))
print(f"ROC-AUC      : {roc_auc_score(y_true, y_score):.4f}")
print(f"PR-AUC       : {average_precision_score(y_true, y_score):.4f}")
print(f"Confusion matrix:\n{confusion_matrix(y_true, y_pred)}")

Calibration¶

A model is calibrated if its predicted probability \(\hat{p}\) equals the empirical frequency of the positive class among examples assigned that score. Formally, for all \(p \in [0,1]\):

A model that outputs 0.9 for 1000 examples should have approximately 900 of them be positive. Miscalibration matters enormously in production: a classifier guarding a medical device with \(\hat{p} = 0.9\) should not be trusted if the true frequency is 0.5.

The Expected Calibration Error (ECE) estimates calibration by binning predictions:

where \(B_m\) is the \(m\)-th bin of predictions, \(\text{acc}(B_m)\) is the fraction of positives in that bin, and \(\text{conf}(B_m)\) is the mean predicted probability.

Neural networks are often overconfident — their raw softmax scores are not well-calibrated. Temperature scaling is a simple post-hoc fix: divide logits by a learned scalar \(T > 1\) before softmax, which smooths the output distribution. In scikit-learn, calibration_curve plots the reliability diagram and CalibratedClassifierCV wraps a fitted estimator with Platt scaling (method="sigmoid") or isotonic regression; you implement ECE and temperature scaling yourself in Exercises 5 and 6.

Calibration is not just a classical-ML concern. The same sampling temperature that controls generation diversity at inference (see Sampling Strategies & Decoding Algorithms) is literally this \(T\), and an LLM’s calibration is routinely measured on multiple-choice benchmarks by comparing its normalized answer log-probabilities against empirical accuracy — the standard lm-evaluation-harness reports exactly these log-likelihoods.

Regression Metrics¶

For language models, the primary metric is perplexity, which is the exponentiated average cross-entropy per token:

A perplexity of 10 means the model is, on average, as uncertain as choosing uniformly among 10 equally likely next tokens.

Perplexity is not comparable across tokenizers. A model with a larger vocabulary packs more characters into each token, so it makes fewer, harder predictions; its per-token perplexity can look worse while the model is objectively better at compressing the same text. The tokenizer-independent fix is bits-per-byte (BPB) — total negative log-likelihood measured in bits, divided by the number of UTF-8 bytes in the raw text:

where \(\mathcal{L}_{\text{CE}}\) is mean cross-entropy in nats per token, \(T\) the token count and \(B\) the byte count. For example, at a validation cross-entropy of \(3.0\) nats/token with a byte-level BPE tokenizer achieving \(4.3\) bytes/token, \(\text{BPB} = (3.0 / 0.6931) / 4.3 \approx 1.01\) bits per byte. Report BPB whenever you compare two models that do not share a tokenizer — it is the metric of choice for the capstone’s held-out evaluation (see A Byte-Level BPE Tokenizer From Scratch (and Why Vocab Size Is a Design Lever at 100M) and Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop).

import math

def bits_per_byte(mean_ce_nats: float, n_tokens: int, n_bytes: int) -> float:
    """Convert mean per-token cross-entropy (nats) to tokenizer-independent BPB."""
    total_bits = mean_ce_nats * n_tokens / math.log(2)
    return total_bits / n_bytes

# 1M tokens of validation text that is 4.3M UTF-8 bytes long
print(f"{bits_per_byte(3.0, 1_000_000, 4_300_000):.4f}")   # 1.0065

The Generalization Puzzle¶

How does any learning algorithm generalize at all? The deep answer lies in inductive biases — assumptions baked into the model architecture and training procedure that help it prefer simpler, more structured solutions.

The No Free Lunch theorem (Wolpert, 1997) states that no algorithm outperforms all others averaged over all possible problem distributions. Generalization only makes sense relative to a prior on the problem class. Neural networks are not magic; they generalize because the inductive biases of weight sharing (CNNs), sequential attention (Transformers), and gradient descent with small learning rate happen to align well with the structure of natural data (images, language, code).

Occam’s Razor in practice. Regularization implements a soft preference for simpler models. The minimum description length (MDL) principle formalizes this: the best model is the one that minimizes total description length of both the model and the data given the model. L2 regularization corresponds to preferring models near the origin (short code) under a Gaussian prior.

Implicit regularization by SGD. Stochastic gradient descent, even without explicit regularization, biases toward flat minima (Keskar et al., 2017) and low-rank solutions (Li et al., 2018). The noise injected by mini-batches prevents convergence to sharp, narrow minima that generalize poorly. This is directly relevant to LLM training — the batch size, learning rate schedule, and optimizer choice all implicitly regularize the model. See Optimizers: SGD, Adam, Adafactor, Lion, Muon & Shampoo for the optimizer perspective.

Grokking. First documented by Power et al. (2022): a model first memorizes training data (overfitting), then — after many more training steps — suddenly generalizes. Follow-up mechanistic-interpretability work (Nanda et al., 2023) reverse-engineered the learned algorithm on small algorithmic tasks and showed the “sudden” jump is actually the endpoint of a gradual internal phase transition, where a generalizing circuit forms and only later displaces the memorizing one. This suggests that generalization can emerge from extended optimization, not just early stopping, and challenges the simple narrative that training longer always hurts.

A Complete Training Pipeline in Code¶

The following implements a clean train/val/test workflow with L2 regularization, dropout, early stopping, and evaluation reporting. It is intentionally simple enough to read in one sitting.

"""
A self-contained binary classification training pipeline demonstrating:
  - Proper train/val/test split
  - L2 weight decay + dropout regularization
  - Early stopping with best-checkpoint restoration
  - Precision, Recall, F1, ROC-AUC reporting
"""

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, roc_auc_score
from sklearn.preprocessing import StandardScaler
import numpy as np
import copy

# ─── 1. Dataset ───────────────────────────────────────────────────────────────
X, y = make_classification(
    n_samples=4000, n_features=30, n_informative=15,
    n_redundant=5, random_state=0
)

# Split: 70% train, 15% val, 15% test
X_train_raw, X_test_raw, y_train, y_test = train_test_split(
    X, y, test_size=0.15, stratify=y, random_state=42)
X_train_raw, X_val_raw, y_train, y_val = train_test_split(
    X_train_raw, y_train, test_size=0.15/(1-0.15), stratify=y_train, random_state=42)

# ─── 2. Preprocessing — fit ONLY on training data ─────────────────────────────
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train_raw)   # fit+transform
X_val   = scaler.transform(X_val_raw)          # transform only
X_test  = scaler.transform(X_test_raw)         # transform only

# Convert to torch tensors
def to_tensor(X, y):
    return TensorDataset(
        torch.tensor(X, dtype=torch.float32),
        torch.tensor(y, dtype=torch.float32)
    )

train_loader = DataLoader(to_tensor(X_train, y_train), batch_size=64, shuffle=True)
val_loader   = DataLoader(to_tensor(X_val,   y_val),   batch_size=256)
test_loader  = DataLoader(to_tensor(X_test,  y_test),  batch_size=256)

# ─── 3. Model ──────────────────────────────────────────────────────────────────
class Classifier(nn.Module):
    def __init__(self, in_dim: int, hidden: int = 128, dropout_p: float = 0.3):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(in_dim, hidden), nn.ReLU(), nn.Dropout(dropout_p),
            nn.Linear(hidden, hidden), nn.ReLU(), nn.Dropout(dropout_p),
            nn.Linear(hidden, 1)         # raw logit; sigmoid applied in loss
        )

    def forward(self, x):
        return self.net(x).squeeze(-1)   # shape: (batch,)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model  = Classifier(X_train.shape[1]).to(device)

optimizer = optim.AdamW(
    model.parameters(),
    lr=1e-3,
    weight_decay=1e-4   # L2 regularization via AdamW's decoupled weight decay
)
criterion = nn.BCEWithLogitsLoss()   # numerically stable sigmoid + BCE

# ─── 4. Early Stopping ─────────────────────────────────────────────────────────
patience, best_val_loss, wait = 10, float('inf'), 0
best_state = None

# ─── 5. Training loop ──────────────────────────────────────────────────────────
for epoch in range(100):
    # --- Train ---
    model.train()
    for X_b, y_b in train_loader:
        X_b, y_b = X_b.to(device), y_b.to(device)
        optimizer.zero_grad()
        loss = criterion(model(X_b), y_b)
        loss.backward()
        optimizer.step()

    # --- Validate ---
    model.eval()
    val_losses = []
    with torch.no_grad():
        for X_b, y_b in val_loader:
            X_b, y_b = X_b.to(device), y_b.to(device)
            val_losses.append(criterion(model(X_b), y_b).item())
    val_loss = np.mean(val_losses)

    if val_loss < best_val_loss - 1e-4:
        best_val_loss = val_loss
        wait          = 0
        best_state    = copy.deepcopy(model.state_dict())
    else:
        wait += 1
        if wait >= patience:
            print(f"Early stop at epoch {epoch+1}  (best val_loss={best_val_loss:.4f})")
            break

# ─── 6. Restore best and evaluate on test set ──────────────────────────────────
model.load_state_dict(best_state)
model.eval()

all_logits, all_labels = [], []
with torch.no_grad():
    for X_b, y_b in test_loader:
        logits = model(X_b.to(device)).cpu().numpy()
        all_logits.extend(logits)
        all_labels.extend(y_b.numpy())

y_score = torch.sigmoid(torch.tensor(all_logits)).numpy()
y_pred  = (y_score >= 0.5).astype(int)
y_true  = np.array(all_labels).astype(int)

print(classification_report(y_true, y_pred, digits=4))
print(f"ROC-AUC: {roc_auc_score(y_true, y_score):.4f}")

Interview Corner¶

Key Takeaways¶

Further Reading¶

• Goodfellow, Bengio & Courville, Deep Learning (2016) — Chapters 5–7 cover the statistical foundations, ML basics, and regularization with full mathematical derivations.
• Vapnik, The Nature of Statistical Learning Theory (1995) — the original framework for VC dimension, structural risk minimization, and support vector machines.
• Srivastava et al., “Dropout: A Simple Way to Prevent Neural Networks from Overfitting,” JMLR 2014.
• Belkin et al., “Reconciling modern machine-learning practice and the classical bias–variance trade-off,” PNAS 2019 — double-descent phenomenon.
• Power et al., “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets,” ICLR 2022 workshop — late generalization in neural networks.
• Guo et al., “On Calibration of Modern Neural Networks,” ICML 2017 — temperature scaling and ECE.
• Keskar et al., “On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima,” ICLR 2017 — why batch size affects generalization.
• Wolpert, “The Lack of A Priori Distinctions Between Learning Algorithms,” Neural Computation 1996 — No Free Lunch theorems.

Exercises¶

1. You train two models on the same task and observe the following:

• Model A: training loss \(= 0.62\), validation loss \(= 0.65\).
• Model B: training loss \(= 0.02\), validation loss \(= 0.71\).

For each model, decide whether it is primarily underfitting or overfitting, name the dominant error term (bias vs. variance) using the decomposition in this chapter, and give one concrete fix from the chapter’s regularization toolkit that is appropriate for each case.

2. A disease-screening classifier is evaluated on 1000 patients, of whom 200 truly have the disease (positive class). The model produces this confusion matrix:

• \(TP = 150\), \(FN = 50\), \(FP = 100\), \(TN = 700\).

Compute (by hand) precision, recall, \(F_1\), and accuracy. Then compute \(F_2\) (i.e. \(\beta = 2\)). Which of \(F_1\) or \(F_2\) is the more appropriate headline metric here, and why?

3. L2 regularization implements weight decay through the factor \((1 - \eta\lambda)\) applied to each weight every step. Suppose \(\eta = 0.02\) and \(\lambda = 0.5\).

(a) What is the per-step multiplicative shrink factor? (b) Consider a single weight \(w = 4.0\) that sits in a flat region of the loss, so its data-gradient \(\nabla_\theta \mathcal{L}_{\text{data}} \approx 0\). What value does \(w\) decay to after 100 steps? © Explain in one sentence why this is called “decay” and what prior this corresponds to under the Bayesian view.

4. You bin a calibrated-classifier’s validation predictions into 3 confidence bins (\(N = 100\) examples total) and record:

Compute the Expected Calibration Error (ECE). Is this model over- or under-confident overall, and which bin contributes the most to the ECE?

5. Implement the Expected Calibration Error as a standalone function expected_calibration_error(y_true, y_score, n_bins=10) using only NumPy, following the binning formula from this chapter. Then verify it reproduces the hand-computed answer from Exercise 4 using equal-width bins.

import numpy as np

def expected_calibration_error(y_true, y_score, n_bins=10):
    """ECE via equal-width binning of predicted probabilities.

    y_true : array of {0,1} labels
    y_score: array of predicted P(y=1) in [0,1]
    """
    y_true  = np.asarray(y_true,  dtype=float)
    y_score = np.asarray(y_score, dtype=float)
    N = len(y_score)
    edges = np.linspace(0.0, 1.0, n_bins + 1)

    ece = 0.0
    for m in range(n_bins):
        lo, hi = edges[m], edges[m + 1]
        # include the right edge only in the final bin
        if m == n_bins - 1:
            mask = (y_score >= lo) & (y_score <= hi)
        else:
            mask = (y_score >= lo) & (y_score < hi)
        n_m = mask.sum()
        if n_m == 0:
            continue
        acc  = y_true[mask].mean()    # fraction of positives in bin
        conf = y_score[mask].mean()   # mean predicted probability
        ece += (n_m / N) * abs(acc - conf)
    return ece

# Bin 1: 40 examples, conf 0.75, acc 0.70  -> 28 positives
# Bin 2: 35 examples, conf 0.88, acc 0.80  -> 28 positives
# Bin 3: 25 examples, conf 0.95, acc 0.72  -> 18 positives
y_true, y_score = [], []
for count, conf, acc in [(40, 0.75, 0.70), (35, 0.88, 0.80), (25, 0.95, 0.72)]:
    n_pos = round(count * acc)
    y_true  += [1] * n_pos + [0] * (count - n_pos)
    y_score += [conf] * count           # all scores in a bin share its conf

# Three distinct score values -> put each in its own bin via fine binning
ece = expected_calibration_error(y_true, y_score, n_bins=100)
print(f"ECE = {ece:.4f}")   # ECE = 0.1055

6. The chapter fixes miscalibration with temperature scaling: divide the logits by a learned scalar \(T\) before the sigmoid/softmax. Implement fit_temperature(logits, labels) that finds the \(T > 0\) minimizing validation cross-entropy for a binary classifier, then show how to apply it at inference. Explain why \(T\) must be fit on the validation set rather than the training set.

import torch
import torch.nn as nn

def fit_temperature(logits, labels, max_iter=50):
    """Learn a scalar temperature T>0 that calibrates binary logits.
    logits, labels: 1-D arrays over the VALIDATION set. Returns float T.
    """
    logits = torch.as_tensor(logits, dtype=torch.float32)
    labels = torch.as_tensor(labels, dtype=torch.float32)

    log_T = torch.zeros(1, requires_grad=True)   # T = exp(log_T) = 1.0 init
    optimizer = torch.optim.LBFGS([log_T], lr=0.1, max_iter=max_iter)
    bce = nn.BCEWithLogitsLoss()

    def closure():
        optimizer.zero_grad()
        T = torch.exp(log_T)
        loss = bce(logits / T, labels)       # divide logits by T
        loss.backward()
        return loss

    optimizer.step(closure)
    return torch.exp(log_T).item()

# --- Usage: fit on validation logits, then apply at inference ---
# T = fit_temperature(val_logits, val_labels)
# calibrated_prob = torch.sigmoid(torch.as_tensor(test_logits) / T)