Machine Learning · Distribution-Free Guarantees

A Beginner's Guide to Learn Then Test

How to put a formal guarantee on a model you've already trained — without retraining it.

Abdalla Alhajeri · May 2026 · 12 min read

A gentle walk through Learn Then Test — a framework that takes a trained model, a tunable knob, and a held-out dataset, and hands you back a setting of the knob with a high-probability bound on whatever risk you care about.

Learn Then Test: choosing a threshold $\hat{\lambda}$ that provably bounds the risk on future data. — From a frozen model and a calibration set, LTT selects a $\hat{\lambda}$ with a high-probability bound on the chosen risk.

Most of what we do in ML is "train a model and hope it generalises." That's fine until you want to put something in front of users and promise — out loud, on paper — that the model won't be wrong more than $\alpha$ of the time.

That kind of promise is what Learn Then Test (LTT, Angelopoulos et al., 2021) is for. The idea is disarmingly simple: you've already trained your model — fine, leave it alone. Now we'll find a setting of one downstream hyperparameter (a confidence threshold, a temperature, a top-$k$) such that some risk you choose is provably bounded, with high probability, on future data drawn from the same distribution.

By the end of this post you'll understand:

What guarantee LTT actually gives you (and what it doesn't).
The five-line recipe: loss → empirical risk → p-values → Bonferroni → pick $\hat{\lambda}$.
How to implement it on a toy binary classifier in a few dozen lines of Python.

The full notebook is in LTT_guide.ipynb if you want to follow along.

The Guarantee, Stated Carefully

Pick a model that outputs a soft score for each input. Pick a knob $\lambda$ that turns that score into a decision (we'll use a threshold: predict "positive" if score $\geq \lambda$). Pick a loss $\ell$ that returns a number in $[0, 1]$ for each sample. The risk at $\lambda$ is the expected loss:

$$R(\lambda) = \mathbb{E}\big[\ell(\text{score}, \text{label}, \lambda)\big]$$

You give LTT two numbers: a risk budget $\alpha$ and a failure probability $\delta$. It hands back a $\hat{\lambda}$ chosen from a finite grid $\Lambda$, with the following promise:

$$\mathbb{P}\big(R(\hat{\lambda}) > \alpha\big) \leq \delta$$

In words: "the chance that the $\hat{\lambda}$ we picked has a true risk above your budget is at most $\delta$." That probability is over the random draw of the calibration set — different calibration sets give different $\hat{\lambda}$, and at most a $\delta$ fraction of them will pick a $\hat{\lambda}$ that fails.

What LTT does not do — It does not improve your model. It does not retrain anything. It does not protect you against distribution shift. It is a post-hoc selection rule for a hyperparameter, built on top of a model you've already trained.

The Setup: A Toy Binary Classifier

To keep things concrete, we use a synthetic binary classification problem (no images, no CIFAR — just sklearn.make_classification). A logistic regression model is trained on 800 samples, and the remaining 1,200 are split equally into a calibration set (used by LTT) and a test set (used only at the end to verify the guarantee).

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=2000, n_features=10,
                           n_informative=5, random_state=42)

X_train, X_temp,  y_train, y_temp  = train_test_split(X, y, test_size=0.6, random_state=42)
X_calib, X_test,  y_calib, y_test  = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)

model = LogisticRegression().fit(X_train, y_train)
calib_scores = model.predict_proba(X_calib)[:, 1]
test_scores  = model.predict_proba(X_test)[:, 1]

From here on, the model is fixed. The only thing we'll vary is $\lambda$ — the threshold at which a score becomes a "positive" prediction.

alpha   = 0.1    # max tolerable false-negative rate
delta   = 0.1    # failure probability of the guarantee
lambdas = np.linspace(0.01, 0.99, 100)
n_calib = len(calib_scores)

Step 1 — Pick a Loss

LTT works for any loss that lives in $[0, 1]$. We'll use one that fires when the model misses a true positive — a false negative:

$$\ell(\text{score}, \text{label}, \lambda) = \mathbf{1}\big[\,\text{label} = 1 \text{ and score} < \lambda\,\big]$$

A small intuition pump: the threshold $\lambda$ is the score above which we'll call a sample positive. If $\lambda$ is low, almost everything clears the bar — we rarely miss a true positive, so the false-negative rate (FNR) is low. If $\lambda$ is high, only very confident predictions pass — and many true positives slip through, so the FNR climbs.

The loss is a design choice, not a fixed rule. Worried about false positives instead? Define $\ell = \mathbf{1}[\text{label}=0 \text{ and score} \geq \lambda]$. Multi-class? Use top-$k$ miscoverage. Generative model? Use anything bounded — toxicity score, BLEU below threshold, expected dollars lost. LTT doesn't care what's inside $\ell$, only that it lies in $[0,1]$.

def loss(score, label, lam):
    return 1 if (label == 1 and score < lam) else 0

Step 2 — Empirical Risk on the Grid

For every $\lambda$ in our grid, compute the average loss on the calibration set:

$$\hat{R}(\lambda) = \frac{1}{n} \sum_{i=1}^{n} \ell(\text{score}_i, \text{label}_i, \lambda)$$

This $\hat{R}(\lambda)$ is our noisy estimate of the true risk $R(\lambda)$. If our calibration set were infinite we'd be done — just pick the largest $\lambda$ with $\hat{R}(\lambda) \leq \alpha$. The whole rest of LTT exists because the calibration set is finite, and $\hat{R}$ can be lucky.

def compute_empirical_risks(scores, labels, lambdas):
    return np.array([
        np.mean([loss(s, l, lam) for s, l in zip(scores, labels)])
        for lam in lambdas
    ])

risks = compute_empirical_risks(calib_scores, y_calib, lambdas)

Plotted, the curve is monotonically increasing in $\lambda$ — exactly as the intuition above predicts.

Step 3 — Turn Each $\hat{R}$ Into a P-value

This is the step where the framework earns its keep. For each $\lambda$, we ask:

$$H_0(\lambda):\ R(\lambda) > \alpha \quad \text{(this $\lambda$ is unsafe)}$$

A p-value is a number $p(\lambda) \in [0, 1]$ with one defining property: if $H_0$ is true, then for any threshold $t$,

$$\mathbb{P}\big(p(\lambda) \leq t \,\big|\, H_0\big) \leq t$$

That's the whole magic. If we reject $H_0$ whenever $p \leq t$, then in the worlds where $H_0$ is actually true, we'll only reject in at most a $t$ fraction of them. So a small p-value is evidence against $H_0$ — evidence that $\lambda$ is, in fact, safe.

Clearing up a misconception — A valid p-value is not "uniform on $[0, \alpha]$." Under $H_0$ it's super-uniform on $[0, 1]$ (the inequality above can be strict). The only property we ever use is that controlling $p \leq t$ controls the false-rejection rate at $t$.

To actually compute a p-value we need a concentration inequality. Since our loss is bounded in $[0, 1]$, Hoeffding gives us one for free:

$$p(\lambda) = \exp\!\big(-2 n \cdot \max(0,\ \alpha - \hat{R}(\lambda))^2\big)$$

Read this formula slowly. If $\hat{R}(\lambda)$ is well below $\alpha$, the exponent is large and negative, $p$ collapses toward zero — strong evidence the true risk is below $\alpha$. If $\hat{R}(\lambda) \geq \alpha$, the $\max$ clamps to $0$, $p = 1$, and we have no evidence of safety at all.

def compute_p_values(risks, n, alpha):
    return np.exp(-2 * n * np.maximum(0, alpha - risks) ** 2)

Step 4 — Bonferroni, Because We Tested 100 Hypotheses

If we had only one $\lambda$ to evaluate, we'd certify it whenever $p \leq \delta$ and walk away. But we have a grid of $|\Lambda| = 100$. Testing each one at level $\delta$ independently and then keeping all that pass is a classic mistake — the family-wise error rate (the probability that any of our certifications is wrong) blows up with the number of tests.

The numbers are unforgiving. If you run $m = 100$ independent tests, each with a 10% false-rejection rate, the chance of at least one false rejection is $1 - 0.9^{100} \approx 0.99997$. Practically guaranteed to fail somewhere.

Bonferroni is the bluntest possible fix: divide $\delta$ by the number of tests and require

$$\lambda \text{ certified} \iff p(\lambda) \leq \frac{\delta}{|\Lambda|}$$

A union bound then guarantees the family-wise error rate stays $\leq \delta$. Bonferroni is conservative — there are tighter procedures (Simes, fixed-sequence testing) — but it always works and it's two lines of code.

def bonferroni_filter(lambdas, p_values, delta):
    threshold = delta / len(lambdas)
    return [lam for lam, p in zip(lambdas, p_values) if p < threshold]

Step 5 — Pick One $\hat{\lambda}$ to Deploy

The Bonferroni filter hands back a set of certified thresholds. Any one of them carries the guarantee — so the question becomes: which one do you want?

For false-negative control, the answer is usually the smallest certified $\lambda$. Lower threshold means more samples pass as positive, which means better recall while still respecting the FNR budget.

def select_lambda_hat(certified):
    if len(certified) == 0:
        return None   # grid too coarse, or calibration set too small
    return np.min(certified)

Empty certified set? It means none of your $\lambda$'s had enough statistical support to clear the Bonferroni bar. Three fixes, in order of effort: (1) collect more calibration data — the Hoeffding bound tightens with $n$; (2) relax $\alpha$ or $\delta$; (3) swap the loss for one that's actually achievable.

Verifying the Guarantee Empirically

Running the pipeline once and seeing test-FNR below $\alpha$ tells you very little — you might have got lucky on a single calibration draw. The real claim is about repeated draws: across many random calibration sets, the resulting $\hat{\lambda}$ should fail (have true risk above $\alpha$) in at most a $\delta$ fraction of trials.

So we re-split the calibration/test data 200 times and count violations:

n_trials, violations, test_fnrs = 200, 0, []

for seed in range(n_trials):
    X_c, X_t, y_c, y_t = train_test_split(X_temp, y_temp, test_size=0.5, random_state=seed)
    s_c = model.predict_proba(X_c)[:, 1]
    s_t = model.predict_proba(X_t)[:, 1]

    r        = compute_empirical_risks(s_c, y_c, lambdas)
    pvals    = compute_p_values(r, len(s_c), alpha)
    cert     = bonferroni_filter(lambdas, pvals, delta)
    lhat     = select_lambda_hat(cert)

    if lhat is not None:
        fnr = np.mean([loss(s, l, lhat) for s, l in zip(s_t, y_t)])
        test_fnrs.append(fnr)
        if fnr > alpha:
            violations += 1

print(f"Violations: {violations}/{n_trials} = {violations/n_trials:.3f}  (expected ≤ δ = {delta})")

On the toy setup, the violation rate sits comfortably below $\delta = 0.1$, often well below — Hoeffding is loose, and Bonferroni piles more conservatism on top. That's the price of distribution-free, finite-sample guarantees: when they hold, they hold; when they're slack, they're very slack.

Answering the Questions I Was Too Confused to Ask

While writing this up I kept tripping over the same things. Three of them are worth spelling out, because they're not really explained in the original paper.

"Does $\lambda$ change the model? Do I retrain?"

No. The model is frozen the moment training ends. $\lambda$ is a post-processing knob — in our case, a threshold applied to the model's score at inference time. LTT searches over settings of that knob; it never touches the weights.

"Does my loss have to be a threshold on a softmax?"

No. That's just the easiest example. Any function $\ell$ that takes (model output, label, $\lambda$) and returns a value in $[0, 1]$ qualifies. The knob $\lambda$ doesn't have to be a threshold either — it can be a temperature, a top-$k$, a prompt template, or even a tuple of all of those (the grid $\Lambda$ then becomes a Cartesian product, and Bonferroni divides by the total grid size).

"What if I want to guarantee something weird — like 'predict dog more than cat when the tongue is long'?"

You'd encode it as a loss. Something like $\ell = \mathbf{1}[\text{tongue\_long}(x) \text{ and prediction} \neq \text{dog}]$, then run LTT against an $\alpha$ you'd be willing to defend. The framework doesn't care whether the property is statistically natural or product-team-driven — only that the loss is bounded and you can evaluate it on calibration data.

That's LTT. A loss you choose, a grid of knob settings, a finite-sample concentration bound, and Bonferroni to glue the multiple-testing together. Five steps and you've turned a vague "the model seems to work" into a probabilistic promise you can sign your name on.

If you spot a mistake in the explanation — especially around p-values, where I had to rewrite this section three times — please get in touch. I'd love to fix it.