How it compares
Heuristic confidence scores, and what replaces them
Most LLM stacks gate automation on a confidence number: a softmax probability, a verbalized score, or a single-model confidence API. Here is what changes when that number becomes a calibrated procedure.
| Dimension | Heuristic confidence scores | Conformal Logit Inference |
|---|---|---|
| What you receive | A single number between 0 and 1 | A set, an interval, a probability interval, or a decision, each with a guarantee card |
| What is promised | Nothing formal. “Calibrated” describes past behaviour in aggregate. | Coverage, risk, FDR, cost, or human-agreement bounds that hold marginally, or per group, over exchangeable data |
| Evidence behind it | Rarely stated | Calibration profile, sample size, and last audit date on every guaranteed answer; sets and intervals add an interval for realized coverage |
| Choosing the threshold | Picked by hand, often by reading a few examples | Computed by a calibration procedure against the risk level you set |
| Many decisions at once | No control over the error rate across a batch | False-discovery-rate control across a batch, including e-BH under arbitrary dependence |
| After a model update | Keeps being applied, silently | Anytime-valid monitors raise an alarm with a controlled false-alarm rate |
| Which models | Tied to one model or one vendor’s single-model confidence API | Any closed API, self-hosted engine, or your own model, with the same guarantee shape |
| Labelling cost | Not applicable: nothing is calibrated | Prediction-powered inference combines a small human-labelled sample with many judge labels |
| Where it runs | Wherever the model vendor serves it | Hosted API, your own VPC, or a fully offline statistics package |
What neither approach gives you is a guarantee about one specific decision in isolation. CLI states every guarantee over the population of exchangeable decisions, and says so on the answer itself.