What the numbers mean

Every number Kno reports is built to survive a skeptical reader. Three rules do most of the work.

1. Deltas carry intervals, or they are not reported

A delta without its confidence interval is a point estimate pretending to be truth. Kno reports paired 95% intervals. If a sample is too small to produce a meaningful one, the run says underpowered and reports no delta at all. A blank is a better answer than a bad number.

2. The holdout stays sealed

A third of your dev set is reserved as a control reserve before routing, and the holdout is sealed at ingestion. Selection is free to overfit the dev set, that is what selection does. The holdout is the number that cannot have been peeked at, and it is the one you may put in a slide.

3. The winner’s curse is named, not hidden

When you select the best of many candidates, the best looks better than it is. Kno corrects for the multiplicity, Bonferroni-corrected intervals on every keep/reject decision, and then says the quiet part out loud: the portfolio interval is still inflated by the selection effect. The honest number is the validate report’s holdout gain.

What the numbers do not claim

• A delta is not a prediction. It is a measurement on your cases, your agent, your goal.
• “No harm found” is bounded by min_detectable_harm, small samples cannot rule out small regressions.
• Selection-time gains are estimates, inflated by design.

The full treatment: What the numbers mean.