Confidence and significance

Why every number has a range, and how Onym decides a change is real.

AI answers vary. Ask the same question twice and you can get different brands. So a single answer tells you little, and a change from 31% to 34% might be nothing at all. Onym shows how sure each number is, and only calls a change real when the evidence supports it.

#Every rate has a sample size

Rates come with the counts behind them:

{ "k": 167, "n": 444, "rate": 0.376, "low": 0.332, "high": 0.422 }

k answers out of n named you. With n of 444 the range is about ±4.5 points. With n of 40 it would be about ±15.

#The 95% range

low and high are a 95% Wilson score interval: the range the true rate most likely sits in, given the answers so far. Onym uses Wilson rather than the simpler textbook formula because it stays sensible near 0% and 100% and with small samples. With no answers at all the range is 0 to 1.

#When a change is significant

A change between two periods is significant only when their ranges don't overlap. It's a strict test: it misses some small real changes, but it rarely raises a false alarm.

Onym also compares like with like. It only uses prompts and surfaces measured in both periods (pairs in the API). Adding ten new prompts, or a surface that failed for a day, can't make your number jump.

{
  "delta": 0.194,
  "significant": true,
  "direction": "up",
  "pairs": 111,
  "prev": { "k": 101, "n": 555, "rate": 0.182, "low": 0.152, "high": 0.216 },
  "cur": { "k": 167, "n": 444, "rate": 0.376, "low": 0.332, "high": 0.422 }
}

direction is flat unless the change is significant. change is null when there's no earlier period (with range=all) or nothing was measured in both.

#Ranks with ties

Leaderboards rank a brand by how many others are clearly ahead of it: one plus the number of brands whose entire range sits above its range. Brands whose ranges overlap share a rank.

#What this means for you

• Read the range, not only the rate. A rate on a small sample is a guess.
• Wait for significant before reporting a win or a loss. The Changes feed, email digest, Slack and webhooks already do.
• More prompts give tighter ranges sooner than more runs do.