Abstention
An abstention is a declined verdict: A8 returns no answer for a criterion whose evidence cannot support the accuracy the request asked for. It is the mechanism that gives an answer its meaning — the rate attached to answered requests holds because the cases the evidence does not cover leave the answered set rather than entering it as guesses.
One number, min_accuracy, sets the terms of that trade.
§1What an abstention is
The response carries no verdict. A top-level abstention says why, and at what level the same request is answered. The request succeeded: the status is 200 (reference).
| Field | Meaning |
|---|---|
kind | Which of four things happened. See §2.2. |
asked | The min_accuracy the request asked for. |
answerable_at | The highest min_accuracy at which this same request is answered. 0 means it is answered only with no promise. null means no level answers it. |
earned | The highest level the criterion has earned, or null where it has earned none. |
missing | The standards the criterion was taught against that the request did not send, as a list of reference, context or question. null where the request lacked none. |
Table 1. The fields of an abstention. Send the request again with min_accuracy set to answerable_at and it returns a verdict.
The same thing in words sits where each API puts an answer the model declined to give: message.refusal in Chat Completions. That text is for reading. Branch on kind.
Where a schema asks for several fields, each is judged on its own criterion (reference), so some can answer while others decline. The declined fields are null in the verdict, and the abstentions arrive keyed by field:
§2Setting and earning an accuracy level
min_accuracy is one integer from 0 to 99, default 90 (reference). It is the accuracy a criterion’s evidence must demonstrate before A8 returns a verdict.
| Value | Meaning |
|---|---|
1–99 | Answer only where at least that share of answers is correct. Higher values abstain more often. |
0 | No promise: answer wherever a reading exists, with nothing promised and no interval. |
Table 2. The parameter takes one integer from 0 to 99, and the two ranges mean different things.
0 is the same scale read at its bottom: requiring 0% accuracy requires nothing, so nothing is withheld.
§2.1The earned set is uneven
Each accuracy level is earned separately, per criterion, on data held out of fine-tuning — which is why a brand-new criterion declines at every level, and why the levels a criterion has earned form an uneven set. Evidence accumulates in whole samples, so achievable rates move in steps: a criterion can support 90 and genuinely not 91.
An ask therefore lands on the nearest earned level at or above it; where the criterion has earned no such level, it declines. Every answer reports the level it was served at, in min_accuracy on the response body, so a request for 91 may be answered at 95:
You do not have to discover those levels by probing. Settings → Criteria lists what each of your criteria has earned, so you can see which one to teach more before raising the number you ask for.
Asking for more than a criterion has earned is a useful probe: the abstentions that appear at 95 but not at 90 measure how much evidence that criterion still needs.
§2.2The four ways a criterion declines
Four kinds, each with a different remedy.
kind | What happened | What clears it |
|---|---|---|
no_evidence | Nothing to judge against: a criterion with no examples and no standard supplied in the request, or a criterion taught against a standard the request did not send. | Send the standards named in missing. Where it is null, fine-tune the criterion, or supply a reference or context in the request. |
level_not_earned | The criterion has evidence, but not enough to back the level asked for. | Fine-tune it, or ask at answerable_at. |
unfamiliar_subject | This subject sits outside the material the criterion has seen, so the promise does not cover it. | Fine-tune on material like it. |
too_close | The criterion has earned the level and the subject is familiar. This case sat too close to the line. | Nothing, usually. Asking at answerable_at answers it. |
Table 3. Four things produce an abstention, and each points at a different remedy.
The first three concern the model’s evidence and shrink with fine-tuning. The fourth concerns the subject and persists at some rate at every level above 0.
§3What the promise means
At min_accuracy: N, of the requests A8 answers, at least N% are correct. The quantifier ranges over the answered set — not over every request submitted, and not over any single verdict.
| What it says | What it does not say |
|---|---|
| Across the verdicts returned at accuracy level 90, at least 90% are right. | That any particular verdict is 90% likely to be right. |
| The rate holds on what was answered. | Anything about how much gets answered. Coverage is the cost of the level, not part of the promise. |
Table 4. The promise ranges over answered requests only; these are the two readings it does not license.
A verdict carries no probability. It is a determination A8 returned because the evidence behind that criterion meets the requested accuracy. “Correct” reads two ways, depending on the verdict’s shape:
| Verdict | Correct means |
|---|---|
| A label — a category or a yes/no | The label matches. At level 90, at least 90% of returned labels are the right one. |
| A score — a number on the schema’s declared scale | The returned interval contains the true value. At level 90, at least 90% of intervals do. |
Table 5. “Correct” is defined by the verdict’s shape: a label matches, or an interval contains the true value.
The trade is coverage for accuracy: min_accuracy raises the share of answers that are correct by lowering the share of requests that are answered. What that buys is separation. Every eval model meets material it cannot judge well, and one that must always answer resolves those cases by producing something anyway — output indistinguishable from the confident cases. No component generates a fallback verdict here when the evidence runs out; no language model runs on the response path to improvise one (see determinism). The hard cases therefore arrive as abstentions, and the answers that do arrive carry a rate that can be measured against the promise.
§3.1Auditing the promise
The served level is on every answer so the promise can be audited. Run samples whose right answer is already known, and group the answers by the level they were served at; within each group, the share that came out right should clear that group’s number.
- Abstentions leave the denominator. Scoring them as errors measures a different quantity: the accuracy rate is defined over the answered set only.
- Track coverage separately. The share of requests answered is what a level costs, and it is the number to watch when raising one.
- Group by the served level, not the asked one. They differ whenever an ask resolved upward.
- Small counts move. A 90% promise can show 87% on thirty answers without having been broken. Compare against an interval, not a point.
A group that misses its number over enough answers to mean something is a defect in the promise, and worth reporting.
§4Edge cases
| Case | Behaviour |
|---|---|
min_accuracy: 0 | Every verdict answers where a reading exists, with nothing promised and no interval. None of §3 applies. |
| A brand-new criterion | Declines at every level above 0 until fine-tuning earns one. |
| A scored verdict near the boundary | Answers rather than declining: a score honours the level by interval width. The width states how far the evidence constrains the value, so there is no width threshold to configure. |
| A criterion whose traffic has shifted | Levels earned on the old material can be withdrawn, and every level above 0 declines until fine-tuning on the new material re-earns them. |
Table 6. Four situations produce behaviour that the general rule does not describe.
The last is the promise’s one standing condition: incoming requests must resemble the material the level was earned on. Pointed at a genuinely different population — a new domain, another language, output from a different system — a criterion’s evidence no longer describes the subjects under judgment.
§5Related concepts
- Overview — the shape of a call, and where
min_accuracysits in it. - Fine-tuning — how a criterion earns its levels.
- Reference — the wire contract and status codes.