Record the clarify head and the confidence it replaces (V-661)

This commit is contained in:
2026-08-08 20:59:44 +04:00
parent 3024f76e5f
commit c310115fd2
2 changed files with 26 additions and 1 deletions
+25
View File
@@ -257,6 +257,31 @@ Recall is 15/15 and world is 5/5. Intent is 93.6% mean over three seeds. That is
**not** comparable to the 76.0% and 84.4% those two arms scored: a softmax has no
clarify class, so the head's fixture is the 88 cases carrying an intent.
**A fourth head asks instead of guessing, same day** (V-661,
`docs/evals/2026-08-08-clarify-head-four-head.md`). Clarify is not a value
of intent, so a softmax cannot emit it. It is a second question over the
same pooled vector: can Maven act on this at all. That is why the head's
fixture was 88 cases and not 96. Over three seeds it catches **7.0 of the 8
`want_clarify` cases and produces 2.3 false clarifies of 88**. The cascade
today misses 1 and produces 2, so this is parity with no rules in front of
it. Accuracy is the wrong number here and a head that never asks scores
91.7%. Confidence is the other half. Max softmax over the intent head reads
**0.851 where it is right against 0.604 where it is wrong**, ranking right
above wrong in 83.4% of pairs. `Confidence: 1.0` was a hardcode, and this
replaces it with a signal. The two are not the same signal: one says which
intent is unclear, the other says the utterance carries too little to act
on. **The fourth head is not free the way the third was.** Intent,
destination and slot F1 each move down one to four points, inside the seed
spread. `поужинал` is a false clarify on every seed, which is the same
defect `thinSingleToken` was narrowed for on 2026-08-01.
The corpus for it is generated, because every existing row is answerable by
construction. **The router-prompt agreement filter cannot work here**, since
`routeGrammar` has no clarify value and a generated line always agrees with
itself. A gemma judge replaces it. The first judge called 24 of 40
answerable rows underspecified, because it judged against a generic
assistant rather than against Maven's contract.
**Mood is cut, not deferred.** The enum describes her own reply state, not the
speaker's emotion, and no dataset maps onto it.
@@ -92,7 +92,7 @@ fix.
than an error. The utterance is underspecified, and V-498 decided stage 0 claims
it for the calendar on purpose.
`ну это` is missed on two seeds. `амб-003` is the shortest case in the fixture
`ну это` is missed on two seeds. `amb-003` is the shortest case in the fixture
and the generated demonstratives are longer.
## Confidence