diff --git a/CLAUDE.md b/CLAUDE.md index dc9ad69..f878bc6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -257,6 +257,31 @@ Recall is 15/15 and world is 5/5. Intent is 93.6% mean over three seeds. That is **not** comparable to the 76.0% and 84.4% those two arms scored: a softmax has no clarify class, so the head's fixture is the 88 cases carrying an intent. +**A fourth head asks instead of guessing, same day** (V-661, +`docs/evals/2026-08-08-clarify-head-four-head.md`). Clarify is not a value +of intent, so a softmax cannot emit it. It is a second question over the +same pooled vector: can Maven act on this at all. That is why the head's +fixture was 88 cases and not 96. Over three seeds it catches **7.0 of the 8 +`want_clarify` cases and produces 2.3 false clarifies of 88**. The cascade +today misses 1 and produces 2, so this is parity with no rules in front of +it. Accuracy is the wrong number here and a head that never asks scores +91.7%. Confidence is the other half. Max softmax over the intent head reads +**0.851 where it is right against 0.604 where it is wrong**, ranking right +above wrong in 83.4% of pairs. `Confidence: 1.0` was a hardcode, and this +replaces it with a signal. The two are not the same signal: one says which +intent is unclear, the other says the utterance carries too little to act +on. **The fourth head is not free the way the third was.** Intent, +destination and slot F1 each move down one to four points, inside the seed +spread. `поужинал` is a false clarify on every seed, which is the same +defect `thinSingleToken` was narrowed for on 2026-08-01. + +The corpus for it is generated, because every existing row is answerable by +construction. **The router-prompt agreement filter cannot work here**, since +`routeGrammar` has no clarify value and a generated line always agrees with +itself. A gemma judge replaces it. The first judge called 24 of 40 +answerable rows underspecified, because it judged against a generic +assistant rather than against Maven's contract. + **Mood is cut, not deferred.** The enum describes her own reply state, not the speaker's emotion, and no dataset maps onto it. diff --git a/docs/evals/2026-08-08-clarify-head-four-head.md b/docs/evals/2026-08-08-clarify-head-four-head.md index e47c32d..9a23dc3 100644 --- a/docs/evals/2026-08-08-clarify-head-four-head.md +++ b/docs/evals/2026-08-08-clarify-head-four-head.md @@ -92,7 +92,7 @@ fix. than an error. The utterance is underspecified, and V-498 decided stage 0 claims it for the calendar on purpose. -`ну это` is missed on two seeds. `амб-003` is the shortest case in the fixture +`ну это` is missed on two seeds. `amb-003` is the shortest case in the fixture and the generated demonstratives are longer. ## Confidence