124 lines
5.7 KiB
Markdown
124 lines
5.7 KiB
Markdown
# A clarify head, and a confidence that is not a hardcode
|
|
|
|
Measured 2026-08-08 on workpc, the same day and the same fixtures as
|
|
`2026-08-08-routing-heads-two-head.md` and `2026-08-08-slot-head-three-head.md`.
|
|
New script `gen_clarify.py`, new fixture `eval_fixture_clarify.jsonl`.
|
|
|
|
## A softmax has no clarify class
|
|
|
|
That sentence closed the two-head measurement. It is why the head's fixture was
|
|
88 cases and not 96. The eight `want_clarify` cases sat outside every number
|
|
measured, and the head had no way to produce the answer they wanted.
|
|
|
|
A fourth head is the answer. Clarify is not a value of intent. It is a second
|
|
question asked of the same pooled vector: can Maven act on this at all.
|
|
|
|
## The corpus had one class
|
|
|
|
Every row in `train_heads_slots.jsonl` was generated FOR an intent or a
|
|
destination. So every row is answerable by construction. A head trained on that
|
|
alone sees one class and learns to say yes.
|
|
|
|
`gen_clarify.py` makes the other class. Five shapes, ten topics. The shapes are
|
|
the gate's own reasons in `gateLLMDecision` plus the two the fixture carries:
|
|
bare noun, bare verb, demonstrative, deictic time, dangling reference.
|
|
|
|
**The agreement filter that worked for destination cannot work here.**
|
|
`routeGrammar` has no clarify value. So the router always names an intent, and
|
|
any generated line always agrees with itself. The second pass is a judge
|
|
instead. Gemma is asked, without seeing the label, whether Maven would have to
|
|
ask a question back.
|
|
|
|
## The first judge was worthless and the second was measured
|
|
|
|
The first judge said "needs clarify" on 24 of 40 plainly answerable corpus
|
|
rows. It flagged `запиши что я пообедал` and `Покажи расписание поездов на
|
|
вечер`. It was judging against a generic assistant, one that asks "where?"
|
|
about lunch. Maven writes that note.
|
|
|
|
Rewriting it to state what she can already do took false positives to 16 of 60.
|
|
It also catches all eight fixture clarifies. So the judge discriminates.
|
|
|
|
On the generated pile it removed 7 of 306, a 97.7% keep rate. That is not the
|
|
judge failing. The generator is aimed at underspecified lines, so there is
|
|
little for a filter to catch. The 27% false-positive rate is the number to
|
|
quote, and it is label noise on the positive class.
|
|
|
|
**Both passes are gemma-4-12b.** Generation and judging. So the corpus is
|
|
gemma's opinion of what is underspecified, and the head distills that opinion.
|
|
What keeps it honest is the fixture. Those eight cases were written by the owner
|
|
and gemma never saw them.
|
|
|
|
299 rows kept, against 3604 answerable. The positive class carries `intent:
|
|
null`, so it costs the intent head nothing.
|
|
|
|
## Result
|
|
|
|
Three seeds, 24 epochs, epoch still chosen on the intent dev slice.
|
|
|
|
| | two heads | three heads | four heads |
|
|
|---|---|---|---|
|
|
| intent mean | 93.6% | 92.8% | 91.7% |
|
|
| destination mean | 80.8% | 82.8% | 79.8% |
|
|
| slot span F1 mean | — | 72.4% | 68.3% |
|
|
| clarify caught | — | — | 7.0 of 8 |
|
|
| false clarifies | — | — | 2.3 of 88 |
|
|
|
|
**The fourth head is not free the way the third was.** Intent, destination and
|
|
slot F1 all move down. The drop is one to four points, and the seed spread is
|
|
wide enough to contain it. Seed 2 scores intent 94.3% and destination 84.8%, both above
|
|
every three-head seed. Read the drop as unproven rather than as absent.
|
|
|
|
Accuracy is the wrong number for this head and is reported for completeness at
|
|
95.8% to 96.9%. Eight of ninety-six cases are positive, so a head that never
|
|
asks scores 91.7%. Recall on those eight is the number.
|
|
|
|
Compare it to what ships. The cascade today misses 1 clarify and produces 2
|
|
false ones. The head catches 7 of 8 and produces 2.3 false ones. That is
|
|
parity, from a 118M encoder with no rules in front of it.
|
|
|
|
The saved checkpoint is seed 2 at epoch 10. Intent 94.3%, destination 28/33,
|
|
slot F1 73.6%, clarify 7 of 8 with 3 false. `heads.pt` carries four state dicts.
|
|
|
|
## What it gets wrong is consistent across seeds
|
|
|
|
`поужинал` is a false clarify on all three seeds. That utterance is already
|
|
recorded as a real defect. `thinSingleToken` was narrowed on 2026-08-01 to spare
|
|
a token carrying a Russian verb ending. One word is routinely a whole sentence
|
|
in Russian. The head relearned the mistake the rule was narrowed to
|
|
fix.
|
|
|
|
`что дальше?` is a false clarify on two seeds. That one is a disagreement rather
|
|
than an error. The utterance is underspecified, and V-498 decided stage 0 claims
|
|
it for the calendar on purpose.
|
|
|
|
`ну это` is missed on two seeds. `amb-003` is the shortest case in the fixture
|
|
and the generated demonstratives are longer.
|
|
|
|
## Confidence
|
|
|
|
`Confidence: 1.0` was a hardcode in `llmrouter.go`, so a correct low confidence
|
|
could not exist. Max softmax over the intent head is the replacement. It is only worth reading
|
|
if it is lower where the head is wrong.
|
|
|
|
It is. Mean 0.851 where the head is right against 0.604 where it is wrong. It
|
|
ranks a right case above a wrong one in 83.4% of pairs.
|
|
|
|
So there are two signals now and they are not the same signal. Confidence says
|
|
the head is unsure which intent this is. The clarify head says the utterance
|
|
does not carry enough to act on. A confident wrong route and an honest "I cannot
|
|
tell" are different failures, and one number cannot report both.
|
|
|
|
## What this does not measure
|
|
|
|
The same gap as every head run. **Nothing of this runs in Go.** Four heads
|
|
instead of three does not change that.
|
|
|
|
There is no threshold. Both signals are reported as raw numbers. Turning either
|
|
into a gate needs a decision about where to cut, and that trades false clarifies
|
|
against wrong acts. The fixture has 8 positives, which is too few to fit a
|
|
threshold on.
|
|
|
|
The 299 generated rows have no held-out slice of their own. Clarify is scored on
|
|
the fixture alone.
|