label seeds with the stage 0 grammars and gemma, and measure both (V-546)
The plan calls the labeled set the whole project and names the stage 0 grammars as the label functions. cmd/labelgen runs them, the real ones in buildRouter order, so a rule change moves the training data with it. Gemma labels the rest at 334ms/call with nothing unparsed, which matches the plan's estimate. It agrees with the seed files on 197/277, and reading the disagreements is the finding: the seeds and the router prompt hold different definitions of system, of a world question and of a bare verb. V-626.
This commit is contained in:
@@ -0,0 +1,82 @@
|
||||
# Gemma as a label function, and what it found in the seeds
|
||||
|
||||
**06-08-2026. V-546.** Measured on workpc against gemma-4-12b-it-qat-UD-Q4_K_XL.
|
||||
|
||||
`docs/plans/18-routing-heads-on-e5-small.md` puts the labeled set at 20k examples through
|
||||
gemma, costing 2 to 4 hours of the card. This is the check before spending that. Gemma
|
||||
labels the 344 hand-written classifier seeds. Agreement with the label a person already
|
||||
chose is a precision number rather than a guess.
|
||||
|
||||
## What ran
|
||||
|
||||
`cmd/labelgen` runs the stage 0 grammars. The real ones, in `buildRouter` order, minus
|
||||
`wakeword-act`, whose allowlist is a deployment's enabled tool names. It labels 62 of 339
|
||||
seed lines and leaves the rest.
|
||||
|
||||
The remaining 277 went to gemma through the daemon's own `routeSystem` prompt and
|
||||
`routeGrammar`, both extracted from `internal/router/llmrouter.go` at run time rather than
|
||||
retyped. Temperature 0.
|
||||
|
||||
## Cost
|
||||
|
||||
**334ms per call, 0 unparsed of 277.** The GBNF held every time. At that rate the plan's
|
||||
20k examples is under two hours of card, which matches its estimate.
|
||||
|
||||
## The stage 0 rules as label functions
|
||||
|
||||
Agreement between the grammar's label and the seed file the line came from:
|
||||
|
||||
| seed intent | agree |
|
||||
|---|---|
|
||||
| reminder | 37/37 |
|
||||
| query | 9/10 |
|
||||
| system | 7/8 |
|
||||
| act | 2/2 |
|
||||
| chat | 0/4 |
|
||||
| note | 0/1 |
|
||||
|
||||
`ReminderGrammar` at 37/37 is the evidence the plan wanted. The chat column is a defect
|
||||
rather than a disagreement: `chatNarrativeTopics` is Russian-only, so `tell me about
|
||||
yourself` survives the decline and routes IntentQuery with topic `yourself`. Filed as
|
||||
V-625, which also records that `как дела у сервера` appears verbatim in two seed files
|
||||
under two intents.
|
||||
|
||||
## Gemma against the seeds
|
||||
|
||||
**197/277, 71.1%.** By intent:
|
||||
|
||||
| seed intent | agree |
|
||||
|---|---|
|
||||
| note | 33/33 |
|
||||
| act | 57/64 |
|
||||
| fact | 37/40 |
|
||||
| query | 51/54 |
|
||||
| chat | 15/35 |
|
||||
| system | 4/43 |
|
||||
| reminder | 0/8 |
|
||||
|
||||
The number is not gemma's error rate. Reading the 80 disagreements, most are the seed files
|
||||
and the prompt holding different definitions of the same intent. Three boundaries carry 42
|
||||
of them, and V-626 is the fix:
|
||||
|
||||
- **system, 26 lines.** The prompt restricts system to the clock, the calendar date and the
|
||||
assistant itself. The seeds also put sensor and host state there. That is the V-374 edit
|
||||
of 31-07-2026, which the seeds never received.
|
||||
- **world questions, 8 lines.** `почему небо голубое`, `why is the sky blue`. Written when
|
||||
chat was the only honest destination for a question nothing could answer, and external
|
||||
search now answers them.
|
||||
- **bare verbs, 8 lines.** `поставь напоминание` with nothing to remind about. The prompt
|
||||
calls that unknown. This one is not staleness. A nearest-neighbour centroid wants the
|
||||
bare verb phrase, and that is what a seed file is for.
|
||||
|
||||
Four intents have not been redefined since the seeds were written: note, fact, query and
|
||||
act. They agree at 178 of 191.
|
||||
|
||||
## What this says about the plan
|
||||
|
||||
Gemma is usable as a label function on those four and not on system, chat or a bare verb.
|
||||
The plan already budgets a day of the owner reading the set. This says where to spend it.
|
||||
|
||||
It also says the two engines in the cascade are being taught different rules on 80 lines.
|
||||
A routing measurement that swaps between the classifier and the router is measuring some of
|
||||
that disagreement rather than the models.
|
||||
Reference in New Issue
Block a user