label seeds with the stage 0 grammars and gemma, and measure both (V-546)

The plan calls the labeled set the whole project and names the stage 0
grammars as the label functions. cmd/labelgen runs them, the real ones in
buildRouter order, so a rule change moves the training data with it.

Gemma labels the rest at 334ms/call with nothing unparsed, which matches the
plan's estimate. It agrees with the seed files on 197/277, and reading the
disagreements is the finding: the seeds and the router prompt hold different
definitions of system, of a world question and of a bare verb. V-626.
This commit is contained in:
2026-08-06 13:21:06 +04:00
parent fa67dd82fe
commit 2b3e34c7e8
2 changed files with 195 additions and 0 deletions
@@ -0,0 +1,82 @@
# Gemma as a label function, and what it found in the seeds
**06-08-2026. V-546.** Measured on workpc against gemma-4-12b-it-qat-UD-Q4_K_XL.
`docs/plans/18-routing-heads-on-e5-small.md` puts the labeled set at 20k examples through
gemma, costing 2 to 4 hours of the card. This is the check before spending that. Gemma
labels the 344 hand-written classifier seeds. Agreement with the label a person already
chose is a precision number rather than a guess.
## What ran
`cmd/labelgen` runs the stage 0 grammars. The real ones, in `buildRouter` order, minus
`wakeword-act`, whose allowlist is a deployment's enabled tool names. It labels 62 of 339
seed lines and leaves the rest.
The remaining 277 went to gemma through the daemon's own `routeSystem` prompt and
`routeGrammar`, both extracted from `internal/router/llmrouter.go` at run time rather than
retyped. Temperature 0.
## Cost
**334ms per call, 0 unparsed of 277.** The GBNF held every time. At that rate the plan's
20k examples is under two hours of card, which matches its estimate.
## The stage 0 rules as label functions
Agreement between the grammar's label and the seed file the line came from:
| seed intent | agree |
|---|---|
| reminder | 37/37 |
| query | 9/10 |
| system | 7/8 |
| act | 2/2 |
| chat | 0/4 |
| note | 0/1 |
`ReminderGrammar` at 37/37 is the evidence the plan wanted. The chat column is a defect
rather than a disagreement: `chatNarrativeTopics` is Russian-only, so `tell me about
yourself` survives the decline and routes IntentQuery with topic `yourself`. Filed as
V-625, which also records that `как дела у сервера` appears verbatim in two seed files
under two intents.
## Gemma against the seeds
**197/277, 71.1%.** By intent:
| seed intent | agree |
|---|---|
| note | 33/33 |
| act | 57/64 |
| fact | 37/40 |
| query | 51/54 |
| chat | 15/35 |
| system | 4/43 |
| reminder | 0/8 |
The number is not gemma's error rate. Reading the 80 disagreements, most are the seed files
and the prompt holding different definitions of the same intent. Three boundaries carry 42
of them, and V-626 is the fix:
- **system, 26 lines.** The prompt restricts system to the clock, the calendar date and the
assistant itself. The seeds also put sensor and host state there. That is the V-374 edit
of 31-07-2026, which the seeds never received.
- **world questions, 8 lines.** `почему небо голубое`, `why is the sky blue`. Written when
chat was the only honest destination for a question nothing could answer, and external
search now answers them.
- **bare verbs, 8 lines.** `поставь напоминание` with nothing to remind about. The prompt
calls that unknown. This one is not staleness. A nearest-neighbour centroid wants the
bare verb phrase, and that is what a seed file is for.
Four intents have not been redefined since the seeds were written: note, fact, query and
act. They agree at 178 of 191.
## What this says about the plan
Gemma is usable as a label function on those four and not on system, chat or a bare verb.
The plan already budgets a day of the owner reading the set. This says where to spend it.
It also says the two engines in the cascade are being taught different rules on 80 lines.
A routing measurement that swaps between the classifier and the router is measuring some of
that disagreement rather than the models.