claude
2b3e34c7e8
The plan calls the labeled set the whole project and names the stage 0 grammars as the label functions. cmd/labelgen runs them, the real ones in buildRouter order, so a rule change moves the training data with it. Gemma labels the rest at 334ms/call with nothing unparsed, which matches the plan's estimate. It agrees with the seed files on 197/277, and reading the disagreements is the finding: the seeds and the router prompt hold different definitions of system, of a world question and of a bare verb. V-626.
83 lines
3.3 KiB
Markdown
83 lines
3.3 KiB
Markdown
# Gemma as a label function, and what it found in the seeds
|
|
|
|
**06-08-2026. V-546.** Measured on workpc against gemma-4-12b-it-qat-UD-Q4_K_XL.
|
|
|
|
`docs/plans/18-routing-heads-on-e5-small.md` puts the labeled set at 20k examples through
|
|
gemma, costing 2 to 4 hours of the card. This is the check before spending that. Gemma
|
|
labels the 344 hand-written classifier seeds. Agreement with the label a person already
|
|
chose is a precision number rather than a guess.
|
|
|
|
## What ran
|
|
|
|
`cmd/labelgen` runs the stage 0 grammars. The real ones, in `buildRouter` order, minus
|
|
`wakeword-act`, whose allowlist is a deployment's enabled tool names. It labels 62 of 339
|
|
seed lines and leaves the rest.
|
|
|
|
The remaining 277 went to gemma through the daemon's own `routeSystem` prompt and
|
|
`routeGrammar`, both extracted from `internal/router/llmrouter.go` at run time rather than
|
|
retyped. Temperature 0.
|
|
|
|
## Cost
|
|
|
|
**334ms per call, 0 unparsed of 277.** The GBNF held every time. At that rate the plan's
|
|
20k examples is under two hours of card, which matches its estimate.
|
|
|
|
## The stage 0 rules as label functions
|
|
|
|
Agreement between the grammar's label and the seed file the line came from:
|
|
|
|
| seed intent | agree |
|
|
|---|---|
|
|
| reminder | 37/37 |
|
|
| query | 9/10 |
|
|
| system | 7/8 |
|
|
| act | 2/2 |
|
|
| chat | 0/4 |
|
|
| note | 0/1 |
|
|
|
|
`ReminderGrammar` at 37/37 is the evidence the plan wanted. The chat column is a defect
|
|
rather than a disagreement: `chatNarrativeTopics` is Russian-only, so `tell me about
|
|
yourself` survives the decline and routes IntentQuery with topic `yourself`. Filed as
|
|
V-625, which also records that `как дела у сервера` appears verbatim in two seed files
|
|
under two intents.
|
|
|
|
## Gemma against the seeds
|
|
|
|
**197/277, 71.1%.** By intent:
|
|
|
|
| seed intent | agree |
|
|
|---|---|
|
|
| note | 33/33 |
|
|
| act | 57/64 |
|
|
| fact | 37/40 |
|
|
| query | 51/54 |
|
|
| chat | 15/35 |
|
|
| system | 4/43 |
|
|
| reminder | 0/8 |
|
|
|
|
The number is not gemma's error rate. Reading the 80 disagreements, most are the seed files
|
|
and the prompt holding different definitions of the same intent. Three boundaries carry 42
|
|
of them, and V-626 is the fix:
|
|
|
|
- **system, 26 lines.** The prompt restricts system to the clock, the calendar date and the
|
|
assistant itself. The seeds also put sensor and host state there. That is the V-374 edit
|
|
of 31-07-2026, which the seeds never received.
|
|
- **world questions, 8 lines.** `почему небо голубое`, `why is the sky blue`. Written when
|
|
chat was the only honest destination for a question nothing could answer, and external
|
|
search now answers them.
|
|
- **bare verbs, 8 lines.** `поставь напоминание` with nothing to remind about. The prompt
|
|
calls that unknown. This one is not staleness. A nearest-neighbour centroid wants the
|
|
bare verb phrase, and that is what a seed file is for.
|
|
|
|
Four intents have not been redefined since the seeds were written: note, fact, query and
|
|
act. They agree at 178 of 191.
|
|
|
|
## What this says about the plan
|
|
|
|
Gemma is usable as a label function on those four and not on system, chat or a bare verb.
|
|
The plan already budgets a day of the owner reading the set. This says where to spend it.
|
|
|
|
It also says the two engines in the cascade are being taught different rules on 80 lines.
|
|
A routing measurement that swaps between the classifier and the router is measuring some of
|
|
that disagreement rather than the models.
|