move the seed files onto the router prompt's intent boundaries (V-626)

The classifier learns models/seeds and the router is prompted with
routeSystem, and they held different definitions on 80 lines. Sensor and
host state was system in the seeds and is query in the prompt, which is the
V-374 edit the seeds never received. World questions were chat, written
before external search could answer them.

64/91 to 66/91 on the fixture. en-sys-002 and ru-query-011 gain, nothing
regresses, clarify counts unchanged.

The third disagreement is measured and rejected. Dropping the eight bare
reminder verbs scores 65, because a centroid is a shape to be near and the
bare verb phrase is part of that shape. A seed file and a prompt have
different jobs there.
This commit is contained in:
2026-08-06 13:26:23 +04:00
parent b86172a98d
commit e7537d032e
4 changed files with 93 additions and 35 deletions
@@ -0,0 +1,58 @@
# Moving the seed files onto the router prompt's boundaries
**06-08-2026. V-626.** Measured with `TestONNXBaseline`, 91-case RU routing fixture,
classifier plus the ONNX embedder. No LLM arm in this run.
`docs/evals/2026-08-06-seed-labels-vs-router-prompt.md` found three intent boundaries where
`models/seeds` and `routeSystem` disagree. This applies two of them and rejects the third,
because the third was measured and it costs a case.
## Baseline
**64/91, 70.3% full.** Latency p50 22.9ms.
## What moved
**Sensor and host state, system to query. 26 lines.** `какая температура воздуха`,
`сколько памяти занято`, `какой статус сервисов`. The prompt restricts system to the clock,
the calendar date and the assistant itself, which is the V-374 edit of 31-07-2026.
**World questions, chat to query. 8 lines.** `почему небо голубое`, `why is the sky blue`,
`как работает интернет`. Only the genuine world-knowledge lines. An opener about herself
stays in chat. `как тебя зовут` is a question word by rule 4 and about the assistant by
rule 8. The rules are ordered and rule 4 fires first, which reads wrong. That is a prompt
question rather than a seed question.
`system.txt` goes from 43 lines to 17 and `query.txt` from 64 to 98.
## Result
**66/91, 72.5% full.** Two cases gained, none lost.
- `en-sys-002` "turn quiet mode back on", quiet 2/3 to 3/3
- `ru-query-011` "почему сервер тормозит", homelab 5/6 to 6/6
Clarify counts unchanged at 0 false and 8 missed. The eight missed clarifies are the
`ambiguous` tag and this change does not touch them. `TestONNXRecall`, `TestONNXTopics`,
`TestONNXPersonalBoundary` and `TestONNXClaimConfidenceDistribution` all pass.
Thinning system to 17 lines did not hurt it. The two remaining system failures,
`какое число завтра` and `какой день недели послезавтра`, both failed at baseline too.
## The third boundary, measured and rejected
`reminder.txt` holds eight bare verbs: `поставь напоминание`, `создай напоминание`,
`set a reminder`. Rule 9 of the prompt calls an utterance with no named subject unknown.
By the prompt they do not belong in a reminder seed set.
Dropping them scores **65/91**, one below keeping them. `ru-rem-004` "поставь напоминание
через полчаса" falls from reminder to fact, because the centroid loses the phrase the
utterance is built from.
So the seed file and the prompt are not stale against each other here. They have different
jobs. A prompt classifies one utterance and can say it cannot. A nearest-neighbour centroid
is a shape to be near, and a bare verb phrase is part of that shape. The eight lines stay.
That distinction matters past this file. V-546 trains a classification head on labeled
utterances rather than a centroid, and the head is the prompt's kind of thing. These eight
lines are seed data and not training data.