claude
e7537d032e
The classifier learns models/seeds and the router is prompted with routeSystem, and they held different definitions on 80 lines. Sensor and host state was system in the seeds and is query in the prompt, which is the V-374 edit the seeds never received. World questions were chat, written before external search could answer them. 64/91 to 66/91 on the fixture. en-sys-002 and ru-query-011 gain, nothing regresses, clarify counts unchanged. The third disagreement is measured and rejected. Dropping the eight bare reminder verbs scores 65, because a centroid is a shape to be near and the bare verb phrase is part of that shape. A seed file and a prompt have different jobs there.
3.0 KiB
Moving the seed files onto the router prompt's boundaries
06-08-2026. V-626. Measured with TestONNXBaseline, 91-case RU routing fixture,
classifier plus the ONNX embedder. No LLM arm in this run.
docs/evals/2026-08-06-seed-labels-vs-router-prompt.md found three intent boundaries where
models/seeds and routeSystem disagree. This applies two of them and rejects the third,
because the third was measured and it costs a case.
Baseline
64/91, 70.3% full. Latency p50 22.9ms.
What moved
Sensor and host state, system to query. 26 lines. какая температура воздуха,
сколько памяти занято, какой статус сервисов. The prompt restricts system to the clock,
the calendar date and the assistant itself, which is the V-374 edit of 31-07-2026.
World questions, chat to query. 8 lines. почему небо голубое, why is the sky blue,
как работает интернет. Only the genuine world-knowledge lines. An opener about herself
stays in chat. как тебя зовут is a question word by rule 4 and about the assistant by
rule 8. The rules are ordered and rule 4 fires first, which reads wrong. That is a prompt
question rather than a seed question.
system.txt goes from 43 lines to 17 and query.txt from 64 to 98.
Result
66/91, 72.5% full. Two cases gained, none lost.
en-sys-002"turn quiet mode back on", quiet 2/3 to 3/3ru-query-011"почему сервер тормозит", homelab 5/6 to 6/6
Clarify counts unchanged at 0 false and 8 missed. The eight missed clarifies are the
ambiguous tag and this change does not touch them. TestONNXRecall, TestONNXTopics,
TestONNXPersonalBoundary and TestONNXClaimConfidenceDistribution all pass.
Thinning system to 17 lines did not hurt it. The two remaining system failures,
какое число завтра and какой день недели послезавтра, both failed at baseline too.
The third boundary, measured and rejected
reminder.txt holds eight bare verbs: поставь напоминание, создай напоминание,
set a reminder. Rule 9 of the prompt calls an utterance with no named subject unknown.
By the prompt they do not belong in a reminder seed set.
Dropping them scores 65/91, one below keeping them. ru-rem-004 "поставь напоминание
через полчаса" falls from reminder to fact, because the centroid loses the phrase the
utterance is built from.
So the seed file and the prompt are not stale against each other here. They have different jobs. A prompt classifies one utterance and can say it cannot. A nearest-neighbour centroid is a shape to be near, and a bare verb phrase is part of that shape. The eight lines stay.
That distinction matters past this file. V-546 trains a classification head on labeled utterances rather than a centroid, and the head is the prompt's kind of thing. These eight lines are seed data and not training data.