kami
c7c44229a2
#319 asks for a measurement before #320 flips the route decider from the classifier cascade to the resident model. There was nothing to measure against: the only routing tests assert single utterances, and the classifier's seed corpus is its own training set — scoring it there measures memorisation of frozen centroids, which is the illusion that hid the weak RU query handling in the first place. internal/router/eval is a separate package so both paths can be scored from outside router (including cmd/mavend, where the real llama-server client lives). The fixture is embedded; the scorer takes a Router interface, so *router.Router and a bare LLM stage both go through the same 76 cases. The fixture is a CONTRACT, not a snapshot: cases the cascade fails today stay in the file and fail loudly. TestFixtureIsHeldOut enforces that no utterance appears verbatim in models/seeds/*.txt. Baseline, hash embedder at the deployed 0.55 gate: 9/76 (11.8%), 63 false clarifies, 0 missed clarifies, p50 9µs. Almost everything falls to the confidence gate — the documented floor behaviour, not a new bug. The number worth comparing is TestONNXBaseline's (skipped without MAVEN_ONNX_LIB); the assertions here are a regression ratchet plus a tight bound on the dangerous direction: ambiguous utterances must not start being routed confidently. Seeding is order-fixed on purpose — a few phrases appear under two intents and map iteration handed them to a different centroid each run, which made the score jitter between 9 and 10. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01X5JApcrCRVGmqrxnhynSik
97 lines
12 KiB
JSON
97 lines
12 KiB
JSON
{
|
|
"schema_version": 1,
|
|
"name": "ru_routing_v1",
|
|
"reference_now": "2026-07-30T12:00:00Z",
|
|
"notes": [
|
|
"Held-out routing contract. Every utterance here is absent from models/seeds/*.txt (TestFixtureIsHeldOut enforces it verbatim) — scoring a classifier on its own seed phrases measures memorisation, not routing.",
|
|
"This is a CONTRACT, not a snapshot of current behaviour. Cases the classifier cascade fails today are expected to stay in the file and fail loudly; that failure count is the number Vikunja #319 compares against the LLM router before #320 flips the default.",
|
|
"Slot expectations are deployment-independent on purpose. want_fn is a boolean (the act must resolve to SOME allowlisted fn) because the allowlist lives in deploy config, not here. want_fact_key names the loop's rule keys (water/meal/sleep/break/shower) — a fact that lands under the wrong key silently starves the predicate that reads it.",
|
|
"want_clarify cases carry intent \"\": the contract is that the router refuses rather than guesses. A confident answer there is a worse failure than a miss."
|
|
],
|
|
"cases": [
|
|
{ "id": "ru-query-001", "utterance": "сколько воды я выпил с утра", "lang": "ru", "intent": "query", "tags": ["aggregate"] },
|
|
{ "id": "ru-query-002", "utterance": "я сегодня вообще пил воду", "lang": "ru", "intent": "query", "tags": ["hard", "fact-shaped"], "note": "past-tense fact lexicon in a question — the classifier's fact centroid pulls this hard" },
|
|
{ "id": "ru-query-003", "utterance": "во сколько я лёг вчера", "lang": "ru", "intent": "query", "tags": ["temporal"] },
|
|
{ "id": "ru-query-004", "utterance": "давно я не тренировался", "lang": "ru", "intent": "query", "tags": ["hard", "no-question-word"] },
|
|
{ "id": "ru-query-005", "utterance": "напоминания на завтра есть", "lang": "ru", "intent": "query", "tags": ["hard", "reminder-shaped"], "note": "asks about reminders, does not create one" },
|
|
{ "id": "ru-query-006", "utterance": "что я записывал про кота", "lang": "ru", "intent": "query", "tags": ["recall"] },
|
|
{ "id": "ru-query-007", "utterance": "сколько раз я ел вчера", "lang": "ru", "intent": "query", "tags": ["aggregate", "hard"] },
|
|
{ "id": "ru-query-008", "utterance": "мой вес за последний месяц", "lang": "ru", "intent": "query", "tags": ["no-verb"] },
|
|
{ "id": "ru-query-009", "utterance": "когда я в последний раз принимал витамины", "lang": "ru", "intent": "query", "tags": ["temporal"] },
|
|
{ "id": "ru-query-010", "utterance": "есть новости по бэкапу базы", "lang": "ru", "intent": "query", "tags": ["homelab"] },
|
|
{ "id": "ru-query-011", "utterance": "почему сервер тормозит", "lang": "ru", "intent": "query", "tags": ["homelab", "hard"], "note": "diagnostic question, not a chat opener" },
|
|
{ "id": "ru-query-012", "utterance": "какие заметки я оставил про полив", "lang": "ru", "intent": "query", "tags": ["recall"] },
|
|
{ "id": "ru-query-013", "utterance": "во сколько у меня встреча", "lang": "ru", "intent": "query", "tags": ["calendar"] },
|
|
{ "id": "ru-query-014", "utterance": "я успеваю до дедлайна", "lang": "ru", "intent": "query", "tags": ["hard", "no-question-word"] },
|
|
{ "id": "ru-query-015", "utterance": "сколько я прошёл шагов", "lang": "ru", "intent": "query", "tags": ["aggregate"] },
|
|
{ "id": "ru-query-016", "utterance": "покажи давление за неделю", "lang": "ru", "intent": "query", "tags": ["hard", "imperative"], "note": "imperative form but a read — must not route to act" },
|
|
{ "id": "ru-query-017", "utterance": "чем я занимался в среду", "lang": "ru", "intent": "query", "tags": ["hard", "chat-shaped"] },
|
|
{ "id": "ru-query-018", "utterance": "хватает ли места под новые бэкапы", "lang": "ru", "intent": "query", "tags": ["homelab"] },
|
|
{ "id": "en-query-001", "utterance": "did I take my vitamins today", "lang": "en", "intent": "query", "tags": ["fact-shaped"] },
|
|
{ "id": "en-query-002", "utterance": "how long since the last backup finished", "lang": "en", "intent": "query", "tags": ["temporal"] },
|
|
{ "id": "en-query-003", "utterance": "show me this week's weight", "lang": "en", "intent": "query", "tags": ["imperative"] },
|
|
|
|
{ "id": "ru-fact-001", "utterance": "только что выпил кружку воды", "lang": "ru", "intent": "fact", "want_fact_key": "water" },
|
|
{ "id": "ru-fact-002", "utterance": "воды попил наконец", "lang": "ru", "intent": "fact", "want_fact_key": "water", "tags": ["inverted"] },
|
|
{ "id": "ru-fact-003", "utterance": "поужинал", "lang": "ru", "intent": "fact", "want_fact_key": "meal", "tags": ["single-word"] },
|
|
{ "id": "ru-fact-004", "utterance": "отметь что я позавтракал овсянкой", "lang": "ru", "intent": "fact", "want_fact_key": "meal" },
|
|
{ "id": "ru-fact-005", "utterance": "поспал часов пять", "lang": "ru", "intent": "fact", "want_fact_key": "sleep" },
|
|
{ "id": "ru-fact-006", "utterance": "сходил в душ", "lang": "ru", "intent": "fact", "want_fact_key": "shower" },
|
|
{ "id": "ru-fact-007", "utterance": "отдохнул минут двадцать", "lang": "ru", "intent": "fact", "want_fact_key": "break" },
|
|
{ "id": "ru-fact-008", "utterance": "запиши вес 74 килограмма", "lang": "ru", "intent": "fact", "tags": ["unparsed-key"], "note": "no recognizer for weight yet — intent must still be fact; HasKey false is allowed" },
|
|
{ "id": "ru-fact-009", "utterance": "отметь что я выпил таблетки утром", "lang": "ru", "intent": "fact", "tags": ["unparsed-key"] },
|
|
{ "id": "ru-fact-010", "utterance": "сделал зарядку двадцать минут", "lang": "ru", "intent": "fact", "tags": ["unparsed-key"] },
|
|
{ "id": "en-fact-001", "utterance": "just drank a glass of water", "lang": "en", "intent": "fact", "want_fact_key": "water" },
|
|
{ "id": "en-fact-002", "utterance": "slept about seven hours", "lang": "en", "intent": "fact", "want_fact_key": "sleep" },
|
|
|
|
{ "id": "ru-rem-001", "utterance": "напомни через сорок минут выключить свет", "lang": "ru", "intent": "reminder", "want_time": true },
|
|
{ "id": "ru-rem-002", "utterance": "напомни завтра в 7 позвонить в клинику", "lang": "ru", "intent": "reminder", "want_time": true },
|
|
{ "id": "ru-rem-003", "utterance": "не забудь напомнить мне про счёт за свет", "lang": "ru", "intent": "reminder", "tags": ["no-time"], "note": "no datetime — HasTime false is correct; the daemon asks for a time" },
|
|
{ "id": "ru-rem-004", "utterance": "поставь напоминание через полчаса", "lang": "ru", "intent": "reminder", "want_time": true },
|
|
{ "id": "ru-rem-005", "utterance": "разбуди меня в 6:30", "lang": "ru", "intent": "reminder", "want_time": true, "tags": ["hard"], "note": "wake-me phrasing, no напомни stem" },
|
|
{ "id": "ru-rem-006", "utterance": "напомни послезавтра в 12 забрать заказ", "lang": "ru", "intent": "reminder", "want_time": true },
|
|
{ "id": "ru-rem-007", "utterance": "через два часа напомни проверить бэкап", "lang": "ru", "intent": "reminder", "want_time": true, "tags": ["inverted"] },
|
|
{ "id": "en-rem-001", "utterance": "remind me in 45 minutes to stretch", "lang": "en", "intent": "reminder", "want_time": true },
|
|
{ "id": "en-rem-002", "utterance": "wake me at 6:15", "lang": "en", "intent": "reminder", "want_time": true, "tags": ["hard"] },
|
|
|
|
{ "id": "ru-act-001", "utterance": "перезапусти докер", "lang": "ru", "intent": "act", "want_fn": true, "tags": ["homelab", "destructive"] },
|
|
{ "id": "ru-act-002", "utterance": "выключи свет в спальне", "lang": "ru", "intent": "act", "want_fn": true },
|
|
{ "id": "ru-act-003", "utterance": "maven, останови бэкап", "lang": "ru", "intent": "act", "want_fn": true, "tags": ["wake-token"] },
|
|
{ "id": "ru-act-004", "utterance": "включи вытяжку", "lang": "ru", "intent": "act", "want_fn": true },
|
|
{ "id": "ru-act-005", "utterance": "запусти бэкап сейчас", "lang": "ru", "intent": "act", "want_fn": true },
|
|
{ "id": "ru-act-006", "utterance": "закрой жалюзи", "lang": "ru", "intent": "act", "want_fn": true },
|
|
{ "id": "en-act-001", "utterance": "maven, restart the media server", "lang": "en", "intent": "act", "want_fn": true, "tags": ["wake-token"] },
|
|
{ "id": "en-act-002", "utterance": "turn off the kitchen light", "lang": "en", "intent": "act", "want_fn": true },
|
|
|
|
{ "id": "ru-note-001", "utterance": "заметка: продлить домен в августе", "lang": "ru", "intent": "note" },
|
|
{ "id": "ru-note-002", "utterance": "запиши что кран на кухне снова капает", "lang": "ru", "intent": "note" },
|
|
{ "id": "ru-note-003", "utterance": "заметка про настройку vlan на свитче", "lang": "ru", "intent": "note", "tags": ["homelab"] },
|
|
{ "id": "ru-note-004", "utterance": "запиши идею: гидропоника на балконе", "lang": "ru", "intent": "note" },
|
|
{ "id": "ru-note-005", "utterance": "запиши что сосед просил номер электрика", "lang": "ru", "intent": "note" },
|
|
{ "id": "en-note-001", "utterance": "note: rotate the kuma api key", "lang": "en", "intent": "note", "tags": ["homelab"] },
|
|
|
|
{ "id": "ru-sys-001", "utterance": "сколько сейчас времени в киеве", "lang": "ru", "intent": "system", "tags": ["time"] },
|
|
{ "id": "ru-sys-002", "utterance": "какое число завтра", "lang": "ru", "intent": "system", "tags": ["date"] },
|
|
{ "id": "ru-sys-003", "utterance": "переходи в тихий режим", "lang": "ru", "intent": "system", "tags": ["quiet"] },
|
|
{ "id": "ru-sys-004", "utterance": "хватит тихого режима", "lang": "ru", "intent": "system", "tags": ["quiet", "hard"] },
|
|
{ "id": "ru-sys-005", "utterance": "какой день недели послезавтра", "lang": "ru", "intent": "system", "tags": ["date"] },
|
|
{ "id": "en-sys-001", "utterance": "what time is it now", "lang": "en", "intent": "system", "tags": ["time"] },
|
|
{ "id": "en-sys-002", "utterance": "turn quiet mode back on", "lang": "en", "intent": "system", "tags": ["quiet"] },
|
|
|
|
{ "id": "ru-chat-001", "utterance": "мне немного грустно сегодня", "lang": "ru", "intent": "chat" },
|
|
{ "id": "ru-chat-002", "utterance": "что думаешь про переезд на другую квартиру", "lang": "ru", "intent": "chat" },
|
|
{ "id": "ru-chat-003", "utterance": "расскажи анекдот про программистов", "lang": "ru", "intent": "chat" },
|
|
{ "id": "ru-chat-004", "utterance": "доброе утро", "lang": "ru", "intent": "chat", "tags": ["greeting"] },
|
|
{ "id": "ru-chat-005", "utterance": "спасибо тебе", "lang": "ru", "intent": "chat" },
|
|
{ "id": "en-chat-001", "utterance": "good morning", "lang": "en", "intent": "chat", "tags": ["greeting"] },
|
|
{ "id": "en-chat-002", "utterance": "i had a rough day", "lang": "en", "intent": "chat" },
|
|
|
|
{ "id": "amb-001", "utterance": "вода", "lang": "ru", "want_clarify": true, "tags": ["ambiguous"], "note": "fact write or a query? one noun decides nothing — ask" },
|
|
{ "id": "amb-002", "utterance": "бэкап", "lang": "ru", "want_clarify": true, "tags": ["ambiguous"], "note": "run it, or report on it? a confident act here is destructive" },
|
|
{ "id": "amb-003", "utterance": "ну это", "lang": "ru", "want_clarify": true, "tags": ["ambiguous", "filler"] },
|
|
{ "id": "amb-004", "utterance": "сделай это", "lang": "ru", "want_clarify": true, "tags": ["ambiguous", "anaphora"], "note": "unresolved anaphora with an imperative — must not guess an fn" },
|
|
{ "id": "amb-005", "utterance": "потом", "lang": "ru", "want_clarify": true, "tags": ["ambiguous", "filler"] },
|
|
{ "id": "amb-006", "utterance": "the thing from earlier", "lang": "en", "want_clarify": true, "tags": ["ambiguous", "anaphora"] }
|
|
]
|
|
}
|