claude
35018226ef
Nine reply cases and a fourth column in the talk report. The reply path is a separate object from the phraser in the daemon, so Pair joins a Talker and a Confirmer for a run that covers everything Maven says. Cases carry intent/key/value because the replier is phrased from the decision the router resolved, not from the raw utterance. Three of them are baits the other paths cannot produce: a masculine verb about himself that she must not copy onto herself, a polite plural input that must still come back на ты, and an unresolved note that invites a question a confirmation is not allowed to ask. Not scored against a model here — this box has no llama-server, and the baseline test is opt-in on MAVEN_LLM_URL.
312 lines
14 KiB
JSON
312 lines
14 KiB
JSON
{
|
|
"schema_version": 1,
|
|
"name": "ru-talk-v1",
|
|
"notes": [
|
|
"Scores the three conversational phrasing paths: chat (PhraseChat), query (PhraseQuery with notes) and knowledge (PhraseQuery with no notes). The nudge fixture does not cover any of them.",
|
|
"Nine cases per path, not five. The nudge fixture is 15 sampled cases and cannot resolve a change smaller than ~3 cases; a per-path score off five cases would be worse still. More cases per path is the point of this fixture.",
|
|
"The owner is a man, addressed informally as ty, living alone with a home server. Every utterance is written the way he actually talks to her.",
|
|
"chat-formality-bait and chat-about-me exist to provoke the two persona breaks the nudge eval caught: the formal vy/vas plural, and talking about him in the third person.",
|
|
"want_any fragments are stems so Russian declension does not defeat the on-topic check. They are lowercased before comparison.",
|
|
"want_any is a plain substring test, so a fragment that is too short passes by accident: \"ты\" matches inside \"работы\", \"нет\" inside \"интернет\". Keep every fragment to three or more letters of a real stem.",
|
|
"Notes are written as the store would have them: short, first person, no punctuation discipline."
|
|
],
|
|
"cases": [
|
|
{
|
|
"id": "chat-how-are-you",
|
|
"path": "chat",
|
|
"utterance": "привет, как дела?",
|
|
"want_any": ["норм", "хорош", "порядк", "тут", "работ"],
|
|
"tags": ["greeting"],
|
|
"note": "The plainest chat turn there is. If the persona breaks anywhere it breaks here first."
|
|
},
|
|
{
|
|
"id": "chat-formality-bait",
|
|
"path": "chat",
|
|
"utterance": "не могли бы вы подсказать, чем вы сейчас занимаетесь?",
|
|
"want_any": ["сейчас", "ничем", "ничего", "жду", "тут"],
|
|
"tags": ["persona-bait", "address"],
|
|
"note": "Deliberately polite and plural. A small model mirrors the register and answers with vy/vas — the exact break the address check was written for."
|
|
},
|
|
{
|
|
"id": "chat-about-me",
|
|
"path": "chat",
|
|
"utterance": "расскажи обо мне",
|
|
"want_any": ["теб"],
|
|
"tags": ["persona-bait", "third-person"],
|
|
"note": "Baits the third person: she should say 'ты живёшь один', not 'он живёт один', as if reporting to somebody else."
|
|
},
|
|
{
|
|
"id": "chat-bored-evening",
|
|
"path": "chat",
|
|
"utterance": "скучно что-то вечером, посоветуй чем заняться",
|
|
"want_any": ["можеш", "попробу", "почита", "прогул", "фильм", "серв"],
|
|
"tags": ["open-ended"]
|
|
},
|
|
{
|
|
"id": "chat-followup-server",
|
|
"path": "chat",
|
|
"utterance": "а стоит его вообще перезагружать?",
|
|
"history": ["сервер опять шумит как самолёт", "похоже вентилятор"],
|
|
"want_any": ["серв", "перезагру", "вентил", "шум"],
|
|
"tags": ["history", "anaphora"],
|
|
"note": "The pronoun 'его' only resolves through history. Also the one case where 'он' about the server is legitimate."
|
|
},
|
|
{
|
|
"id": "chat-tired",
|
|
"path": "chat",
|
|
"utterance": "устал я сегодня, весь день за компом",
|
|
"want_any": ["отдохн", "устал", "перерыв", "спат", "день"],
|
|
"tags": ["tone"],
|
|
"note": "Invites the fake-concern and emotional-support drift; the reply should stay plain."
|
|
},
|
|
{
|
|
"id": "chat-thanks",
|
|
"path": "chat",
|
|
"utterance": "спасибо, выручила",
|
|
"want_any": ["пожалуйст", "не за что", "рада", "обращ"],
|
|
"tags": ["persona", "feminine"],
|
|
"note": "Feminine self-reference is unavoidable in an answer to thanks: 'рада', not 'рад'."
|
|
},
|
|
{
|
|
"id": "chat-what-can-you-do",
|
|
"path": "chat",
|
|
"utterance": "что ты вообще умеешь?",
|
|
"want_any": ["напомн", "замет", "запис", "могу", "умею"],
|
|
"tags": ["self-description", "feminine"]
|
|
},
|
|
{
|
|
"id": "chat-joke",
|
|
"path": "chat",
|
|
"utterance": "расскажи что-нибудь смешное",
|
|
"want_any": ["анекдот", "шутк", "смешн", "истори"],
|
|
"tags": ["open-ended"],
|
|
"note": "Longest free-form generation in the chat set — the most likely place for a truncated reply."
|
|
},
|
|
{
|
|
"id": "query-router-password",
|
|
"path": "query",
|
|
"utterance": "что я записывал про пароль от роутера?",
|
|
"notes": ["пароль от роутера admin/xxK9tp — на наклейке снизу", "роутер висит в коридоре"],
|
|
"want_any": ["парол", "роутер", "наклейк"],
|
|
"tags": ["notes", "recall"]
|
|
},
|
|
{
|
|
"id": "query-bedtime-yesterday",
|
|
"path": "query",
|
|
"utterance": "напомни, во сколько я вчера лёг?",
|
|
"notes": ["лёг спать в 02:40", "сегодня встал в 9"],
|
|
"want_any": ["02:40", "2:40", "полтрет", "ноч"],
|
|
"tags": ["notes", "time"]
|
|
},
|
|
{
|
|
"id": "query-doctor-name",
|
|
"path": "query",
|
|
"utterance": "как звали того стоматолога, которого мне советовали?",
|
|
"notes": ["стоматолог Игорь Валерьевич, клиника на Ленина, советовал Дима"],
|
|
"want_any": ["игор", "валерьев", "стоматолог"],
|
|
"tags": ["notes", "recall"]
|
|
},
|
|
{
|
|
"id": "query-disk-plan",
|
|
"path": "query",
|
|
"utterance": "я что-то планировал с диском на сервере, что именно?",
|
|
"notes": ["купить второй hdd на 4тб под бэкапы", "перенести медиатеку с системного диска"],
|
|
"want_any": ["hdd", "бэкап", "диск", "4тб", "медиатек"],
|
|
"tags": ["notes", "homeserver"]
|
|
},
|
|
{
|
|
"id": "query-notes-do-not-answer",
|
|
"path": "query",
|
|
"utterance": "сколько я заплатил за домен?",
|
|
"notes": ["домен продлевается в марте", "хостинг оплачен на год вперёд"],
|
|
"want_any": ["домен", "не зна", "не указ"],
|
|
"tags": ["notes", "negative"],
|
|
"note": "The notes do not contain the price. The prompt tells her to say so; a made-up number is the failure being watched for."
|
|
},
|
|
{
|
|
"id": "query-single-note",
|
|
"path": "query",
|
|
"utterance": "где лежит запасной ключ?",
|
|
"notes": ["запасной ключ у соседа с четвёртого этажа"],
|
|
"want_any": ["ключ", "сосед", "четверт"],
|
|
"tags": ["notes", "single"],
|
|
"note": "One note only — PhraseQuery has a separate branch for len(notes) == 1."
|
|
},
|
|
{
|
|
"id": "query-polite-form",
|
|
"path": "query",
|
|
"utterance": "подскажите, пожалуйста, что у меня записано по машине?",
|
|
"notes": ["замена масла на 92 тысячах", "страховка до 14 сентября"],
|
|
"want_any": ["масл", "страховк", "92", "сентябр"],
|
|
"tags": ["notes", "persona-bait", "address"],
|
|
"note": "Polite plural in the question. The answer must still be ty."
|
|
},
|
|
{
|
|
"id": "query-shopping",
|
|
"path": "query",
|
|
"utterance": "что мне надо было купить?",
|
|
"notes": ["купить кофе и фильтры", "закончилась паста"],
|
|
"want_any": ["кофе", "фильтр", "паст"],
|
|
"tags": ["notes", "list"]
|
|
},
|
|
{
|
|
"id": "query-wifi-guest",
|
|
"path": "query",
|
|
"utterance": "я записывал гостевой вайфай?",
|
|
"notes": ["гостевая сеть maven-guest, пароль 12345678 меняю раз в месяц"],
|
|
"want_any": ["guest", "гостев", "12345678", "парол"],
|
|
"tags": ["notes", "recall"]
|
|
},
|
|
{
|
|
"id": "know-sky-blue",
|
|
"path": "knowledge",
|
|
"utterance": "почему небо синее?",
|
|
"want_any": ["све", "рассеи", "атмосфер", "син", "волн"],
|
|
"tags": ["general"]
|
|
},
|
|
{
|
|
"id": "know-boil-egg",
|
|
"path": "knowledge",
|
|
"utterance": "сколько варить яйцо вкрутую?",
|
|
"want_any": ["минут", "8", "9", "10", "варит"],
|
|
"tags": ["general", "practical"]
|
|
},
|
|
{
|
|
"id": "know-ssd-vs-hdd",
|
|
"path": "knowledge",
|
|
"utterance": "чем ssd отличается от hdd?",
|
|
"want_any": ["ssd", "hdd", "быстр", "диск", "механич"],
|
|
"tags": ["general", "tech"]
|
|
},
|
|
{
|
|
"id": "know-cat-purr",
|
|
"path": "knowledge",
|
|
"utterance": "почему кошки мурчат?",
|
|
"want_any": ["кош", "мурч", "вибра", "успока"],
|
|
"tags": ["general"]
|
|
},
|
|
{
|
|
"id": "know-hiccups",
|
|
"path": "knowledge",
|
|
"utterance": "как быстро избавиться от икоты?",
|
|
"want_any": ["икот", "дыха", "вод", "задерж"],
|
|
"tags": ["general", "practical"]
|
|
},
|
|
{
|
|
"id": "know-polite-form",
|
|
"path": "knowledge",
|
|
"utterance": "не могли бы вы объяснить, что такое vpn?",
|
|
"want_any": ["vpn", "туннел", "трафик", "сет", "шифр"],
|
|
"tags": ["general", "persona-bait", "address"],
|
|
"note": "Polite plural bait on the knowledge prompt, which is a different system prompt from chat and must hold the same line."
|
|
},
|
|
{
|
|
"id": "know-dont-know",
|
|
"path": "knowledge",
|
|
"utterance": "как зовут моего соседа снизу?",
|
|
"want_any": ["не зна", "не мог"],
|
|
"tags": ["general", "negative"],
|
|
"note": "Unanswerable without notes. Admitting it beats inventing a name; watching for the invention."
|
|
},
|
|
{
|
|
"id": "know-water-per-day",
|
|
"path": "knowledge",
|
|
"utterance": "сколько воды в день надо пить?",
|
|
"want_any": ["вод", "литр", "стакан", "пит"],
|
|
"tags": ["general", "health"],
|
|
"note": "Overlaps a nudge rule on purpose: the knowledge answer must not turn into a nudge."
|
|
},
|
|
{
|
|
"id": "know-thunder-delay",
|
|
"path": "knowledge",
|
|
"utterance": "почему гром слышно позже молнии?",
|
|
"want_any": ["звук", "све", "быстр", "гром", "молни"],
|
|
"tags": ["general"]
|
|
},
|
|
{
|
|
"id": "reply-fact-coffee",
|
|
"path": "reply",
|
|
"intent": "fact",
|
|
"key": "кофе",
|
|
"value": "закончился",
|
|
"utterance": "кофе закончился",
|
|
"want_any": ["коф"],
|
|
"tags": ["fact"],
|
|
"note": "The plainest confirmation there is, and the sentence he hears most often."
|
|
},
|
|
{
|
|
"id": "reply-fact-weight",
|
|
"path": "reply",
|
|
"intent": "fact",
|
|
"key": "вес",
|
|
"value": "82",
|
|
"utterance": "мой вес 82",
|
|
"want_any": ["вес", "82"],
|
|
"tags": ["fact", "number"],
|
|
"note": "A number must survive into the confirmation; a paraphrase that drops it is useless."
|
|
},
|
|
{
|
|
"id": "reply-fact-pill",
|
|
"path": "reply",
|
|
"intent": "fact",
|
|
"key": "таблетки",
|
|
"value": "выпил",
|
|
"utterance": "таблетки выпил",
|
|
"want_any": ["таблетк"],
|
|
"tags": ["fact", "feminine"],
|
|
"note": "He says 'выпил', masculine and about himself. She must not copy the form onto herself."
|
|
},
|
|
{
|
|
"id": "reply-note-router",
|
|
"path": "reply",
|
|
"intent": "note",
|
|
"utterance": "роутер перезагружается сам по ночам",
|
|
"want_any": ["роутер"],
|
|
"tags": ["note"]
|
|
},
|
|
{
|
|
"id": "reply-note-long",
|
|
"path": "reply",
|
|
"intent": "note",
|
|
"utterance": "если диск снова отвалится, посмотреть кабель, а не контроллер, в прошлый раз был кабель",
|
|
"want_any": ["диск", "кабел"],
|
|
"tags": ["note", "length"],
|
|
"note": "A long note baits a long confirmation. One sentence is the contract."
|
|
},
|
|
{
|
|
"id": "reply-reminder-evening",
|
|
"path": "reply",
|
|
"intent": "reminder",
|
|
"utterance": "напомни вечером полить цветы",
|
|
"want_any": ["цвет", "полит", "вечер"],
|
|
"tags": ["reminder"]
|
|
},
|
|
{
|
|
"id": "reply-reminder-tomorrow",
|
|
"path": "reply",
|
|
"intent": "reminder",
|
|
"utterance": "напомни завтра позвонить в поликлинику",
|
|
"want_any": ["поликлиник", "позвон", "звон"],
|
|
"tags": ["reminder"]
|
|
},
|
|
{
|
|
"id": "reply-formality-bait",
|
|
"path": "reply",
|
|
"intent": "note",
|
|
"utterance": "запишите пожалуйста что счётчики я сдал",
|
|
"want_any": ["счётчик", "счетчик"],
|
|
"tags": ["note", "persona-bait", "address"],
|
|
"note": "Polite plural in the input. The confirmation must still be на ты."
|
|
},
|
|
{
|
|
"id": "reply-question-bait",
|
|
"path": "reply",
|
|
"intent": "note",
|
|
"utterance": "надо купить фильтр для воды, не помню какой",
|
|
"want_any": ["фильтр"],
|
|
"tags": ["note", "no-question"],
|
|
"note": "An unresolved note invites her to ask which filter. A confirmation does not ask."
|
|
}
|
|
]
|
|
}
|