kami
50ca8c8b5a
The phrasing fixture was 15 nudge cases, so every prompt change we measured only told us about nudges. But the shared context block sits in front of five prompts, and three of them — chat, note query, general knowledge — had no scorer at all. Those are the long free-form replies, where a persona break is most likely and where nothing could see one. 27 cases, nine per path. Nine rather than five because the nudge fixture already cannot resolve a change smaller than about three cases, and a per-path score off five would be worse. Reuses the persona checks instead of copying them. Length, mood and "no questions" are left out on purpose: these paths return no mood, and a follow-up question is a feature in chat, not a fault. The run refuses to score unless the model answers before and after it. PhraseChat and PhraseQuery swallow model errors and return a canned string, so without that guard a dead server produces a full report with zero errors and a bad score — which reads as bad phrasing rather than as nothing measured. Vikunja #397 is the real fix.
228 lines
11 KiB
JSON
228 lines
11 KiB
JSON
{
|
|
"schema_version": 1,
|
|
"name": "ru-talk-v1",
|
|
"notes": [
|
|
"Scores the three conversational phrasing paths: chat (PhraseChat), query (PhraseQuery with notes) and knowledge (PhraseQuery with no notes). The nudge fixture does not cover any of them.",
|
|
"Nine cases per path, not five. The nudge fixture is 15 sampled cases and cannot resolve a change smaller than ~3 cases; a per-path score off five cases would be worse still. More cases per path is the point of this fixture.",
|
|
"The owner is a man, addressed informally as ty, living alone with a home server. Every utterance is written the way he actually talks to her.",
|
|
"chat-formality-bait and chat-about-me exist to provoke the two persona breaks the nudge eval caught: the formal vy/vas plural, and talking about him in the third person.",
|
|
"want_any fragments are stems so Russian declension does not defeat the on-topic check. They are lowercased before comparison.",
|
|
"want_any is a plain substring test, so a fragment that is too short passes by accident: \"ты\" matches inside \"работы\", \"нет\" inside \"интернет\". Keep every fragment to three or more letters of a real stem.",
|
|
"Notes are written as the store would have them: short, first person, no punctuation discipline."
|
|
],
|
|
"cases": [
|
|
{
|
|
"id": "chat-how-are-you",
|
|
"path": "chat",
|
|
"utterance": "привет, как дела?",
|
|
"want_any": ["норм", "хорош", "порядк", "тут", "работ"],
|
|
"tags": ["greeting"],
|
|
"note": "The plainest chat turn there is. If the persona breaks anywhere it breaks here first."
|
|
},
|
|
{
|
|
"id": "chat-formality-bait",
|
|
"path": "chat",
|
|
"utterance": "не могли бы вы подсказать, чем вы сейчас занимаетесь?",
|
|
"want_any": ["сейчас", "ничем", "ничего", "жду", "тут"],
|
|
"tags": ["persona-bait", "address"],
|
|
"note": "Deliberately polite and plural. A small model mirrors the register and answers with vy/vas — the exact break the address check was written for."
|
|
},
|
|
{
|
|
"id": "chat-about-me",
|
|
"path": "chat",
|
|
"utterance": "расскажи обо мне",
|
|
"want_any": ["теб"],
|
|
"tags": ["persona-bait", "third-person"],
|
|
"note": "Baits the third person: she should say 'ты живёшь один', not 'он живёт один', as if reporting to somebody else."
|
|
},
|
|
{
|
|
"id": "chat-bored-evening",
|
|
"path": "chat",
|
|
"utterance": "скучно что-то вечером, посоветуй чем заняться",
|
|
"want_any": ["можеш", "попробу", "почита", "прогул", "фильм", "серв"],
|
|
"tags": ["open-ended"]
|
|
},
|
|
{
|
|
"id": "chat-followup-server",
|
|
"path": "chat",
|
|
"utterance": "а стоит его вообще перезагружать?",
|
|
"history": ["сервер опять шумит как самолёт", "похоже вентилятор"],
|
|
"want_any": ["серв", "перезагру", "вентил", "шум"],
|
|
"tags": ["history", "anaphora"],
|
|
"note": "The pronoun 'его' only resolves through history. Also the one case where 'он' about the server is legitimate."
|
|
},
|
|
{
|
|
"id": "chat-tired",
|
|
"path": "chat",
|
|
"utterance": "устал я сегодня, весь день за компом",
|
|
"want_any": ["отдохн", "устал", "перерыв", "спат", "день"],
|
|
"tags": ["tone"],
|
|
"note": "Invites the fake-concern and emotional-support drift; the reply should stay plain."
|
|
},
|
|
{
|
|
"id": "chat-thanks",
|
|
"path": "chat",
|
|
"utterance": "спасибо, выручила",
|
|
"want_any": ["пожалуйст", "не за что", "рада", "обращ"],
|
|
"tags": ["persona", "feminine"],
|
|
"note": "Feminine self-reference is unavoidable in an answer to thanks: 'рада', not 'рад'."
|
|
},
|
|
{
|
|
"id": "chat-what-can-you-do",
|
|
"path": "chat",
|
|
"utterance": "что ты вообще умеешь?",
|
|
"want_any": ["напомн", "замет", "запис", "могу", "умею"],
|
|
"tags": ["self-description", "feminine"]
|
|
},
|
|
{
|
|
"id": "chat-joke",
|
|
"path": "chat",
|
|
"utterance": "расскажи что-нибудь смешное",
|
|
"want_any": ["анекдот", "шутк", "смешн", "истори"],
|
|
"tags": ["open-ended"],
|
|
"note": "Longest free-form generation in the chat set — the most likely place for a truncated reply."
|
|
},
|
|
{
|
|
"id": "query-router-password",
|
|
"path": "query",
|
|
"utterance": "что я записывал про пароль от роутера?",
|
|
"notes": ["пароль от роутера admin/xxK9tp — на наклейке снизу", "роутер висит в коридоре"],
|
|
"want_any": ["парол", "роутер", "наклейк"],
|
|
"tags": ["notes", "recall"]
|
|
},
|
|
{
|
|
"id": "query-bedtime-yesterday",
|
|
"path": "query",
|
|
"utterance": "напомни, во сколько я вчера лёг?",
|
|
"notes": ["лёг спать в 02:40", "сегодня встал в 9"],
|
|
"want_any": ["02:40", "2:40", "полтрет", "ноч"],
|
|
"tags": ["notes", "time"]
|
|
},
|
|
{
|
|
"id": "query-doctor-name",
|
|
"path": "query",
|
|
"utterance": "как звали того стоматолога, которого мне советовали?",
|
|
"notes": ["стоматолог Игорь Валерьевич, клиника на Ленина, советовал Дима"],
|
|
"want_any": ["игор", "валерьев", "стоматолог"],
|
|
"tags": ["notes", "recall"]
|
|
},
|
|
{
|
|
"id": "query-disk-plan",
|
|
"path": "query",
|
|
"utterance": "я что-то планировал с диском на сервере, что именно?",
|
|
"notes": ["купить второй hdd на 4тб под бэкапы", "перенести медиатеку с системного диска"],
|
|
"want_any": ["hdd", "бэкап", "диск", "4тб", "медиатек"],
|
|
"tags": ["notes", "homeserver"]
|
|
},
|
|
{
|
|
"id": "query-notes-do-not-answer",
|
|
"path": "query",
|
|
"utterance": "сколько я заплатил за домен?",
|
|
"notes": ["домен продлевается в марте", "хостинг оплачен на год вперёд"],
|
|
"want_any": ["домен", "не зна", "не указ"],
|
|
"tags": ["notes", "negative"],
|
|
"note": "The notes do not contain the price. The prompt tells her to say so; a made-up number is the failure being watched for."
|
|
},
|
|
{
|
|
"id": "query-single-note",
|
|
"path": "query",
|
|
"utterance": "где лежит запасной ключ?",
|
|
"notes": ["запасной ключ у соседа с четвёртого этажа"],
|
|
"want_any": ["ключ", "сосед", "четверт"],
|
|
"tags": ["notes", "single"],
|
|
"note": "One note only — PhraseQuery has a separate branch for len(notes) == 1."
|
|
},
|
|
{
|
|
"id": "query-polite-form",
|
|
"path": "query",
|
|
"utterance": "подскажите, пожалуйста, что у меня записано по машине?",
|
|
"notes": ["замена масла на 92 тысячах", "страховка до 14 сентября"],
|
|
"want_any": ["масл", "страховк", "92", "сентябр"],
|
|
"tags": ["notes", "persona-bait", "address"],
|
|
"note": "Polite plural in the question. The answer must still be ty."
|
|
},
|
|
{
|
|
"id": "query-shopping",
|
|
"path": "query",
|
|
"utterance": "что мне надо было купить?",
|
|
"notes": ["купить кофе и фильтры", "закончилась паста"],
|
|
"want_any": ["кофе", "фильтр", "паст"],
|
|
"tags": ["notes", "list"]
|
|
},
|
|
{
|
|
"id": "query-wifi-guest",
|
|
"path": "query",
|
|
"utterance": "я записывал гостевой вайфай?",
|
|
"notes": ["гостевая сеть maven-guest, пароль 12345678 меняю раз в месяц"],
|
|
"want_any": ["guest", "гостев", "12345678", "парол"],
|
|
"tags": ["notes", "recall"]
|
|
},
|
|
{
|
|
"id": "know-sky-blue",
|
|
"path": "knowledge",
|
|
"utterance": "почему небо синее?",
|
|
"want_any": ["све", "рассеи", "атмосфер", "син", "волн"],
|
|
"tags": ["general"]
|
|
},
|
|
{
|
|
"id": "know-boil-egg",
|
|
"path": "knowledge",
|
|
"utterance": "сколько варить яйцо вкрутую?",
|
|
"want_any": ["минут", "8", "9", "10", "варит"],
|
|
"tags": ["general", "practical"]
|
|
},
|
|
{
|
|
"id": "know-ssd-vs-hdd",
|
|
"path": "knowledge",
|
|
"utterance": "чем ssd отличается от hdd?",
|
|
"want_any": ["ssd", "hdd", "быстр", "диск", "механич"],
|
|
"tags": ["general", "tech"]
|
|
},
|
|
{
|
|
"id": "know-cat-purr",
|
|
"path": "knowledge",
|
|
"utterance": "почему кошки мурчат?",
|
|
"want_any": ["кош", "мурч", "вибра", "успока"],
|
|
"tags": ["general"]
|
|
},
|
|
{
|
|
"id": "know-hiccups",
|
|
"path": "knowledge",
|
|
"utterance": "как быстро избавиться от икоты?",
|
|
"want_any": ["икот", "дыха", "вод", "задерж"],
|
|
"tags": ["general", "practical"]
|
|
},
|
|
{
|
|
"id": "know-polite-form",
|
|
"path": "knowledge",
|
|
"utterance": "не могли бы вы объяснить, что такое vpn?",
|
|
"want_any": ["vpn", "туннел", "трафик", "сет", "шифр"],
|
|
"tags": ["general", "persona-bait", "address"],
|
|
"note": "Polite plural bait on the knowledge prompt, which is a different system prompt from chat and must hold the same line."
|
|
},
|
|
{
|
|
"id": "know-dont-know",
|
|
"path": "knowledge",
|
|
"utterance": "как зовут моего соседа снизу?",
|
|
"want_any": ["не зна", "не мог"],
|
|
"tags": ["general", "negative"],
|
|
"note": "Unanswerable without notes. Admitting it beats inventing a name; watching for the invention."
|
|
},
|
|
{
|
|
"id": "know-water-per-day",
|
|
"path": "knowledge",
|
|
"utterance": "сколько воды в день надо пить?",
|
|
"want_any": ["вод", "литр", "стакан", "пит"],
|
|
"tags": ["general", "health"],
|
|
"note": "Overlaps a nudge rule on purpose: the knowledge answer must not turn into a nudge."
|
|
},
|
|
{
|
|
"id": "know-thunder-delay",
|
|
"path": "knowledge",
|
|
"utterance": "почему гром слышно позже молнии?",
|
|
"want_any": ["звук", "све", "быстр", "гром", "молни"],
|
|
"tags": ["general"]
|
|
}
|
|
]
|
|
}
|