Score the conversational phrasing paths, not just nudges #37

2026-07-31T14:52:35+02:00

claude commented

2026-07-31 14:52:35 +02:00

Vikunja #395.

The phrasing fixture was 15 nudge cases. So every prompt change we have measured — the address rule, the examples experiment, the shared context block — only ever told us about nudges. Meanwhile that context block sits in front of five prompts, and three of them had no scorer at all: chat, note query, general knowledge. Those are the long free-form replies. They are where a persona break is most likely, and until this commit nothing could see one.

27 cases, nine per path. Nine and not five because the nudge fixture already cannot resolve a change smaller than about three cases, and a per-path score built on five would be worse than useless.

It reuses the persona checks rather than copying them. Length, mood and "no questions" are deliberately not applied: these paths return no mood at all, and a follow-up question is a feature in chat rather than a fault.

One thing worth knowing about the guard. PhraseChat and PhraseQuery swallow every model error and return a canned string with a nil error, so the Errors column on these paths is structurally always zero. Without a guard, a run against a dead server produces a full report with zero errors and a bad score — which reads as "the prompt is bad" when it means "nothing was measured". The run now refuses to score unless the model answers both before and after. Filed the real fix as #397; the fallback belongs to the daemon, not to the code that measures it.

Second commit is a fix to the address check that came straight out of a real eval reply: "Смотрите на его потребление воды" breaks in two ways at once, and only the plural imperative printed. The check returned on its first hit, and separately его was never detected — looksVerb knows the -й/-йте imperative but not the -те plural, so смотрите was counted as the person being talked about. That is the third time a verb form has blinded this check; a fourth means it needs a morphology table instead of another suffix.

Larger than the usual ~300 lines and I am flagging it rather than splitting it. The bulk is the 227-line JSON fixture, which is data you can skim by case. The Go is talk.go at 265 and its test at 163, and splitting the scorer from the fixture it scores would give you two PRs where neither runs.

Not in scope: the reply path, the confirmation you hear after every fact and reminder. It cannot be reached from here because llmReplier lives in package main. Filed as #396.

Vikunja #395. The phrasing fixture was 15 nudge cases. So every prompt change we have measured — the address rule, the examples experiment, the shared context block — only ever told us about nudges. Meanwhile that context block sits in front of five prompts, and three of them had no scorer at all: chat, note query, general knowledge. Those are the long free-form replies. They are where a persona break is most likely, and until this commit nothing could see one. 27 cases, nine per path. Nine and not five because the nudge fixture already cannot resolve a change smaller than about three cases, and a per-path score built on five would be worse than useless. It reuses the persona checks rather than copying them. Length, mood and "no questions" are deliberately not applied: these paths return no mood at all, and a follow-up question is a feature in chat rather than a fault. One thing worth knowing about the guard. `PhraseChat` and `PhraseQuery` swallow every model error and return a canned string with a nil error, so the Errors column on these paths is structurally always zero. Without a guard, a run against a dead server produces a full report with zero errors and a bad score — which reads as "the prompt is bad" when it means "nothing was measured". The run now refuses to score unless the model answers both before and after. Filed the real fix as #397; the fallback belongs to the daemon, not to the code that measures it. Second commit is a fix to the address check that came straight out of a real eval reply: `"Смотрите на его потребление воды"` breaks in two ways at once, and only the plural imperative printed. The check returned on its first hit, and separately `его` was never detected — `looksVerb` knows the -й/-йте imperative but not the -те plural, so `смотрите` was counted as the person being talked about. That is the third time a verb form has blinded this check; a fourth means it needs a morphology table instead of another suffix. **Larger than the usual ~300 lines** and I am flagging it rather than splitting it. The bulk is the 227-line JSON fixture, which is data you can skim by case. The Go is `talk.go` at 265 and its test at 163, and splitting the scorer from the fixture it scores would give you two PRs where neither runs. Not in scope: the reply path, the confirmation you hear after every fact and reminder. It cannot be reached from here because `llmReplier` lives in `package main`. Filed as #396.

kami added 2 commits 2026-07-31 14:52:57 +02:00

Score the chat, query and knowledge phrasing paths (#395 ) 50ca8c8b5a

The phrasing fixture was 15 nudge cases, so every prompt change we
measured only told us about nudges. But the shared context block sits in
front of five prompts, and three of them — chat, note query, general
knowledge — had no scorer at all. Those are the long free-form replies,
where a persona break is most likely and where nothing could see one.

27 cases, nine per path. Nine rather than five because the nudge fixture
already cannot resolve a change smaller than about three cases, and a
per-path score off five would be worse.

Reuses the persona checks instead of copying them. Length, mood and
"no questions" are left out on purpose: these paths return no mood, and
a follow-up question is a feature in chat, not a fault.

The run refuses to score unless the model answers before and after it.
PhraseChat and PhraseQuery swallow model errors and return a canned
string, so without that guard a dead server produces a full report with
zero errors and a bad score — which reads as bad phrasing rather than as
nothing measured. Vikunja #397 is the real fix.

Report every address break, and stop -те verbs blinding the check 0110e9bc8c

From a real reply in a nudge eval run: "Смотрите на его потребление
воды" is a plural imperative AND third person about him. Only the plural
printed.

Two separate faults. The check returned on its first hit, so the second
break stayed invisible and the failure read as milder than it was; it now
joins them. And "его" was not detected at all — looksVerb knows the
-й/-йте imperative but not the -те plural, so "смотрите" counted as the
person being talked about, which is what an antecedent means here.
pluralVerb already knows that form, so the antecedent test uses it too.

Third time a verb form has blinded this check. A fourth means it wants a
morphology table rather than another suffix.

kami force-pushed overnight/talk-eval from 64e5f3bdc1 to 0110e9bc8c

2026-07-31 14:52:57 +02:00

Compare

kami referenced this pull request

2026-07-31 19:37:41 +02:00

Tonight's work as one branch: all 46 reviewed PRs, verified green #47

kami commented

2026-07-31 20:22:32 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:33 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#37