Score the chat, query and knowledge phrasing paths (#395)
The phrasing fixture was 15 nudge cases, so every prompt change we measured only told us about nudges. But the shared context block sits in front of five prompts, and three of them — chat, note query, general knowledge — had no scorer at all. Those are the long free-form replies, where a persona break is most likely and where nothing could see one. 27 cases, nine per path. Nine rather than five because the nudge fixture already cannot resolve a change smaller than about three cases, and a per-path score off five would be worse. Reuses the persona checks instead of copying them. Length, mood and "no questions" are left out on purpose: these paths return no mood, and a follow-up question is a feature in chat, not a fault. The run refuses to score unless the model answers before and after it. PhraseChat and PhraseQuery swallow model errors and return a canned string, so without that guard a dead server produces a full report with zero errors and a bad score — which reads as bad phrasing rather than as nothing measured. Vikunja #397 is the real fix.
This commit is contained in:
@@ -466,6 +466,7 @@ func checkAddress(body string) Result {
|
||||
fmt.Sprintf("third person %q with nobody else named — she talks to him, not about him", w)}
|
||||
}
|
||||
}
|
||||
|
||||
return Result{CheckAddress, true, ""}
|
||||
}
|
||||
|
||||
@@ -574,12 +575,47 @@ func checkCringe(body string) Result {
|
||||
// checkOnTopic — the message must name the thing the rule is about. A nudge
|
||||
// that never mentions water leaves the operator with a chime and no action.
|
||||
func checkOnTopic(c Case, body string) Result {
|
||||
return checkOnTopicAny(c.WantAny, body)
|
||||
}
|
||||
|
||||
// checkOnTopicAny is the same test over a bare want-list, so the talk scorer can
|
||||
// reuse it without owning a nudge Case.
|
||||
func checkOnTopicAny(wantAny []string, body string) Result {
|
||||
low := strings.ToLower(body)
|
||||
for _, want := range c.WantAny {
|
||||
for _, want := range wantAny {
|
||||
if strings.Contains(low, strings.ToLower(want)) {
|
||||
return Result{CheckOnTopic, true, ""}
|
||||
}
|
||||
}
|
||||
return Result{CheckOnTopic, false,
|
||||
fmt.Sprintf("mentions none of %v", c.WantAny)}
|
||||
fmt.Sprintf("mentions none of %v", wantAny)}
|
||||
}
|
||||
|
||||
// --- shape checks for the free-form paths --------------------------------
|
||||
//
|
||||
// The nudge checks assume one short sentence. Chat and query replies are longer
|
||||
// by design, so the only shape worth testing there is that the model produced a
|
||||
// reply at all and did not trail off. Both are failure modes the fallbacks in
|
||||
// llmphraser.go hide: a truncated or empty generation still returns nil error.
|
||||
|
||||
const (
|
||||
CheckNonEmpty = "nonempty" // she said something
|
||||
CheckEllipsis = "ellipsis" // she finished the sentence
|
||||
)
|
||||
|
||||
func checkNonEmpty(body string) Result {
|
||||
if strings.TrimSpace(body) == "" {
|
||||
return Result{CheckNonEmpty, false, "empty reply"}
|
||||
}
|
||||
return Result{CheckNonEmpty, true, ""}
|
||||
}
|
||||
|
||||
// checkEllipsis — a reply ending in "…" or "..." is a generation that ran out of
|
||||
// tokens, not a stylistic pause. Mid-sentence ellipses are left alone.
|
||||
func checkEllipsis(body string) Result {
|
||||
trimmed := strings.TrimRight(strings.TrimSpace(body), `"'»)`)
|
||||
if strings.HasSuffix(trimmed, "…") || strings.HasSuffix(trimmed, "...") {
|
||||
return Result{CheckEllipsis, false, "reply trails off in an ellipsis — likely truncated"}
|
||||
}
|
||||
return Result{CheckEllipsis, true, ""}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user