Score the chat, query and knowledge phrasing paths (#395)
The phrasing fixture was 15 nudge cases, so every prompt change we measured only told us about nudges. But the shared context block sits in front of five prompts, and three of them — chat, note query, general knowledge — had no scorer at all. Those are the long free-form replies, where a persona break is most likely and where nothing could see one. 27 cases, nine per path. Nine rather than five because the nudge fixture already cannot resolve a change smaller than about three cases, and a per-path score off five would be worse. Reuses the persona checks instead of copying them. Length, mood and "no questions" are left out on purpose: these paths return no mood, and a follow-up question is a feature in chat, not a fault. The run refuses to score unless the model answers before and after it. PhraseChat and PhraseQuery swallow model errors and return a canned string, so without that guard a dead server produces a full report with zero errors and a bad score — which reads as bad phrasing rather than as nothing measured. Vikunja #397 is the real fix.
This commit is contained in:
@@ -103,14 +103,17 @@ eval-router:
|
||||
eval-recall:
|
||||
MAVEN_ONNX_LIB="$(MAVEN_ONNX_LIB)" $(GO) test -v -count=1 ./internal/memory/recalleval/
|
||||
|
||||
# eval-phrasing -- score nudge phrasing (internal/phraser/eval). Verbose so the
|
||||
# eval-phrasing -- score nudge phrasing AND the conversational paths (chat,
|
||||
# query, general knowledge) in internal/phraser/eval. Verbose so the
|
||||
# report and every generated message land in the terminal. With no environment
|
||||
# it scores the deterministic Stub only, which is what CI runs. Set
|
||||
# MAVEN_LLM_URL to add the resident model:
|
||||
# MAVEN_LLM_URL=http://127.0.0.1:18099 make eval-phrasing
|
||||
# The model run is slow (minutes) -- the timeout is raised to match.
|
||||
# The model run is slow (minutes) -- the timeout is raised to match. It covers
|
||||
# two fixtures now (15 nudges + 27 conversational cases, and the chat replies are
|
||||
# the long ones), hence 90m rather than 40m.
|
||||
eval-phrasing:
|
||||
$(GO) test -v -count=1 -timeout 40m ./internal/phraser/eval/
|
||||
$(GO) test -v -count=1 -timeout 90m ./internal/phraser/eval/
|
||||
|
||||
# eval-models — score ONE llama-server against the same fixture, for the
|
||||
# resident-model bake-off (#278, #250). Start a server with the gguf you want,
|
||||
|
||||
Reference in New Issue
Block a user