Salvage two eval harnesses: nudge phrasing and per-model routing #13

2026-07-31T08:12:25+02:00

claude commented

2026-07-31 08:12:25 +02:00

Salvage. Two agents were building measurement harnesses when the session hit its API limit and both were killed mid-run. Each had committed one self-contained piece first, so those are here rather than thrown away. Neither result is finished — this is the tooling only.

make eval-phrasing — a deterministic scorer for nudge phrasing (Vikunja #323). Scores the Stub with no environment, which is what CI can run; add MAVEN_LLM_URL to score the resident model too.

The one number it got before dying is worth repeating even unfinished: the baseline scored 0 out of 15, and in at least one case the model echoed the "..." placeholder straight out of the prompt back as the nudge text. Treat that as a smoke signal, not a measurement — nothing has re-run it since, and the p50 in that run was 48s under contention.

make eval-models — scores one llama-server against the routing fixture and labels the report with the model the server actually has loaded, so two checkpoints cannot be confused for each other. That labelling is the point: the bake-off runs one model at a time on a 13GB box, and unlabelled reports from sequential runs are easy to mix up.

Both targets exist and run. The measurements they were built for still need doing — the bake-off (#278/#250) and the phrasing tuning (#323) are both still open.

Salvage. Two agents were building measurement harnesses when the session hit its API limit and both were killed mid-run. Each had committed one self-contained piece first, so those are here rather than thrown away. **Neither result is finished — this is the tooling only.** **`make eval-phrasing`** — a deterministic scorer for nudge phrasing (Vikunja #323). Scores the Stub with no environment, which is what CI can run; add `MAVEN_LLM_URL` to score the resident model too. The one number it got before dying is worth repeating even unfinished: the baseline scored **0 out of 15**, and in at least one case the model echoed the `"..."` placeholder straight out of the prompt back as the nudge text. Treat that as a smoke signal, not a measurement — nothing has re-run it since, and the p50 in that run was 48s under contention. **`make eval-models`** — scores one llama-server against the routing fixture and labels the report with the model the server actually has loaded, so two checkpoints cannot be confused for each other. That labelling is the point: the bake-off runs one model at a time on a 13GB box, and unlabelled reports from sequential runs are easy to mix up. Both targets exist and run. The measurements they were built for still need doing — the bake-off (#278/#250) and the phrasing tuning (#323) are both still open.

kami commented

2026-07-31 18:45:59 +02:00

this one shouldn't go in master.

kami changed target branch from master to overnight/fmt-gate

2026-07-31 19:12:50 +02:00

kami added 4 commits 2026-07-31 19:12:50 +02:00

Label LLM eval runs with the model llama-server has loaded 94eb92fb15

The bake-off in #278/#250 needs two models' scores side by side, and the
report names only carried the config, so the rows were indistinguishable.
ModelID reads /v1/models instead of taking a string that goes stale.
New target: make eval-models MAVEN_LLM_URL=...

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Add a deterministic scorer for nudge phrasing (Vikunja #323 ) 4ba9a6f422

Review internal/phraser/eval/checks.go -- it IS the measurement. Each check
names in a comment which DESIGN.md line it defends: length, feminine
self-reference (windowed around "я" so the operator's own masculine
second-person forms are not flagged), the cringe list (pet names, emoji,
"!!", fake concern, apology, emotional support, asking how he feels,
praise), on-topic, mood enum. No send/veto signal anywhere, per
DESIGN.md § "Rules decide, LLM phrases".
Fixture (158 lines) and tests (252) do not count toward the diff ceiling;
the scorer itself is still ~650. Splitting eval.go from checks.go would
give two commits neither of which measures anything.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Merge commit '4ba9a6f' into overnight-jul31 0914e0a3d5

# Conflicts:
#	Makefile

Merge commit '94eb92f' into overnight-jul31 1c4eab2107

# Conflicts:
#	Makefile

kami commented

2026-07-31 20:21:49 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:21:49 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#13