diff --git a/CLAUDE.md b/CLAUDE.md index f297282..26b9bf5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -69,19 +69,30 @@ protocol; the config in `deploy/mavend.json` (with `${VAR}` env expansion from g ## Routing — read this before touching the router -`internal/router/` has TWO layered engines and the committed default is an **interim -stopgap, not the intended design** (see memory `routing-architecture-target`): +`internal/router/` has TWO layered engines. **The LLM router is now the default and it is +on in deploy** — this section used to say it was wired `nil`, which stopped being true on +2026-07-31. -- **Target (REARCH.md):** LLM-as-router. One resident Qwen3-1.7B (`llmrouter.go`) emits - GBNF-constrained structured JSON, and the SAME model phrases replies. Embedder is demoted - from a routing gate to a RAG hint. -- **Current stopgap:** `llmrouter` is wired `nil` (around `voice.go`), so the - `classifier.go` + `embedder.go` nearest-neighbour cascade actually runs. It routes by - similarity to frozen seed phrases — the known cause of weak RU query handling. +- **LLM router (the intended design, REARCH.md):** the resident Qwen3-1.7B (`llmrouter.go`) + emits GBNF-constrained structured JSON, and the SAME model phrases replies. Embedder is + demoted from a routing gate to a RAG hint. Wired at `voice.go:214` via + `pickLLMRouter(cfg.Voice.UseLLMRouter(), llmClient)`; the flag is `voice.llm_router` + (`config.go`), `DefaultLLMRouter` is **on**, and `deploy/mavend.json` sets it `true`. +- **Classifier cascade (the failure floor, not dead code):** `classifier.go` + + `embedder.go` nearest-neighbour over frozen seed phrases. It runs when the LLM router is + off, when there is no llama-server to talk to (`pickLLMRouter` logs that and degrades), + and on any per-turn LLM error. Do not delete it — routing by seed similarity is the known + cause of weak RU query handling, but a turn must never break on the model. Cascade order: `stage0.go` exact-match fast-path → LLM router (when non-nil) → classifier fallback. Any LLM error falls through to the classifier so a turn never breaks on the model. +Measured on the 77-case RU fixture (`MODEL-BAKEOFF-31-07-2026.md`): the classifier scores +36.8% full accuracy at p50 31ms; Qwen3-1.7B scores 67.5% intent-only / 72.7% through the +cascade at p50 ≈2.7s. Accuracy roughly doubled, latency is ~90× worse, and that trade was +accepted deliberately. Still open: `Confidence: 1.0` is hardcoded in `llmrouter.go`, so the +LLM path never asks for clarification (6/6 refusal cases missed) — Vikunja #359. + ## LLM output contract All phrasing paths emit `{"response":"...","mood":"..."}` (parsed in `replier_llm.go` and @@ -94,7 +105,11 @@ workspace enforces that the Go and relabelling prompts remain identical. ## Non-goals (hard constraints) Not a nag, not autonomous. Maven's persona is **feminine** — Russian -self-reference must use feminine forms (the user is male; see memory `maven-persona-gender`). +self-reference must use feminine forms — `рада`, not `рад`; `поняла`, not `понял`. The owner +is male and is addressed informally: "ты", singular, never "вы"/"ваш" and never "он"/"его" +(she talks TO him, not about him). Pet names ("милый", "дорогой") are forbidden; his name +("Ками") is not. The eval enforces this: `CheckAddress`, `CheckFeminine` and `CheckCringe` in +`internal/phraser/eval/checks.go`, scored by `make eval-phrasing`. **"Never phones home" is DEPRECATED** (owner's call, 2026-07-31). It used to be a hard constraint and it is not one any more: a 0.8B — and a 1.7B — does not know enough to answer diff --git a/MODEL-BAKEOFF-31-07-2026.md b/MODEL-BAKEOFF-31-07-2026.md index 0a14993..1c55f2c 100644 --- a/MODEL-BAKEOFF-31-07-2026.md +++ b/MODEL-BAKEOFF-31-07-2026.md @@ -19,7 +19,10 @@ Settles Vikunja **#278 / #250**. - Same fixture and scorer as `ROUTING-EVAL-31-07-2026.md`: `internal/router/eval/` (`ru_routing_v1.json`, 76 held-out cases). - Reproduce: `MAVEN_LLM_URL=http://127.0.0.1: make eval-router` - (`TestLLMRouterBaseline`). Note: there is no `make eval-models` target. + (`TestLLMRouterBaseline`). (This line used to say there is no `make eval-models` target. + There is one now — start a server with the gguf you want, then + `make eval-models MAVEN_LLM_URL=http://127.0.0.1:`. It runs only the LLM test, since + the classifier baselines do not depend on the model.) - All three models served by the same `llama-server` flags — `-c 2048 -ngl 99 -t 6`, only `-m` and `--port` differ. One server at a time on an otherwise idle box, so latencies are real and not contention. @@ -207,10 +210,14 @@ swapped again when the CPT lands. - Routing is one run per model, not three. The gaps between families are far larger than the run-to-run spread seen on the talk fixture, but the 2B-vs-1.7B gap (62.3 vs 67.5) is not safe to call on one run. -- The routing numbers only reach production once the LLM router is wired on. It is - still `nil`. -- `/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download - in the wrong directory. The good 1.13 GB copy is in `qwen3/`. Delete the stray one. +- ~~The routing numbers only reach production once the LLM router is wired on. It is + still `nil`.~~ **Resolved the same evening:** the LLM router is wired at `voice.go:214` + behind `voice.llm_router`, the default is on, and `deploy/mavend.json` sets it `true`. + These numbers are the production path now, so the p50 ≈2.7s is a real per-turn cost and + not a bench artifact. +- ~~`/mnt/hdd1/llms/LFM2.5/Qwen3-1.7B-UD-Q4_K_XL.gguf` is a 293 MB truncated download + in the wrong directory.~~ **Deleted 2026-07-31.** The good 1.13 GB copy in `qwen3/` is + what `deploy/mavend.json` loads. - Harness: `scratchpad/bakeoff.sh`, one server at a time, health-checked before each run, `/v1/models` recorded per run. Never run two LLM consumers at once — see the contamination note in `TALK-EVAL-31-07-2026.md`.