claude
246db4e609
The swap itself already landed: deploy loads models/embedder/multilingual-e5-small/model_quantized.onnx, and onnxembedder.go grew EmbedQuery/EmbedPassage with the query:/passage: prefixes the model was trained with. What was missing is the half of #371 that says "re-run make eval-recall and compare against the recorded numbers", so nothing in the repo says whether it worked. It worked, on every axis at once. recall@1 60.0% → 70.4%, recall@3 80.0% → 85.2%, answered after the gate 48.0% → 63.0%, false recall 1/5 → 0/5, and latency p50 59ms → 23ms because the quantized file is 118MB against the 470MB fp32 one the old config loaded. The guitar-chords note no longer beats the docker-logs note. One premise of the task did not come true and the new doc says so. #371 expected a better retriever to separate the score distributions and make query_min_score tunable. It did not: right-first top-1 runs 0.791-0.890 and must-stay-silent runs 0.795-0.835, still overlapping, just higher and tighter. The margin separates them instead — 0.024 median against 0.002 — and 0.008 is the knee where all five silent cases are silenced at no cost. The score gate is close to inert now; the margin is the live dial. Neither is changed here, since #412 is where a sweep belongs. docs/evals/2026-08-04-recall-e5-small.md is the dated measurement. rearchitecture.md's "upgrade MiniLM → bge-m3 later" is now done and says so, CLAUDE.md names the retriever and the prefix rule where it already promises the embedder never leaves homesrv, and the Makefile comment points at this eval instead of the one that asked for the swap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
101 lines
4.8 KiB
Markdown
101 lines
4.8 KiB
Markdown
# Maven — Re-architecture (Qwen3 resident model, revised 2026-07-18)
|
|
|
|
*Last verified: 2026-08-04 @ b6abb19. Living doc: correct it in place, do not append.*
|
|
|
|
> Supersedes the classifier-first routing model. Agreed in a design session
|
|
> after diagnosing that homesrv deploys with a **stub phraser** (no LLM
|
|
> running) and an embedder-classifier that routes by nearest-neighbor between
|
|
> frozen seed phrases — the structural cause of "she messes up queries."
|
|
>
|
|
> Hardware reality: homesrv = Ryzen 5 5600U laptop, Vega iGPU, 14 GB shared
|
|
> RAM. Workstation (RX 7900 XT) is NOT the deploy target and is often busy.
|
|
> So: small models, on-demand where heavy, always-on where cheap.
|
|
|
|
## Principle
|
|
|
|
The LLM is **not** the center of everything. Deterministic tools handle the
|
|
bulk. One locally trained **Qwen3-1.7B** resident model is used for routing and
|
|
talking back. Its system prompt selects one of two independently evaluated
|
|
contracts: route actions or persona responses.
|
|
|
|
**If the router is good, Maven feels good.** Routing is the linchpin.
|
|
|
|
## The spine
|
|
|
|
```
|
|
utterance
|
|
→ [world-state context] cheap: time, presence, calendar_busy, weather (no LLM)
|
|
→ ROUTER = Qwen3-1.7B (always-on, grammar-constrained)
|
|
reads utterance + context + tool schema, emits a STRUCTURED action:
|
|
• call a tool (deterministic) • answer directly
|
|
→ tools (deterministic, fast)
|
|
→ PHRASER = Qwen3-1.7B (same resident process) → TTS / text
|
|
```
|
|
|
|
- **Router = Phraser = one resident Qwen3-1.7B llama-server**, with separate
|
|
route and persona prompts/contracts. Always warm; classifier/stub remain the
|
|
failure floors.
|
|
- The resident checkpoint is trained end-to-end as: Qwen3-1.7B-Base → RU CPT →
|
|
joint persona/router SFT → merge → GGUF. Larger on-demand reasoning models
|
|
are deferred until the main feature set is complete.
|
|
- **Embedder demoted from router to tool** — it now backs `memory.search`
|
|
(RAG) and gives the router a cheap "similar past notes/intents" hint. The
|
|
router no longer depends on it clearing a threshold. The MiniLM upgrade is
|
|
done: it is multilingual-e5-small, asymmetric, with the `query:`/`passage:`
|
|
prefixes (Vikunja #371, `docs/evals/2026-08-04-recall-e5-small.md`). A
|
|
further swap is a model swap, not architecture (Vikunja #412).
|
|
|
|
### Router output
|
|
- Constrained structured JSON action `{tool, args, escalate}` — NOT free-form
|
|
multi-step function-calling. Sub-1B is far more reliable emitting a fixed
|
|
schema. Enforce with a **GBNF grammar** in llama.cpp (near-bulletproof).
|
|
- Keep the existing **stage-0 exact-match fast-path** for dead-obvious commands
|
|
(skips the router entirely) — cheap insurance, already built.
|
|
|
|
## The proactive / memory half — one background engine
|
|
|
|
"Take notes," "remember," "reflect," "suggest do you want to add X?", "remind"
|
|
are NOT request-path features. They are one **digestion worker**:
|
|
|
|
```
|
|
DIGESTION WORKER (periodic + event-driven, off the request path)
|
|
• reads new facts/notes since last pass
|
|
• RAG-consolidates: dedupe, link, summarize into durable memory
|
|
• reflects: detect patterns ("mentioned X three times")
|
|
• proposes: "want me to add X / remind you about Y?" → nudge dispatcher
|
|
• surfaces due reminders
|
|
uses the resident Qwen3 model for bounded synthesis — never blocks a turn
|
|
```
|
|
|
|
Notes capture is a deterministic Tier-0 tool; making notes *mean something
|
|
later* is the worker + RAG.
|
|
|
|
## Layer table
|
|
|
|
| Layer | What | Runs |
|
|
|---|---|---|
|
|
| Context | world-state (time/presence/calendar/weather) | always, no LLM |
|
|
| **Router** | Qwen3 structured-action orchestrator — linchpin | **always-on** |
|
|
| Tools | note/reminder/memory/calendar/weather/act (deterministic) | always |
|
|
| Reasoner | larger specialist model | **deferred** |
|
|
| Phraser | Qwen3 persona response | **always-on (same proc as router)** |
|
|
| Digestion worker | reflection → suggestions/nudges/memory | **background** |
|
|
| Reach | telegram (+ existing ntfy/voice) | quick win |
|
|
| Voice quality | custom/better TTS | **deferred** (workstation GPU busy) |
|
|
|
|
## Build order
|
|
|
|
1. **Foundation + router** — completed shared llama-server client, router,
|
|
fallback, replier, TTS normalization and telegram reach.
|
|
2. **Qwen3 resident checkpoint** — finish RU CPT, pass the raw-vs-CPT gate,
|
|
jointly SFT persona/router contracts, merge, quantize and deploy.
|
|
3. **Main features** — reflection, proactive suggestions, memory
|
|
consolidation, RAG read-back.
|
|
4. **Deferred work** — larger reasoner, custom Piper voice and other expansions.
|
|
|
|
## Non-goals (unchanged)
|
|
Not a nag. Not autonomous. Feminine-gendered RU self-ref. No telemetry, no
|
|
cloud model, no third-party account — but she MAY read external sources to
|
|
answer world questions (Kiwix first, search optional). "Never phones home" as
|
|
an absolute is deprecated, owner's call 2026-07-31; see CLAUDE.md § Non-goals.
|