Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c5264deb46 | |||
| 1cb269886e | |||
| 7a9b9cc669 | |||
| 31b5093403 | |||
| 229890abd7 | |||
| 0db9ca084c |
@@ -0,0 +1,40 @@
|
|||||||
|
# maven-voice-tunnel — the ssh leg that carries the voice wire to homesrv.
|
||||||
|
#
|
||||||
|
# Runs on workpc, as a user unit (`systemctl --user`), beside mavgpud.service.
|
||||||
|
#
|
||||||
|
# WHY THIS EXISTS AT ALL. internal/voice is plaintext and unauthenticated.
|
||||||
|
# Its own server doc says production binds inside the wg tunnel, because "the
|
||||||
|
# wg layer IS the L0 floor". workpc is not a wg peer, it sits on wlan0. So ssh
|
||||||
|
# is the substitute floor: it authenticates with his key and encrypts the leg,
|
||||||
|
# and mavend's published port stays on homesrv loopback (127.0.0.1:9110).
|
||||||
|
# Nothing about this puts a Maven port on the LAN.
|
||||||
|
#
|
||||||
|
# Do not replace this with a LAN bind. SurfaceVoice caps acts at L0, so an
|
||||||
|
# unauthorized speaker could not run a destructive tool. It would still hear
|
||||||
|
# his facts, his notes and his calendar read back, and L0 does not cap reading.
|
||||||
|
#
|
||||||
|
# install: cp to ~/.config/systemd/user/ on workpc
|
||||||
|
# systemctl --user enable --now maven-voice-tunnel.service
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=SSH tunnel to mavend's voice wire on homesrv
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
# -N: no remote command, forwarding only.
|
||||||
|
# ExitOnForwardFailure: fail loudly rather than sit up with a dead forward,
|
||||||
|
# which is what makes Restart meaningful.
|
||||||
|
# ServerAlive*: a laptop that suspends drops the tunnel silently otherwise.
|
||||||
|
ExecStart=/usr/bin/ssh -N \
|
||||||
|
-o ExitOnForwardFailure=yes \
|
||||||
|
-o ServerAliveInterval=30 \
|
||||||
|
-o ServerAliveCountMax=3 \
|
||||||
|
-o BatchMode=yes \
|
||||||
|
-L 127.0.0.1:9100:127.0.0.1:9110 \
|
||||||
|
kami@192.168.1.104
|
||||||
|
Restart=always
|
||||||
|
RestartSec=5
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# mavwaked — always-on listening, on workpc where the microphone is.
|
||||||
|
#
|
||||||
|
# User unit, beside mavgpud.service and maven-voice-tunnel.service. It is a
|
||||||
|
# user unit because it needs his ALSA session and his ssh agent, and because
|
||||||
|
# it should stop when he logs out.
|
||||||
|
#
|
||||||
|
# THERE IS NO WAKE WORD YET (V-487 stage two). Anything spoken near the fifine
|
||||||
|
# becomes a turn. What makes that safe rather than expensive is voiceSender:
|
||||||
|
# it sends Surface=SurfaceVoice, which caps every command at L0, so no
|
||||||
|
# accidental trigger runs a destructive act. It does not stop her answering
|
||||||
|
# out loud, so this unit is his to stop when the room is not his alone.
|
||||||
|
#
|
||||||
|
# -vad-model is passed on purpose. Silero answers "is this frame speech" where
|
||||||
|
# the energy floor answers "is this frame loud". It declines white noise at
|
||||||
|
# the same RMS 0 frames to 68-99, and still hears all four spoken fixtures
|
||||||
|
# (docs/evals/2026-08-09-silero-vad.md). It costs 509us a frame, 1.7% of one
|
||||||
|
# core, and never touches the GPU. Drop the flag and the energy floor is back.
|
||||||
|
#
|
||||||
|
# -barge-in is NOT passed. The threshold is room-specific and this room has no
|
||||||
|
# number yet. Turn it on only after reading the "suppressed while speaking"
|
||||||
|
# means out of this unit's own journal, never by guessing.
|
||||||
|
#
|
||||||
|
# install: cp to ~/.config/systemd/user/ on workpc
|
||||||
|
# systemctl --user enable --now mavwaked.service
|
||||||
|
|
||||||
|
[Unit]
|
||||||
|
Description=Maven always-on listening (VAD, no wake word yet)
|
||||||
|
# The tunnel is the only path to mavend and the only thing authenticating it.
|
||||||
|
Requires=maven-voice-tunnel.service
|
||||||
|
After=maven-voice-tunnel.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
# card 0 is the fifine USB microphone. Named, and not "default", because the
|
||||||
|
# default device follows whatever pipewire last decided and this daemon should
|
||||||
|
# not change ears when he plugs in a headset.
|
||||||
|
#
|
||||||
|
# plughw and not hw. mavwaked asks arecord for 16kHz mono, which is what the
|
||||||
|
# whole pipeline is canonical in. The fifine offers 2 channels at 44100 or
|
||||||
|
# 48000 and nothing else, so bare hw:0,0 dies on "Channels count non
|
||||||
|
# available" before a frame is read. plughw puts ALSA's downmix and resampler
|
||||||
|
# in front. Any replacement microphone wants the same treatment.
|
||||||
|
Environment=LD_LIBRARY_PATH=%h/.local/lib
|
||||||
|
ExecStart=%h/.local/bin/mavwaked \
|
||||||
|
-device plughw:0,0 \
|
||||||
|
-addr 127.0.0.1:9100 \
|
||||||
|
-lang ru \
|
||||||
|
-vad-model %h/.local/share/maven/models/silero_vad.onnx \
|
||||||
|
-onnx-lib %h/.local/lib/libonnxruntime.so
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=5
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=default.target
|
||||||
@@ -53,6 +53,19 @@ services:
|
|||||||
# the decrypted working copy lives in RAM (see db_tmpfs in mavend.json).
|
# the decrypted working copy lives in RAM (see db_tmpfs in mavend.json).
|
||||||
tmpfs:
|
tmpfs:
|
||||||
- /dev/shm
|
- /dev/shm
|
||||||
|
# the voice wire, for mavwaked and mavenclient on workpc (V-515).
|
||||||
|
#
|
||||||
|
# LOOPBACK ONLY, and that is the whole security argument. internal/voice
|
||||||
|
# is plaintext with no auth: its own server doc says production binds
|
||||||
|
# inside the wg tunnel, "the wg layer IS the L0 floor". workpc is not a wg
|
||||||
|
# peer, it is on wlan0. So the tunnel is ssh instead, terminated on this
|
||||||
|
# loopback address, and nothing new is on the LAN. Anyone who could reach
|
||||||
|
# a LAN-bound port here could push audio and hear his facts read back.
|
||||||
|
# SurfaceVoice caps acts at L0; it does not cap reading.
|
||||||
|
#
|
||||||
|
# Host 9100 is Vikunja's MCP, hence 9110. The container side stays 9100
|
||||||
|
# so mavweb keeps reaching mavend:9100 by name.
|
||||||
|
ports: ["127.0.0.1:9110:9100"]
|
||||||
|
|
||||||
mavsttd:
|
mavsttd:
|
||||||
<<: *image
|
<<: *image
|
||||||
|
|||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# gemma-4-E4B on the phrasing and talk fixtures
|
||||||
|
|
||||||
|
Date: 2026-08-09. Box: workpc up, E4B loaded on 8080.
|
||||||
|
`MAVEN_LLM_URL=http://192.168.1.105:8080 make eval-phrasing`.
|
||||||
|
|
||||||
|
This was the one unmeasured risk of the 2026-08-09 model swap. Routing was
|
||||||
|
measured the same day and E4B lost four destination cases to the 12B. Phrasing
|
||||||
|
was not measured at all, and phrasing is the half the owner hears.
|
||||||
|
|
||||||
|
## Result
|
||||||
|
|
||||||
|
| fixture | E4B | resident Qwen3-1.7B, 2026-08-05 |
|
||||||
|
|---|---|---|
|
||||||
|
| nudges | 15/15 (100%) | 15/15 (100%) |
|
||||||
|
| talk, passes every check | **29/36 (80.6%)** | 25/36 (69.4%) |
|
||||||
|
| lang | 36/36 | — |
|
||||||
|
| feminine | 36/36 | 36/36 |
|
||||||
|
| address | **36/36** | 33/36 |
|
||||||
|
| ontopic | 29/36 | 28/36 |
|
||||||
|
| p50 latency | **516ms** | 2.97s |
|
||||||
|
| p95 latency | 921ms | — |
|
||||||
|
| failed generations | 0 | 0 |
|
||||||
|
|
||||||
|
E4B beats the homesrv floor by four cases and answers about six times faster.
|
||||||
|
Persona is clean: `lang`, `feminine` and `address` are perfect, and `address`
|
||||||
|
is where the resident model still loses three. The 2026-08-05 measurement of the
|
||||||
|
resident model is the comparison, since both ran the same 36-case fixture.
|
||||||
|
|
||||||
|
Every failure is `ontopic`. Nothing failed on persona, nothing failed to parse.
|
||||||
|
|
||||||
|
## The score is at the ceiling, not below it
|
||||||
|
|
||||||
|
The 2026-08-05 temperature sweep found two cases that fail at every temperature
|
||||||
|
in every run: `reply-note-router` and `reply-fact-weight`. It named a defect in
|
||||||
|
the reply phrasing path rather than sampling noise. It put the fixture's ceiling
|
||||||
|
at 30/36 before persona is scored. Both cases are in E4B's failure list.
|
||||||
|
|
||||||
|
So 29/36 is one case off a ceiling nothing about the model can move. The swap is
|
||||||
|
safe on phrasing. Read this next to the routing result, not instead of it. There
|
||||||
|
E4B costs four destination cases and buys 50ms. Here it costs nothing.
|
||||||
|
|
||||||
|
## Two findings no check caught
|
||||||
|
|
||||||
|
**She says she wrote something down when she did not.** Asked what to do this
|
||||||
|
evening, E4B writes "Я записала несколько идей!". Asked for a joke, it writes
|
||||||
|
"Я записала одну забавную ситуацию!". Nothing was stored. No check scores it,
|
||||||
|
because `ontopic` reads the subject and `cringe` reads pet names. A claim to
|
||||||
|
have saved something is a claim about state, and it is wrong.
|
||||||
|
|
||||||
|
**Two `ontopic` failures look like check defects.** `know-dont-know` wants
|
||||||
|
"не зна" or "не мог". It got "Я не умею знать личную информацию о твоих
|
||||||
|
соседях", which declines correctly in words the check does not list.
|
||||||
|
`know-hiccups` is the same shape. Neither is a model failure and both count
|
||||||
|
against the score.
|
||||||
|
|
||||||
|
## Not measured here
|
||||||
|
|
||||||
|
A 12B control on the same fixture, which would need the card reloaded and is the
|
||||||
|
owner's call. The talk fixture through the daemon rather than through the
|
||||||
|
phraser directly. The CPT'd Qwen3-1.7B, which does not exist yet and is the
|
||||||
|
reason `address` is a check at all.
|
||||||
+457
@@ -0,0 +1,457 @@
|
|||||||
|
# Routing
|
||||||
|
|
||||||
|
*Last verified: 2026-08-09 @ 31b5093*
|
||||||
|
|
||||||
|
How an utterance becomes a `Decision`, why each stage exists, and what every
|
||||||
|
stage has measured. `CLAUDE.md` carries the rules an agent must not break. This
|
||||||
|
file carries the reasoning and the history behind them.
|
||||||
|
|
||||||
|
Two decisions come out of a route. **Intent** is one of seven values. **Source**
|
||||||
|
is where the answer lives, and it is read on `IntentQuery` alone. They are scored
|
||||||
|
separately, because one number hides which one moved.
|
||||||
|
|
||||||
|
## The cascade
|
||||||
|
|
||||||
|
| Stage | What it is | Where |
|
||||||
|
|---|---|---|
|
||||||
|
| 0 | Deterministic grammars over the utterance | `stage0.go`, `praxis.go`, `worldquery.go` |
|
||||||
|
| 0b | Four ONNX heads on one e5-small forward pass | `heads.go` |
|
||||||
|
| 1 | The resident model, GBNF-constrained JSON | `llmrouter.go` |
|
||||||
|
| 2 | Nearest neighbour over frozen seed phrases | `classifier.go`, `embedder.go` |
|
||||||
|
|
||||||
|
Every stage may decline, and the next one answers. Any error at stage 0b or 1
|
||||||
|
falls through, so a turn never breaks on a model.
|
||||||
|
|
||||||
|
The resident model arm is wired at `voice.go:214` through
|
||||||
|
`pickLLMRouter(cfg.Voice.UseLLMRouter(), llmClient)`. The flag is
|
||||||
|
`voice.llm_router` in `config.go`, `DefaultLLMRouter` is on, and
|
||||||
|
`deploy/mavend.json` sets it `true`. With no llama-server to talk to,
|
||||||
|
`pickLLMRouter` logs that and degrades to the classifier.
|
||||||
|
|
||||||
|
The classifier is the floor and not dead code. It runs when the resident model
|
||||||
|
is off, when there is no llama-server to talk to, and on any per-turn error.
|
||||||
|
Routing by seed similarity is the known cause of weak Russian queries. Deleting
|
||||||
|
it would make a model outage a broken turn.
|
||||||
|
|
||||||
|
### Why there are two engines at all
|
||||||
|
|
||||||
|
The original design was the classifier alone. `docs/rearchitecture.md` replaced
|
||||||
|
it with a model that emits structured JSON. The same weights phrase the reply.
|
||||||
|
That demoted the embedder from a routing gate to a hint for recall. The model
|
||||||
|
became the default on 2026-07-31.
|
||||||
|
|
||||||
|
The gap it buys is smaller than the design assumed. Measured 2026-08-02 on the
|
||||||
|
77-case Russian fixture, the classifier scores 68.8% full accuracy at p50 16.6µs.
|
||||||
|
Qwen3-1.7B scores 72.7% through the cascade. Four points, not a doubling.
|
||||||
|
|
||||||
|
An older figure of 36.8% for the classifier stood in `CLAUDE.md` until then. It
|
||||||
|
predates the stage 0 rules and the seed additions. Both now score inside the
|
||||||
|
classifier baseline.
|
||||||
|
|
||||||
|
Latency was misreported the same way. A figure of 2.7 seconds stood for two
|
||||||
|
days and was contention rather than the model.
|
||||||
|
`docs/evals/2026-07-31-routing.md` line 61 measures the router at p50 825ms and
|
||||||
|
the cascade at p50 0.80s to 1.04s.
|
||||||
|
|
||||||
|
## Numbers
|
||||||
|
|
||||||
|
Three arms answer, so three numbers are live. Judge a routing change against the
|
||||||
|
classifier and the resident model, since those are what always answer.
|
||||||
|
|
||||||
|
| Arm | Intent | Destination | p50 | Measured |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| classifier + ONNX | 76.0% (73/96) | 36.4% (12/33) | 16.6µs | 2026-08-08 |
|
||||||
|
| resident Qwen3-1.7B, cascade | 80.2% | not measured | 1.19s | 2026-08-05 |
|
||||||
|
| routing heads, cascade | 96.9% | 75.8% | 27.9ms | 2026-08-08 |
|
||||||
|
| gemma-4-12b, cascade | 84.4% | 72.7% | 329ms | 2026-08-02 |
|
||||||
|
| gemma-4-E4B, cascade | 89.6% | 57.6% (19/33) | 294ms | 2026-08-09 |
|
||||||
|
|
||||||
|
The fixture grew from 77 cases to 91 to 96. So a number is comparable only to
|
||||||
|
another number on the same fixture. Sources:
|
||||||
|
`docs/evals/2026-08-05-routing-resident-model.md`,
|
||||||
|
`docs/evals/2026-08-02-workstation-gemma4-12b.md`,
|
||||||
|
`docs/evals/2026-08-09-e4b-vs-12b-routing.md`,
|
||||||
|
`docs/evals/2026-08-08-routing-heads-in-go.md`,
|
||||||
|
`docs/evals/2026-08-08-destination-fixture.md`.
|
||||||
|
|
||||||
|
The resident model alone scores 37.4% full against 61.5% intent-only. The gap is
|
||||||
|
slots and not routing. It routes `reminder` and leaves the time to the daemon,
|
||||||
|
which is what the contract asks.
|
||||||
|
|
||||||
|
To re-run the resident model as router, start a **second** llama-server on a
|
||||||
|
fixed host port. The resident one binds `--port 0` inside the container and no
|
||||||
|
host process can reach it.
|
||||||
|
|
||||||
|
### The workstation is not the better router any more
|
||||||
|
|
||||||
|
It was, from 2026-08-02 until the heads landed. gemma-4-12b beat everything on
|
||||||
|
the box at 84.4% intent and 72.7% destination. The heads beat it on both at a
|
||||||
|
twelfth of the latency. The workstation stays the better phraser.
|
||||||
|
|
||||||
|
E4B replaced the 12B on 2026-08-09 by the owner's call. It is a step down on
|
||||||
|
routing. Against a same-session 12B control it costs four destination cases and
|
||||||
|
buys 50ms. Read destination as the finding. It names nothing where the 12B names
|
||||||
|
`recall` or `calendar`, which is safe but walks the whole chain. It has no MTP
|
||||||
|
and cannot be given any here. The only `gemma4-assistant` draft on disk is
|
||||||
|
trained against the 12B's hidden states.
|
||||||
|
|
||||||
|
## Stage 0: what the grammars claim, and why
|
||||||
|
|
||||||
|
A rule at this stage is a claim. Either the model gets this wrong, or it wastes a
|
||||||
|
second getting it right. Every rule was added against a measurement.
|
||||||
|
|
||||||
|
- **Agenda questions** (`AgendaQueryGrammars`, 2026-08-01). "что у меня сегодня",
|
||||||
|
"во сколько у меня встреча" and anything naming a calendar go to `IntentQuery`.
|
||||||
|
They were going to `IntentSystem`, where `replySystem` has no agenda arm and
|
||||||
|
answered "пока не умею". Worth 2.6 points of full accuracy and calendar 0/2 to
|
||||||
|
2/2.
|
||||||
|
- **Rest of day and narrative** (V-498, 2026-08-04). `rest-of-day-query` claims
|
||||||
|
"что дальше?". `NarrativeQueryGrammar` claims "расскажи про X", "объясни X" and
|
||||||
|
"опиши X". Neither carries a question mark or an interrogative, so the model
|
||||||
|
called both `IntentFact`. `IsQuestionShaped` caught the write downstream, so
|
||||||
|
this was a latency and fixture defect rather than a correctness one. The
|
||||||
|
narrative rule declines `chatNarrativeTopics`, because the query chain has no
|
||||||
|
source that answers a joke or a bedtime story.
|
||||||
|
- **Praxis** (V-516, 2026-08-05). `PraxisGrammars()` fills `Slots.Fn` with a
|
||||||
|
capability name. These grammars are the **only** path to Praxis and not a
|
||||||
|
faster one. The model reaches Praxis 0/12 alone, the same as the classifier.
|
||||||
|
Nothing in the router prompt names a Praxis capability, so there is no string
|
||||||
|
for it to write. Through the cascade it is 11/12. Measured overall 16/30 to
|
||||||
|
27/30, lifecycle 0/5 to 5/5
|
||||||
|
(`docs/evals/2026-08-05-praxis-reach.md`,
|
||||||
|
`docs/evals/2026-08-05-reach-llm-router.md`).
|
||||||
|
`handlePraxisAct` compares `Slots.Fn` to a capability alias. Otherwise that
|
||||||
|
slot is filled from the deployment's enabled tool names, and no Praxis alias
|
||||||
|
is on that list.
|
||||||
|
- **World questions** (`WorldQueryGrammars`, V-655, 2026-08-07). "что такое X"
|
||||||
|
and "сколько будет 17 на 23". Wired after the agenda rules and **before** the
|
||||||
|
feed and list rules. "что такое лента" is a definition question, and the feed
|
||||||
|
rule would take it on the noun alone.
|
||||||
|
|
||||||
|
`calendar-query` and `event-time-query` name the calendar as the destination.
|
||||||
|
The possessive agenda rules deliberately do not. "что у меня в списке покупок"
|
||||||
|
matches `agenda-query`, and naming the calendar there would take the list source
|
||||||
|
off the turn. That caution now costs four destination cases. See the model arm
|
||||||
|
below.
|
||||||
|
|
||||||
|
Go's `\b` is ASCII-only and never fires after a Cyrillic letter. A pattern needs
|
||||||
|
an explicit `(\s|[?!.]|$)`.
|
||||||
|
|
||||||
|
`baselineGrammars` in `eval_test.go` mirrors `buildRouter` and has drifted before.
|
||||||
|
`WorldQueryGrammars` was wired into the daemon by V-655 and not into the mirror,
|
||||||
|
so the fixture scored a grammar set nobody runs. Fixed by V-659, worth 3 points
|
||||||
|
of destination.
|
||||||
|
|
||||||
|
### Praxis lifecycle rules
|
||||||
|
|
||||||
|
A **stative** lifecycle word ("готово", "принято") needs an item named beside it.
|
||||||
|
A bare **imperative** ("закрывай") may ask which one. It also requires a sentence
|
||||||
|
naming no object of its own. Otherwise "закрой шторы в комнате" goes to Praxis
|
||||||
|
instead of the house. A demonstrative ("отметь это как сделанное") resolves
|
||||||
|
against `h.surfacedItems` only when exactly one item was spoken. Otherwise the
|
||||||
|
turn goes back to the cascade rather than transitioning the wrong item.
|
||||||
|
|
||||||
|
### Slots on a stage 0 decision
|
||||||
|
|
||||||
|
`fillMatchedSlots` runs the stage 2 extractor over whatever a grammar built
|
||||||
|
(V-572, 2026-08-06). It fills only the slots the grammar left empty. A matched
|
||||||
|
value always wins, because the rule read a literal pattern and the extractor
|
||||||
|
guesses.
|
||||||
|
|
||||||
|
It did not run before. So `ReminderGrammar` handed the daemon `HasTime: false`
|
||||||
|
for "напомни в 11:00 позвонить маме", and `missingFor` read the silence as
|
||||||
|
absence and asked "Когда?". It is inert for every grammar but the reminder:
|
||||||
|
`Extract` fills Time, Fn and Key and nothing else. A stage 0 query costs 3.7µs
|
||||||
|
against 3.9µs before, benchmarked at 20000x.
|
||||||
|
|
||||||
|
`Slots.Text` is deliberately not filled. A grammar that left it empty meant it,
|
||||||
|
and `agendaQueryBuild` hands the query chain the utterance itself.
|
||||||
|
|
||||||
|
## Stage 0b: the routing heads
|
||||||
|
|
||||||
|
Routing has a bounded output space, so it is classification rather than
|
||||||
|
generation (owner's call, V-546,
|
||||||
|
`docs/plans/18-routing-heads-on-e5-small.md`). The 118M multilingual-e5-small is
|
||||||
|
already resident. A softmax cannot emit a value that does not exist, so no
|
||||||
|
grammar is needed. Max softmax is a calibratable confidence, where
|
||||||
|
`Confidence: 1.0` was a hardcode. Training costs roughly 5e15 FLOPs, so 10 to 30
|
||||||
|
minutes on the workstation. A 100M decoder from scratch is 10 to 20 GPU hours.
|
||||||
|
|
||||||
|
**Fine-tune a copy of the weights.** The resident embedder backs memory recall.
|
||||||
|
Training it in place couples routing accuracy to recall@1, with nothing in the
|
||||||
|
suite to name the trade.
|
||||||
|
|
||||||
|
Four heads share one masked mean pool, trained over three days. The measurements
|
||||||
|
are `docs/evals/2026-08-08-routing-heads-two-head.md`,
|
||||||
|
`docs/evals/2026-08-08-slot-head-three-head.md`,
|
||||||
|
`docs/evals/2026-08-08-clarify-head-four-head.md` and
|
||||||
|
`docs/evals/2026-08-08-massive-warm-start.md`.
|
||||||
|
|
||||||
|
| Head | Score | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| intent | 92.8% mean over 3 seeds | fixture is the 88 cases carrying an intent |
|
||||||
|
| destination | 80.8% mean, best 29/33 | beats the 12B teacher it was distilled from |
|
||||||
|
| slot BIO tags | 72.4% span F1 | still climbing when epoch selection stops it |
|
||||||
|
| clarify | catches 7.0 of 8, 2.3 false of 88 | parity with the cascade, no rules in front |
|
||||||
|
|
||||||
|
Read the best destination run as one seed and not a headline. One case is 3
|
||||||
|
points on a fixture this small. Head intent accuracy is **not** comparable to the
|
||||||
|
cascade's 76.0% and 84.4%. A softmax has no clarify class, so the head's fixture
|
||||||
|
is 88 cases and not 96.
|
||||||
|
|
||||||
|
Recall is 15/15 and world is 5/5.
|
||||||
|
|
||||||
|
**Mood is cut, not deferred.** The enum describes her own reply state, not the
|
||||||
|
speaker's emotion, and no dataset maps onto it.
|
||||||
|
|
||||||
|
### The clarify head
|
||||||
|
|
||||||
|
Clarify is not a value of intent, so a softmax cannot emit it. It is a second
|
||||||
|
question over the same pooled vector: can Maven act on this at all. Accuracy is
|
||||||
|
the wrong number here and a head that never asks scores 91.7%.
|
||||||
|
|
||||||
|
Confidence is the other half. Max softmax over the intent head reads 0.851 where
|
||||||
|
it is right and 0.604 where it is wrong. It ranks right above wrong in 83.4% of
|
||||||
|
pairs.
|
||||||
|
|
||||||
|
It is not free the way the slot head was. Intent, destination and slot F1 each
|
||||||
|
move down one to four points, inside the seed spread. `поужинал` is a false
|
||||||
|
clarify on every seed. That is the same defect `thinSingleToken` was narrowed for
|
||||||
|
on 2026-08-01.
|
||||||
|
|
||||||
|
The corpus is generated, because every existing row is answerable by
|
||||||
|
construction. The router-prompt agreement filter cannot work here. `routeGrammar`
|
||||||
|
has no clarify value, and a generated line always agrees with itself. A gemma
|
||||||
|
judge replaces it. The first judge called 24 of 40 answerable rows underspecified.
|
||||||
|
It judged against a generic assistant rather than against Maven's contract.
|
||||||
|
|
||||||
|
### The slot head
|
||||||
|
|
||||||
|
BIO slot tags had no Maven-domain corpus. That was true of found corpora and
|
||||||
|
false of made ones. `label_slots.py` distils spans out of gemma-4-12b under a
|
||||||
|
GBNF closed over Maven's own five slots. A span survives only when it is a
|
||||||
|
literal substring of the utterance, so the agreement filter costs no second call.
|
||||||
|
2178 spans over 1702 rows, 37 dropped, nothing unparsed.
|
||||||
|
|
||||||
|
Epoch selection reads the intent dev slice alone. That costs the slot head about
|
||||||
|
4 points.
|
||||||
|
|
||||||
|
### Warm start and the floor
|
||||||
|
|
||||||
|
The MASSIVE warm-start of step 2 is worth nothing here. Stock e5-small ties it on
|
||||||
|
intent and leads by a third of a case on destination. Nothing argues for keeping
|
||||||
|
that step.
|
||||||
|
|
||||||
|
The floor was a corpus defect and it is fixed. The first 120 floor rows carried
|
||||||
|
one sentence shape, so the head named a destination where the fixture says walk
|
||||||
|
the chain. Rotating six shapes took the floor 3/7 to 6/7 and destination 75.8% to
|
||||||
|
80.8%.
|
||||||
|
|
||||||
|
What is left is calendar at 3/6 on every seed, which training cannot move. The
|
||||||
|
possessive agenda rules claim those cases at stage 0 and name nothing, so no
|
||||||
|
label reaches the head.
|
||||||
|
|
||||||
|
### Reading them in Go
|
||||||
|
|
||||||
|
`RouterHeads` in `internal/router/heads.go` loads `router_heads.onnx` (V-664,
|
||||||
|
2026-08-08). It reads intent, destination and clarify off one forward pass.
|
||||||
|
|
||||||
|
Three rules around it, each measured:
|
||||||
|
|
||||||
|
- The **clarify head decides first**, before the intent threshold. It answers a
|
||||||
|
different question. A thin utterance scores low intent by construction, so
|
||||||
|
gating it cost 6 of 8 ambiguous cases.
|
||||||
|
- The **destination head is read on `IntentQuery` only**, since no other intent
|
||||||
|
reaches `queryWalk`.
|
||||||
|
- `headsThreshold` is 0.6, the measured knee. Every value up to 0.85 drops right
|
||||||
|
answers and keeps the same two wrong ones.
|
||||||
|
|
||||||
|
`voice.embedder.heads_path` is the whole switch. Empty, missing or unloadable
|
||||||
|
means the heads are nil. The cascade is then byte-for-byte what shipped before
|
||||||
|
them.
|
||||||
|
|
||||||
|
### The tokenizer bug the heads found
|
||||||
|
|
||||||
|
`encodeWord` in `onnxembedder.go` read every long word backwards until 2026-08-08.
|
||||||
|
It cost recall@1 7.4 points and recall@3 11.1. Nothing caught it, because seeds
|
||||||
|
and queries were mangled the same way and cosine survived. The heads found it.
|
||||||
|
They are trained through transformers and read through this.
|
||||||
|
|
||||||
|
The embedder id now carries a tokenizer revision (`@384/tok2`). So fixing the
|
||||||
|
tokenizer triggers `ReembedAll` the way swapping the model file does. Bump
|
||||||
|
`tokenizerRev` on any change to what it emits.
|
||||||
|
|
||||||
|
## Clarify
|
||||||
|
|
||||||
|
`Confidence: 1.0` was hardcoded in `llmrouter.go`. So the model path could never
|
||||||
|
ask for clarification, and it missed 6 of 6 refusal cases (V-359). The bug had a
|
||||||
|
second half. The model branch never consulted `r.threshold` at all, so a correct
|
||||||
|
low confidence would have been discarded anyway.
|
||||||
|
|
||||||
|
Fixed 2026-07-31 with structural signal feeding the same stage 3 gate the
|
||||||
|
classifier path already had (`gateLLMDecision` in `router.go`). Three signals: a
|
||||||
|
single-token utterance, a keyless fact, an act with no allowlisted fn.
|
||||||
|
|
||||||
|
Re-measured: missed clarify 6/6 to 1, at the cost of 3 false clarifies and 2.6
|
||||||
|
points of full accuracy. Two of the three false clarifies are acts the model
|
||||||
|
mis-routed and the gate caught. Asking beats wrongly executing, so the fixture
|
||||||
|
and the daemon disagree about what is correct there.
|
||||||
|
|
||||||
|
The third, `поужинал`, was a real defect. The single-token rule was an English
|
||||||
|
intuition. It does not transfer to Russian, where one word is routinely a whole
|
||||||
|
sentence.
|
||||||
|
|
||||||
|
Narrowed 2026-08-01. `thinSingleToken` (`internal/router/singletoken.go`) still
|
||||||
|
thins a bare one-word nominal. It spares two classes. One is a closed lexicon of
|
||||||
|
social and control singles ("привет", "стоп", "yes"). The other is any token
|
||||||
|
carrying a Russian verb ending, because a verb already contains its subject. Both
|
||||||
|
tests are offline and cost nothing. False clarifies 3 to 2, intent-only 74.0% to
|
||||||
|
75.3%.
|
||||||
|
|
||||||
|
The two remaining false clarifies are the act-with-no-allowlisted-fn arm of the
|
||||||
|
gate, not this rule.
|
||||||
|
|
||||||
|
## The destination
|
||||||
|
|
||||||
|
`query` was a shrug. The cascade sorted an utterance into one of seven intents,
|
||||||
|
then `IntentQuery` handed the turn to `querySources` in the daemon. That is
|
||||||
|
twenty-two branches deciding by seed similarity in a fixed order. It had no
|
||||||
|
fixture, no accuracy number, no model arm and no floor.
|
||||||
|
|
||||||
|
`Decision.Source` (`internal/router/source.go`) is the second half of the route
|
||||||
|
(V-655, 2026-08-07). Twelve destinations, not twenty-two. The three recall passes
|
||||||
|
plus `fact-by-key` are one destination from outside. So are search, Kiwix and the
|
||||||
|
URL reader.
|
||||||
|
|
||||||
|
`queryWalk` in `cmd/mavend/actions_query.go` takes sources **out** and moves none.
|
||||||
|
That is the safety argument. The table's order is load-bearing. Every comment on
|
||||||
|
it argues a reason between two sources. Above all it carries "the owner's data
|
||||||
|
first, then the world". Naming `SourceWorld` does not send the turn outside on its
|
||||||
|
own.
|
||||||
|
|
||||||
|
What comes out is only the sources that **guess**. Those decide a turn is theirs
|
||||||
|
by cosine against frozen seeds, then answer whatever they claimed. They hold no
|
||||||
|
table that could come back empty. Weather is the pure case and has no local data
|
||||||
|
at all. It was measured on the box 2026-08-07
|
||||||
|
(`docs/evals/2026-08-07-week-of-usage.md` section 4). It answered both "что такое
|
||||||
|
TCP?" and "сколько будет 17 на 23?" with "для какого города?". The feed answered
|
||||||
|
"какой у меня любимый язык?" with kernel headlines.
|
||||||
|
|
||||||
|
### Who may drop the personal boundary
|
||||||
|
|
||||||
|
The personal boundary guesses, so naming `SourceWorld` drops it. That is what
|
||||||
|
stops it answering "кто такой Линус Торвальдс?" with "не нашла у тебя такой
|
||||||
|
записи", which it did on 2026-08-07.
|
||||||
|
|
||||||
|
Three deciders name a destination and two of them infer it: the heads and the
|
||||||
|
resident model. An inferred `SourceWorld` on a question about him would reach
|
||||||
|
SearXNG. That widens what is asked rather than costing a local answer. So only a
|
||||||
|
stage 0 grammar may drop it (owner's call, V-666, 2026-08-09).
|
||||||
|
|
||||||
|
`Decision.SourceAnchored` carries the provenance. It is a field and not
|
||||||
|
`Stage == 0`. Stage 0 also means confidence 1.0 and an anchored claim band, and
|
||||||
|
one of those could stop implying the others. `definitionQueryPattern` claims "кто
|
||||||
|
такой X", so the 2026-08-07 case is still anchored and still answered.
|
||||||
|
|
||||||
|
### The destination fixture
|
||||||
|
|
||||||
|
`want_source` on `eval.Case` is a pointer, because the destination has three
|
||||||
|
states and a bare string has two. Absent is every intent but query. Present and
|
||||||
|
empty is the `SourceUnknown` contract: name nothing and walk the chain. Present
|
||||||
|
and named is a destination the route must produce. Thirty-three of ninety-six
|
||||||
|
cases carry one.
|
||||||
|
|
||||||
|
A destination miss does **not** fail the case. It lands in `Outcome.SourceReason`
|
||||||
|
and never in `Reasons`, so `Accuracy` and `IntentAccuracy` mean what they meant.
|
||||||
|
`SourceAccuracy` is a second number over the labelled cases alone.
|
||||||
|
A route that lost its intent scores no destination hit. Otherwise a clarify would
|
||||||
|
satisfy an empty label for free.
|
||||||
|
|
||||||
|
Seven cases assert the floor and five of them are homelab operations. They cluster
|
||||||
|
because `SourceRecall`, `SourceNetwork` and `SourceAttention` overlap on every
|
||||||
|
question about the box. `mavpoll` writes its netdata and uptime-kuma observations
|
||||||
|
into the fact store recall reads. That is a finding about the enum, not a gap in
|
||||||
|
the labelling. The other two are `ru-query-005` and `ru-query-014`. No query
|
||||||
|
source reads the reminder store, and a deadline could sit in tasks, the calendar
|
||||||
|
or Praxis. The owner confirmed all seven floor labels on 2026-08-08.
|
||||||
|
|
||||||
|
### The model arm
|
||||||
|
|
||||||
|
`routeGrammar` carries a `source` rule closed over `router.Sources` plus the
|
||||||
|
empty floor (V-660, 2026-08-08). So the model cannot emit a destination that does
|
||||||
|
not exist. The prompt lists the twelve in Russian and says `""` is a normal answer
|
||||||
|
to give often. `LLMRouter.Route` reads it back through `ValidSource` and on
|
||||||
|
`IntentQuery` alone.
|
||||||
|
|
||||||
|
Against gemma-4-12b the cascade scores destination 24/33 with intent unmoved, and
|
||||||
|
recall goes 0/15 to 14/15.
|
||||||
|
|
||||||
|
**Stage 0 now costs four destination points.** It did not before. The four cases
|
||||||
|
the cascade loses and the model alone wins are all calendar. The possessive agenda
|
||||||
|
rules claim them first and name nothing on purpose. That caution was free while
|
||||||
|
nothing downstream could name anything either. It is not free now, and the fix is
|
||||||
|
the owner's call (V-660 open).
|
||||||
|
|
||||||
|
## The decision trace
|
||||||
|
|
||||||
|
Arbitration between the claimants on the utterance stream is order. It is
|
||||||
|
hardcoded in the pre-route resolver ladder, in `buildRouter` and in
|
||||||
|
`querySources`. Nothing recorded who lost until V-564.
|
||||||
|
|
||||||
|
`internal/decision` records one `Record` per turn. It holds every claimant, what
|
||||||
|
it would have made the turn, the score it reported, and how it ended. A claimant
|
||||||
|
won, declined, lost on score, was thinned by a gate or was **never asked**.
|
||||||
|
|
||||||
|
The record rides the context, the same seam `querysource.go` uses. So a claim
|
||||||
|
site cannot change a route, and a context with no record costs nothing. It is
|
||||||
|
installed in `runTurn`, so the mic, telegram and the web leave the same trail.
|
||||||
|
|
||||||
|
Adding a rung to the ladder in `runTurn` means adding its name to `preRouteLadder`
|
||||||
|
in `cmd/mavend/decisiontrace.go`. Otherwise that rung is silently missing from the
|
||||||
|
record.
|
||||||
|
|
||||||
|
### Why it persists now
|
||||||
|
|
||||||
|
The original rule was that nothing persists, because a turn record is read minutes
|
||||||
|
later or never. Storage was a 25-turn in-memory ring read over `ipc.TurnDecisions`
|
||||||
|
and rendered on `/trace`.
|
||||||
|
|
||||||
|
The owner reversed it on 2026-08-06 (V-629,
|
||||||
|
`docs/plans/21-persisting-the-routing-trace.md`). The routing heads cannot be
|
||||||
|
fitted or calibrated without real utterances. And 9 of the 31 modes in
|
||||||
|
`internal/modes` have no seed example at all.
|
||||||
|
|
||||||
|
The ring did not move. `cmd/mavend/routingtrace.go` is a second sink beside it,
|
||||||
|
writing `routing_traces` (migration #23). The utterance is stored in clear. A
|
||||||
|
384-dimension vector of a short sentence is substantially recoverable, so storing
|
||||||
|
vectors instead would be a privacy claim we cannot support. What makes it safe is
|
||||||
|
the same thing that makes the fact store safe. Retention is 14 days, enforced on
|
||||||
|
write and again on start, so a box that goes quiet does not keep every row.
|
||||||
|
Nothing reads it outward. `Store.Wipe` deletes it with everything else.
|
||||||
|
|
||||||
|
### Corrections
|
||||||
|
|
||||||
|
A correction is promoted out into a seed-shaped row in `routing_labels`
|
||||||
|
(migration #24) and kept, because a label is not a transcript. The transcript
|
||||||
|
still expires.
|
||||||
|
|
||||||
|
A turn marked wrong with no target is a usable negative, so naming the intent is
|
||||||
|
never required. The target is one of the seven intents and never free text.
|
||||||
|
|
||||||
|
All three reaches offer it as of 2026-08-06:
|
||||||
|
|
||||||
|
- `/chat` offers two buttons beside the reply, over `ipc.CorrectTurn` and the
|
||||||
|
trace id that rides back on `ipc.ChatReply`.
|
||||||
|
- Voice offers the `repair` rung, which has read spoken corrections since V-455.
|
||||||
|
It now writes the durable label beside the classifier seed it always wrote. A
|
||||||
|
spoken negative with no target is its own rung, `repair-negative` (V-636,
|
||||||
|
`docs/plans/22-correcting-a-turn.md`).
|
||||||
|
- Telegram offers an inline keyboard under the reply. It needed the chat to become
|
||||||
|
readable first (V-637, `docs/plans/23-inbound-telegram.md`). The poller is dark
|
||||||
|
unless the `telegram` block says `intake`. It long-polls, because the box takes
|
||||||
|
no inbound connections. It accepts `chat_id` and no other sender, and it drops
|
||||||
|
whatever queued while the daemon was down. It reaches the daemon through
|
||||||
|
`ipc.CoreAPI` alone.
|
||||||
|
|
||||||
|
The turn source is still `tap:text` for both telegram and the web. So provenance
|
||||||
|
cannot tell a chat turn from a typed one.
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# Raw transcript, two weeks of usage
|
||||||
|
|
||||||
|
YOU: столица Франции
|
||||||
|
MAVEN: вот что я нашла: Франция — Википедия — Она отличается ... реками. На Сене в 145 км от пролива Ла-Манш расположена столица Франции Париж....
|
||||||
|
[2.3s src=search t=670]
|
||||||
|
|
||||||
|
YOU: что такое фотосинтез
|
||||||
|
MAVEN: есть такое: Фотосинтез: процесс, фазы, функции и значение фотосинтеза — Бестселлеры Фоксфорда: собрали всё, что полюбили тысячи школьников и родителей 🧡 · Фотосинтез происходит в тканях, содержащих хлоропласты, — преимущественно в листе, на который приходится большая часть процессов фотосинтеза. Такая ткань наз…
|
||||||
|
[0.8s src=search t=671]
|
||||||
|
|
||||||
|
YOU: что такое TCP?
|
||||||
|
MAVEN: нашла вот это: TCP — TCP
|
||||||
|
TCP
|
||||||
|
Название
|
||||||
|
Transmission Control Protocol
|
||||||
|
Уровень (по модели OSI )
|
||||||
|
Транспортный
|
||||||
|
Семейство
|
||||||
|
TCP/IP
|
||||||
|
Спецификация
|
||||||
|
RFC 793 (сентябрь 1981 года) / STD 7
|
||||||
|
Основные реализации
|
||||||
|
UNIX , Linux , BSD , Windows
|
||||||
|
Расширяемость
|
||||||
|
Опции
|
||||||
|
Медиафайлы на Викискладе
|
||||||
|
TCP ( англ. Transmission Control Protocol — п…
|
||||||
|
[0.3s src=kiwix t=672]
|
||||||
|
|
||||||
|
```
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
{"reply": "вот что я нашла: Франция — Википедия — Она отличается ... реками. На Сене в 145 км от пролива Ла-Манш расположена столица Франции Париж....", "source": "search", "trace": "670", "secs": 2.295847482047975, "day": 0, "n": 1, "utterance": "столица Франции"}
|
||||||
|
{"reply": "есть такое: Фотосинтез: процесс, фазы, функции и значение фотосинтеза — Бестселлеры Фоксфорда: собрали всё, что полюбили тысячи школьников и родителей 🧡 · Фотосинтез происходит в тканях, содержащих хлоропласты, — преимущественно в листе, на который приходится большая часть процессов фотосинтеза. Такая ткань наз…", "source": "search", "trace": "671", "secs": 0.8353607719764113, "day": 0, "n": 2, "utterance": "что такое фотосинтез"}
|
||||||
|
{"reply": "нашла вот это: TCP — TCP\nTCP\nНазвание\nTransmission Control Protocol\nУровень (по модели OSI )\nТранспортный\nСемейство\nTCP/IP\nСпецификация\nRFC 793 (сентябрь 1981 года) / STD 7\nОсновные реализации\nUNIX , Linux , BSD , Windows\nРасширяемость\nОпции\nМедиафайлы на Викискладе\nTCP ( англ. Transmission Control Protocol — п…", "source": "kiwix", "trace": "672", "secs": 0.29547085403464735, "day": 0, "n": 3, "utterance": "что такое TCP?"}
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Raw transcript, two weeks of usage
|
||||||
|
|
||||||
|
YOU: что такое TCP?
|
||||||
|
MAVEN: Вот что я нашла: TCP, или Transmission Control Protocol, — это один из основных протоколов передачи данных в интернете. Он работает на транспортном уровне модели OSI и предоставляет поток данных с установкой соединения, гарантируя целостность данных, повторно запрашивая потерянные данные и устраняя дублирование.
|
||||||
|
[2.3s src=kiwix t=673]
|
||||||
|
|
||||||
|
YOU: кто такой Линус Торвальдс?
|
||||||
|
MAVEN: Вот что я нашла: Линус Торвальдс — финно-американский программист, который является создателем ядра Linux. Он швед по имени Linus Benedict Torvalds, родился 28 декабря 1969 года в Хельсинки, Финляндия, и его род деятельности включает программирование, разработку программного обеспечения, писательство, изобретательство, ведение блога, а также он инженер, бизнесмен и специалист в области информатики.
|
||||||
|
[2.4s src=kiwix t=674]
|
||||||
|
|
||||||
|
YOU: столица Франции
|
||||||
|
MAVEN: Вот что я нашла: Столица Франции — Париж.
|
||||||
|
[2.0s src=kiwix t=675]
|
||||||
|
|
||||||
|
YOU: почему небо голубое
|
||||||
|
MAVEN: Я не нашла в источниках ответа на твой вопрос, почему небо голубое.
|
||||||
|
[1.3s src=kiwix t=676]
|
||||||
|
|
||||||
|
```
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
{"reply": "Вот что я нашла: TCP, или Transmission Control Protocol, — это один из основных протоколов передачи данных в интернете. Он работает на транспортном уровне модели OSI и предоставляет поток данных с установкой соединения, гарантируя целостность данных, повторно запрашивая потерянные данные и устраняя дублирование.", "source": "kiwix", "trace": "673", "secs": 2.3279738989658654, "day": 0, "n": 1, "utterance": "что такое TCP?"}
|
||||||
|
{"reply": "Вот что я нашла: Линус Торвальдс — финно-американский программист, который является создателем ядра Linux. Он швед по имени Linus Benedict Torvalds, родился 28 декабря 1969 года в Хельсинки, Финляндия, и его род деятельности включает программирование, разработку программного обеспечения, писательство, изобретательство, ведение блога, а также он инженер, бизнесмен и специалист в области информатики.", "source": "kiwix", "trace": "674", "secs": 2.42521625792142, "day": 0, "n": 2, "utterance": "кто такой Линус Торвальдс?"}
|
||||||
|
{"reply": "Вот что я нашла: Столица Франции — Париж.", "source": "kiwix", "trace": "675", "secs": 1.9803519028937444, "day": 0, "n": 3, "utterance": "столица Франции"}
|
||||||
|
{"reply": "Я не нашла в источниках ответа на твой вопрос, почему небо голубое.", "source": "kiwix", "trace": "676", "secs": 1.2865088270045817, "day": 0, "n": 4, "utterance": "почему небо голубое"}
|
||||||
Reference in New Issue
Block a user