The router prompt has no destination, so the model arm of V-655 names nothing #201

2026-08-08T16:23:24+02:00

claude commented

2026-08-08 16:23:24 +02:00

Closes Vikunja #660.

Opened during an unattended overnight run: the diff-budget block was off (task overnight). Read the diff, not only the tests.

Acceptance criteria and quality gate are in TASK.md on this branch.
Review the review, not the diff — leave comments and the agent will apply them via task start 660.

Closes Vikunja #660. Opened during an unattended overnight run: the diff-budget block was off (`task overnight`). Read the diff, not only the tests. Acceptance criteria and quality gate are in `TASK.md` on this branch. Review the review, not the diff — leave comments and the agent will apply them via `task start 660`.

claude added 8 commits 2026-08-08 16:23:24 +02:00

Say where the answer lives, not just that it is a question (V-655) b5500a5be8

A question was sorted twice. The cascade picked one of seven intents with
stage 0 rules, the resident model and the classifier behind it, a 91-case
fixture measuring it and the decision trace recording it. Then IntentQuery
handed the turn to a second dispatch in the daemon, twenty-two branches
deciding by seed similarity in a fixed order, with none of that. The careful
sorter did the easy half.

Decision grows a Source: twelve destinations, not twenty-two, because the
recall passes are one destination from the outside and so are the three world
sources. Empty is a real value and it is the floor — nothing names one, the
daemon walks its whole chain, and that is exactly what shipped before.

Stage 0 fills it where a deterministic rule already knows. Two new world rules
for the shapes measured failing on the box on 2026-08-07: "что такое TCP?" and
"кто такой Линус Торвальдс?" were answered by weather and by the personal
boundary, and "сколько будет 17 на 23?" was answered "для какого города?".
The calendar noun rule and the closed event-noun rule name the calendar. The
possessive agenda rules deliberately do not: "что у меня в списке покупок"
matches agenda-query, and naming the calendar there would take the list off
the turn.

Fixture unchanged at 69/91, which is the point — it scores intent, and none of
these cases changes intent.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A named destination silences the guessers and moves nobody (V-655) ab3e818bb9

querySources splits in two once you look at which sources over-claimed during
the week of 2026-08-07. The clean ones perform a lookup and can come back
empty: fact-by-key, tasks, list, money, calendar, notes. The dirty ones decide
by cosine against frozen seeds and then answer whatever they claimed, because
they have no lookup that could miss. Weather has no local table at all, which
is why "что такое TCP?" became "для какого города?".

So each source now carries its destination and whether it guesses, and
queryWalk takes the guessers that were not named OUT of the chain. It removes
and never reorders, which is the whole safety argument: the table's order is
load-bearing, every comment on it argues a reason between two sources, and
above all it carries "his data first, then the world". Naming SourceWorld does
not send the turn outside. It stops weather claiming a protocol on the way
past. His notes, his facts and the boundary in front of them still run first,
so a wrong destination costs nothing but the guess it prevented.

The skipped sources are recorded as never-asked with the reason, so /trace
shows a narrowed walk rather than a chain that silently shrank.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Record the second half of a route in CLAUDE.md (V-655) 7203cd56fd

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Score the destination apart from the intent (V-659) 2597a7b34a

The fixture measured the first half of a route and stopped. V-655 split a
routing decision in two, and the second half arrived with no fixture, so
Decision.Source had no accuracy number at all.

want_source is a pointer because the destination has three states and a
bare string has two. Absent is every intent but query, which never reaches
queryWalk. Present and empty is the SourceUnknown contract: name nothing
and let the daemon walk the chain, which is right whenever two destinations
can both answer and the utterance does not choose. Present and named is a
destination the route must produce.

A destination miss does not fail the case. It goes in SourceReason, never
in Reasons, so Accuracy and IntentAccuracy stay the numbers they were and
69/91 still means what it meant. SourceAccuracy is the second number, over
the labelled cases only, because a percentage of the whole fixture would be
a percentage of turns that never ask a query source.

A clarified or mis-routed case still counts in the denominator. It named no
destination and that is a miss, not a case to skip, or the denominator drops
every turn the route already lost.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

Label the destination on 33 fixture cases (V-659) b6eaa704a2

Twenty-eight existing query cases get a want_source and five new ones
arrive with theirs. Every label is the destination that SHOULD claim the
turn, which on the five new cases is not the one that did: they were
observed failing on the box on 2026-08-07, so the fixture fails on the day
it is written.

Seven cases assert the SourceUnknown floor, and six of those are homelab
operations. They cluster because SourceRecall, SourceNetwork and
SourceAttention overlap on every question about the box: mavpoll writes its
netdata and uptime-kuma observations into the fact store recall reads.
Naming one destination there takes the other two off a turn that needs
them. That is a finding about the enum, not a gap in the labelling.

The fixture's grammar mirror had drifted. WorldQueryGrammars went into
buildRouter with V-655 and never into baselineGrammars, so the fixture was
scoring a grammar set the daemon does not run — the exact thing the comment
above that function forbids. Adding it moved the destination number 9/33 to
12/33 and moved nothing else.

Measured classifier+onnx: intent 73/96 (76.0%), was 69/91 (75.8%). Four of
the five new cases pass and no existing case moved. Destination 12/33
(36.4%), and the split is the point. World is 5/5, because a stage 0 rule
names it. Calendar is 2/6, because the possessive agenda rules deliberately
do not. Recall is 0/15, because nothing anywhere names it yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

Record the destination number and two stuck measurements (V-659) c15c2b7bd2

CLAUDE.md said the destination had no fixture and no accuracy number. It
has both now: intent 73/96 and destination 12/33 on the classifier cascade,
with the per-destination split, the floor cases and the grammar drift the
labelling turned up. Anyone adding a grammar now reads that baselineGrammars
mirrors buildRouter and drifts silently when it does not.

docs/evals/2026-08-08-massive-warm-start.md was written on the V-655 branch
and parked in .task/, which git excludes, so it was one `task start` away
from being lost. It is a dated measurement and it belongs under docs/evals
whatever branch produced it. Its "destination has no fixture at all" line is
now a pointer to the file beside it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

Give the router prompt a destination to write (V-660) 3513e508b7

V-659 measured the destination at 12/33 on the classifier cascade and named
the gap: recall 0/15, because nothing anywhere names it. The model could not
help, for a structural reason rather than a capability one. Nothing in
routeSystem mentioned a Source and routeGrammar could not emit one, so there
was no string for it to write. Same shape as the Praxis reach V-517
measured at 0/12.

routeGrammar grows a source rule, closed over router.Sources plus the empty
floor. A grammar cannot emit a destination that does not exist, which is the
guarantee V-546 wants from a softmax and gets here for free. The prompt
lists the twelve in Russian, one line each, and says plainly that "" is a
normal answer to give often: two sources that can both answer means the
chain walks, and guessing is the failure mode this whole field exists to
stop.

The read-back goes through ValidSource and runs on IntentQuery alone. The
grammar already bounds the enum, but it is a request to a server that may be
running another build, and only a query reaches queryWalk.

Measured against gemma-4-12b on the workstation, same fixture, cascade with
a hash fallback: destination 24/33 (72.7%) against the classifier's 12/33,
and intent 81/96 (84.4%) which is where it already was. Recall is the whole
move, 0/15 to 14/15. The model alone scores 26/33.

Four cases the cascade loses and llm-only wins are calendar. The possessive
agenda rules claim them at stage 0 and deliberately name nothing, because
"что у меня в списке покупок" matches the same rule and naming the calendar
would take the list source off the turn. So stage 0's caution now costs four
destination points it did not cost before. That is a real trade and it wants
its own argument, not a quiet edit here.

The resident Qwen3-1.7B is unmeasured: it binds --port 0 inside the
container and no host process can reach it.

llm/check_prompt_parity.py in the training workspace compares its copy of
routeSystem to this one and will fail until that copy gets the same edit.
V-362 covers the catch-up.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

Record the model arm, and the stage 0 trade it exposed (V-660) 00f9239ef9

Numbers and the argument in docs/evals/2026-08-08-destination-model-arm.md,
pointer and the short version in CLAUDE.md. The finding worth carrying is
not the 72.7%: it is that stage 0's silence on the possessive agenda rules
used to be free and now costs four destination points, because there is
finally something downstream that would have named the calendar.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

kami merged commit 663b5c47b9 into master

2026-08-08 19:03:29 +02:00

kami referenced this pull request

2026-08-08 19:06:22 +02:00

Routing heads: a slot head, a clarify head, and a two-week baseline to diff against #203

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#201