Routing heads: a slot head, a clarify head, and a two-week baseline to diff against #203

2026-08-08T19:06:22+02:00

kami commented

2026-08-08 19:06:22 +02:00

Closes Vikunja #661.

Rebased onto master after #199, #200 and #201 landed. Replaces #202, which
targeted task/660 and could not be retargeted from the CLI.

Three deliverables, all measured the same day and all documented under docs/evals/.

A third head tags BIO slot spans, distilled from gemma under a closed grammar.
Span F1 72.4% mean over three seeds, at no cost to intent or destination.
A fourth head answers clarify, which a softmax over intents cannot. It catches
7.0 of the 8 want_clarify fixture cases and produces 2.3 false clarifies of 88.
The cascade today misses 1 and produces 2, so this is parity with no rules in
front of it. Confidence separates too: 0.851 where the head is right against
0.604 where it is wrong, replacing the Confidence: 1.0 hardcode with a signal.
scripts/usage-run.py and its 140-turn corpus make the two-week run replayable,
so the next routing change is a diff and not a new opinion.

One correction to CLAUDE.md, in its own commit. The doc said naming SourceWorld
leaves the personal boundary running. The boundary is marked guesses: true, so
queryWalk drops it. That is deliberate and tested, but the doc claimed the
opposite, and whether a model may drop that boundary is the owner's call.

Closes Vikunja #661. Rebased onto master after #199, #200 and #201 landed. Replaces #202, which targeted task/660 and could not be retargeted from the CLI. Three deliverables, all measured the same day and all documented under docs/evals/. - A third head tags BIO slot spans, distilled from gemma under a closed grammar. Span F1 72.4% mean over three seeds, at no cost to intent or destination. - A fourth head answers clarify, which a softmax over intents cannot. It catches 7.0 of the 8 want_clarify fixture cases and produces 2.3 false clarifies of 88. The cascade today misses 1 and produces 2, so this is parity with no rules in front of it. Confidence separates too: 0.851 where the head is right against 0.604 where it is wrong, replacing the Confidence: 1.0 hardcode with a signal. - scripts/usage-run.py and its 140-turn corpus make the two-week run replayable, so the next routing change is a diff and not a new opinion. One correction to CLAUDE.md, in its own commit. The doc said naming SourceWorld leaves the personal boundary running. The boundary is marked guesses: true, so queryWalk drops it. That is deliberate and tested, but the doc claimed the opposite, and whether a model may drop that boundary is the owner's call.

kami added 10 commits 2026-08-08 19:06:23 +02:00

Dump the router prompt where the labeler can read it (V-661) e470435cf1

The training workspace labels with routeSystem and routeGrammar, and it held
its own copies. V-660 changed both. A retyped prompt drifts silently, which is
the problem llm/check_prompt_parity.py exists for on the other side.

Inert unless MAVEN_DUMP_PROMPT names a directory.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

Train the destination head and beat the teacher (V-661) f55bedee2e

Step 3 of the routing-heads plan. Intent and destination share one masked mean
pool on e5-small. Destination scores 26/33 against 12/33 for the classifier
cascade and 24/33 for the cascade with gemma-4-12b, which is the teacher these
labels were distilled from. Recall goes 0/15 to 15/15.

Two heads, not four, and both cuts are label problems rather than GPU time.
Mood describes her own reply state and no dataset maps onto it. BIO slot tags
have no Maven-domain corpus.

The MASSIVE warm-start from step 2 is worth nothing here either. Stock ties it
on intent and leads by a third of a case on destination.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN

Fix the floor corpus and re-measure the destination head (V-661) e69f1bd0cf

The first 120 floor rows carried one sentence shape, because the generator
varies a topic and ambiguity is not a topic. Rotating six shapes takes the
floor 3/7 to 6/7 and the destination mean 75.8% to 80.8%.

Calendar stays 3/6 at every seed. The possessive agenda rules claim those
cases at stage 0 and name nothing, so no label reaches the head.

Also corrects the floor-case count in three files: five of the seven are
homelab, not six.

Record a fortnight of usage as a re-runnable baseline (V-661) 01e80fce4a

The 2026-08-07 week of usage was typed by hand and cannot be replayed, so
it measured a build and not a change. scripts/usage-run.py drives the same
reach from a turns file, which makes the next run a diff.

Baseline is master at beb093a: 140 turns, p50 1.6s, zero errors. Three
defects to move. A parked reminder clarify contaminates 19 later turns and
survives a day boundary. Query sources that guess claim six turns they
cannot answer, which is the class V-655 removes. And one question was read
as a capture.

Also records the slot head: gemma distils 2178 spans, three heads score
intent 92.8%, destination 82.8%, slot span F1 72.4% over three seeds.

Distil a slot head and record it beside the other two (V-661) ed1730431c

BIO tags had no Maven-domain corpus, which was true of found corpora and
false of made ones. A GBNF closed over Maven's five slots plus a
substring check gives 2178 spans out of gemma-4-12b at no second call.

Three heads over one forward pass: intent 92.8%, destination 82.8%, slot
span F1 72.4% over three seeds. The slot head is free.

Epoch selection reads the intent dev slice, so it stops the slot head
about 4 points early. Recorded rather than fixed.

Say that a transport error is not a wrong answer (V-661) 6bc71553ab

A fourth head asks instead of guessing (V-661) 3024f76e5f

Clarify is not a value of intent. It is a second question over the same
pooled vector: can Maven act on this at all. The eight want_clarify fixture
cases sat outside every number the heads measured, because a softmax has no
clarify class.

gen_clarify.py makes the class the corpus lacks. Every existing row was
generated FOR an intent, so every one is answerable. The router-prompt
agreement filter cannot work here, because routeGrammar has no clarify value
and a generated line always agrees with itself. A judge replaces it.

The first judge called 24 of 40 answerable rows underspecified. It judged
against a generic assistant, one that asks where about lunch. Restating
Maven's contract took that to 16 of 60, with all eight fixture cases caught.

Three seeds: 7.0 of 8 caught, 2.3 false of 88. The cascade today misses 1 and
produces 2. Confidence separates too, 0.851 right against 0.604 wrong.

Record the clarify head and the confidence it replaces (V-661) c310115fd2

The personal boundary is a guesser, so say so (V-655) d434f83c2c

CLAUDE.md said naming SourceWorld leaves his notes, his facts and the
personal boundary running first. The first two are true and the third is
not. The boundary is marked guesses: true, so queryWalk drops it whenever
the named destination is not recall.

That is deliberate and tested. It is what stops the boundary answering
'кто такой Линус Торвальдс?' with 'не нашла у тебя такой записи'. But it
means a destination a model wrote can take the boundary off a turn about
him, and the doc claimed the opposite.

Flagged as the owner's call rather than changed. Only the utterance leaves
the box either way.

Merge master after 199-201 landed (V-661) 9a333b23d7

kami merged commit f8fa0d1b44 into master

2026-08-08 19:06:55 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#203