Route with the resident model by default #18

2026-07-31T09:48:24+02:00

claude commented

2026-07-31 09:48:24 +02:00

This is the flip. Maven now routes with the resident model instead of the embedding classifier. Depends on PR 10, PR 15 and PR 17 — merge those first.

You approved this earlier on the condition that the classifier stays wired as a fallback. It does.

	classifier	resident model
held-out intents right	50.0%	63.2%
route errors	0	0
per turn	~30ms	~1s

The two reasons it was unsafe to turn on are both closed:

It can refuse now, and a refusal drops the turn to the classifier (PR 15).
Slot extraction runs on its decisions, so acts get their function and reminders get their time (PR 17).

Any model error does the same thing a refusal does — falls through to the classifier. A turn never breaks because llama-server hiccuped. And if there is no llama-server at all, it logs once and uses the classifier.

One small change in how the setting is read. llm_router is a pointer now, so "not in the config" and "written as false" are different: leaving it out means on, and only an explicit false turns it off. Before, both meant off, which would have made this flip do nothing on any existing config file. Read it with UseLLMRouter(), not the field.

Three tests cover exactly that: absent means on, false means off, true means on.

The honest caveat: 63.2% is better, not good. A third of held-out utterances still route wrong. This is the best config measured on the 0.8B, and the target is still the CPT'd 1.7B. It also costs a second a turn, which you will feel — set "llm_router": false if that is worse for you than the accuracy is worth.

**This is the flip.** Maven now routes with the resident model instead of the embedding classifier. Depends on PR 10, PR 15 and PR 17 — merge those first. You approved this earlier on the condition that the classifier stays wired as a fallback. It does. | | classifier | resident model | |---|---|---| | held-out intents right | 50.0% | **63.2%** | | route errors | 0 | **0** | | per turn | ~30ms | ~1s | The two reasons it was unsafe to turn on are both closed: - It can refuse now, and a refusal drops the turn to the classifier (PR 15). - Slot extraction runs on its decisions, so acts get their function and reminders get their time (PR 17). Any model error does the same thing a refusal does — falls through to the classifier. A turn never breaks because llama-server hiccuped. And if there is no llama-server at all, it logs once and uses the classifier. **One small change in how the setting is read.** `llm_router` is a pointer now, so "not in the config" and "written as false" are different: leaving it out means on, and only an explicit `false` turns it off. Before, both meant off, which would have made this flip do nothing on any existing config file. Read it with `UseLLMRouter()`, not the field. Three tests cover exactly that: absent means on, `false` means off, `true` means on. **The honest caveat:** 63.2% is better, not good. A third of held-out utterances still route wrong. This is the best config measured on the 0.8B, and the target is still the CPT'd 1.7B. It also costs a second a turn, which you will feel — set `"llm_router": false` if that is worse for you than the accuracy is worth.

claude changed target branch from master to overnight/slot-extract

2026-07-31 10:39:23 +02:00

claude added 1 commit 2026-07-31 10:39:23 +02:00

Route with the resident model by default 93c1a41d4a

The two things that made this unsafe are fixed: the router can now
refuse, and slot extraction runs on its decisions.

On the held-out fixture it gets 63.2% of intents right against the
classifier's 50.0%, with no route errors. It costs about a second a
turn instead of 30ms.

The flag is a pointer now, so leaving it out of the config means on
and only writing false turns it off.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

kami commented

2026-07-31 18:49:20 +02:00

is this one even worth reviewing?

kami commented

2026-07-31 20:21:58 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:00 +02:00

kami referenced this issue from a commit

2026-08-04 00:56:13 +02:00

calendar: a Russian summary keeps its letters in the fact key (V-443)

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#18