Say whether clock and calendar questions are system or query (#374) #30

2026-07-31T11:54:22+02:00

claude commented

2026-07-31 11:54:22 +02:00

One commit, 73 lines. Vikunja #374.

Nothing in the router prompt said whether "который час?" is system or query, so the model guessed differently each time.

The rule now says system, and the reason is what the daemon can actually do:

IntentSystem → replySystem, which is the only code holding the clock and the Russian date formatter.
IntentQuery → the embedder and note search, plus the calendar and weather branches.

So the boundary is forced both ways. A clock question routed to query asks the note index what time it is and gets "не знаю". An agenda question routed to system falls to replySystem's default and gets "пока не умею отвечать на этот вопрос". The rule also states both negative sides: what's written in the calendar or memory ("какие есть напоминания") stays query, and a time inside a request ("напомни завтра…") is just a detail of the request. Stage-0 already sent the exact clock grammars to system, so the prompt now agrees with the fast path instead of contradicting it.

The rule sits above the question test on purpose — every one of these utterances contains a question word, so a rule placed later would never be reached.

Numbers — the targeted bug is fixed, the headline didn't move:

	before	after
cascade intent	61.8%	62.3%
cascade full decision	57.9%	59.7%
cascade route errors	0	0

system→query ×4 is gone, and the time and date cases go 0/2 → 2/2 each. But the model over-applies it: query→system ×5 and reminder→system ×2 are new. Net wash inside 77-case noise.

Kept anyway, and here's why — the rule is correct about what the daemon can answer, and it trades a silent failure for a loud one. Before, "который час" got a confident "не знаю" from the note index. Now the misroutes land in replySystem's default arm and say plainly she can't answer that yet.

One fixture case added: an agenda question ("что у меня стоит в календаре на послезавтра" → query) as the guard against an over-broad system rule. The first attempt was rejected by the held-out check for appearing verbatim in the seed phrases, which is the guard working.

The rule was tightened exactly once — the first version had no scope and swept up "напомни завтра в 7". That's a correctness fix, not tuning to the fixture, and it stopped there.

Two things you need to know:

Prompt parity is now broken by three edits and can't be fixed from here. The training workspace and llm/check_prompt_parity.py don't exist on this box. The relabelling copy of the prompt is behind by the query/fact reorder, the unknown wording, and this rule. #362 tracks the first two; this is the third.
This rule makes two real wrong-answer bugs reachable. replySystem answers today's date for "какое число завтра" — it keyword-matches "числ" and ignores the day offset — and its clock arm ignores "в киеве". They were unreachable dead code before, so nothing was lost; now they're live. Filed separately.

One commit, 73 lines. Vikunja #374. Nothing in the router prompt said whether "который час?" is `system` or `query`, so the model guessed differently each time. **The rule now says `system`, and the reason is what the daemon can actually do:** - `IntentSystem` → `replySystem`, which is the **only** code holding the clock and the Russian date formatter. - `IntentQuery` → the embedder and note search, plus the calendar and weather branches. So the boundary is forced both ways. A clock question routed to `query` asks the note index what time it is and gets "не знаю". An agenda question routed to `system` falls to `replySystem`'s default and gets "пока не умею отвечать на этот вопрос". The rule also states both negative sides: what's *written in* the calendar or memory ("какие есть напоминания") stays `query`, and a time inside a request ("напомни завтра…") is just a detail of the request. Stage-0 already sent the exact clock grammars to `system`, so the prompt now agrees with the fast path instead of contradicting it. **The rule sits above the question test on purpose** — every one of these utterances contains a question word, so a rule placed later would never be reached. **Numbers — the targeted bug is fixed, the headline didn't move:** | | before | after | |---|---|---| | cascade intent | 61.8% | 62.3% | | cascade full decision | 57.9% | 59.7% | | cascade route errors | 0 | 0 | `system→query ×4` is **gone**, and the `time` and `date` cases go 0/2 → 2/2 each. But the model over-applies it: `query→system ×5` and `reminder→system ×2` are new. Net wash inside 77-case noise. **Kept anyway, and here's why** — the rule is correct about what the daemon can answer, and it trades a *silent* failure for a *loud* one. Before, "который час" got a confident "не знаю" from the note index. Now the misroutes land in `replySystem`'s default arm and say plainly she can't answer that yet. **One fixture case added:** an agenda question ("что у меня стоит в календаре на послезавтра" → `query`) as the guard against an over-broad system rule. The first attempt was rejected by the held-out check for appearing verbatim in the seed phrases, which is the guard working. The rule was tightened exactly once — the first version had no scope and swept up "напомни завтра в 7". That's a correctness fix, not tuning to the fixture, and it stopped there. **Two things you need to know:** 1. **Prompt parity is now broken by three edits and can't be fixed from here.** The training workspace and `llm/check_prompt_parity.py` don't exist on this box. The relabelling copy of the prompt is behind by the query/fact reorder, the `unknown` wording, and this rule. #362 tracks the first two; this is the third. 2. **This rule makes two real wrong-answer bugs reachable.** `replySystem` answers *today's* date for "какое число завтра" — it keyword-matches "числ" and ignores the day offset — and its clock arm ignores "в киеве". They were unreachable dead code before, so nothing was lost; now they're live. Filed separately.

claude added 2 commits 2026-07-31 11:54:22 +02:00

Give the router prompt a rule for clock and calendar questions (#374 ) bfb57c3148

The prompt named seven intents but never said which one a clock or date
question belongs to, so the model guessed: system->query x4 in every eval
run. The rule now says the clock and the calendar date themselves are
system, what is written in the calendar or in memory stays query, and a
time named inside a request is just part of the request.

That split follows what the daemon can answer. Only replySystem owns the
clock and the date formatter, while the agenda is answered from
CalendarEvents inside the query branch.

Also adds one calendar-agenda fixture case so an over-broad system rule
cannot pass unnoticed, and writes up the before/after numbers. The
targeted confusion is gone; the headline accuracy did not move.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ

Merge the clock and calendar routing rule b6f47fbeb6

kami commented

2026-07-31 20:22:21 +02:00

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47.

Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

Superseded by #47, which landed this whole stack on master as one reviewed integration merge. This PR head is an ancestor of master — its commits are in, nothing here is lost. Closing as merged-by-proxy rather than merged, since the merge came in through #47. Review threads on this PR were answered or acted on before the merge; the Russian wording fixes went in as #48.

kami closed this pull request

2026-07-31 20:22:22 +02:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#30