Measure the fact parser: the closed classes are a floor, not an answer #176

2026-08-06T10:22:51+02:00

kami commented

2026-08-06 10:22:51 +02:00

Stacked on #175. Measures it rather than changing it — slots.go and the lexicon are untouched here.

The routing fixture cannot judge this change. Its three fact cases miss on intent, so the parser is never called. The 64/91 quoted on #175 was flat for that reason, not because behaviour was preserved.

The LLM arm ran, against gemma-4-12b on the workstation. The earlier no llama-server reachable was the shell proxy; NO_PROXY=192.168.1.105 is the whole fix and MAVEN_LLM_URL already existed. Cascade+LLM 85.7% → 86.8%, one case, identical failing sets. Noise.

New corpus, internal/router/factparser_corpus_test.go, 91 cases, both parsers side by side:

	true positives	misfires rejected	false negatives recovered
old (substring)	35/40	8/15	3/36
new (closed class)	39/40	14/15	0/36

36 of 36 false negatives: ем суп, наелся, полдник, глотнул воды, помылся, вздремнул, перекур, i napped and eighteen more. Neither parser gets any of them.

One genuine regression in #175, flagged broken in the table so it cannot rot: допил воду misses because the dictionary lemmatises допил to допилить, to finish sawing.

Write-up: docs/evals/2026-08-06-fact-parser.md.

The conclusion is that the closed classes are the correct floor and V-546 is the answer. This PR is the gap they leave, written down and rerunnable.

Stacked on #175. Measures it rather than changing it — `slots.go` and the lexicon are untouched here. **The routing fixture cannot judge this change.** Its three fact cases miss on intent, so the parser is never called. The 64/91 quoted on #175 was flat for that reason, not because behaviour was preserved. **The LLM arm ran**, against gemma-4-12b on the workstation. The earlier `no llama-server reachable` was the shell proxy; `NO_PROXY=192.168.1.105` is the whole fix and `MAVEN_LLM_URL` already existed. Cascade+LLM 85.7% → 86.8%, one case, identical failing sets. Noise. **New corpus**, `internal/router/factparser_corpus_test.go`, 91 cases, both parsers side by side: | | true positives | misfires rejected | false negatives recovered | |---|---|---|---| | old (substring) | 35/40 | 8/15 | 3/36 | | new (closed class) | 39/40 | 14/15 | 0/36 | **36 of 36 false negatives**: `ем суп`, `наелся`, `полдник`, `глотнул воды`, `помылся`, `вздремнул`, `перекур`, `i napped` and eighteen more. Neither parser gets any of them. One genuine regression in #175, flagged `broken` in the table so it cannot rot: `допил воду` misses because the dictionary lemmatises `допил` to `допилить`, to finish sawing. Write-up: `docs/evals/2026-08-06-fact-parser.md`. The conclusion is that the closed classes are the correct floor and V-546 is the answer. This PR is the gap they leave, written down and rerunnable.

kami added 1 commit 2026-08-06 10:22:52 +02:00

the fact parser gets a corpus, and the LLM arm gets run (V-586) dde556a3d3

V-586 reported 64/91 on the RU routing fixture, unchanged. That number does not
bear on the change: the fixture holds three fact cases and all three miss on
intent, so DefaultFactParser is never reached and any parser edit scores as
"unchanged".

So the parser gets its own corpus, 91 cases, scored against BOTH
implementations — the closed classes that ship and legacyFactParse, a verbatim
copy of the substring parser at 0445693, frozen in the test file so the
comparison reruns. True positives 35/40 to 39/40, misfires rejected 8/15 to
14/15. The rewrite wins every case anyone argued about.

The third case class is the point: 36 sentences a person would plainly say
whose word is in no lexicon set. The old parser caught 3 by accident, the new
one catches 0. "ем суп", "вздремнул", "помылся", "перекур", "i napped". A
silent miss is this parser's worst failure mode and the corpus sizes it.

Two defects recorded rather than fixed, since this branch measures: "допил
воду" misses because the dictionary lemmatises допил to допилить, the same saw
collision drink_verbs carries пил for; and the oblique cases of душ go with the
exact match that keeps the soul out.

The LLM arm the original commit skipped is run here against gemma-4-12b on the
workstation at 192.168.1.105:8080 — it was reachable all along, the failure was
the shell's HTTP_PROXY. cascade+llm 85.7% to 86.8%, one case, same failing set,
variance. Full write-up in docs/evals/2026-08-06-fact-parser.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117tgnmbgZpHVV3XSNw8Qua

claude merged commit 97e1a44c1a into task/586-defaultfactparser-uses-hand-written-russ

2026-08-06 16:21:35 +02:00

claude referenced this pull request

2026-08-06 16:23:54 +02:00

Measure the fact parser: land the corpus on master (V-586) #181

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#176