Measure the fact parser: land the corpus on master (V-586) #181

2026-08-06T16:23:54+02:00

claude commented

2026-08-06 16:23:54 +02:00

Carries PR #176 (the 286-line fact-parser corpus test and docs/evals/2026-08-06-fact-parser.md) up to master. #176 targeted the V-586 code branch, which has since merged.

🤖 Generated with Claude Code

https://claude.ai/code/session_0117tgnmbgZpHVV3XSNw8Qua

Carries PR #176 (the 286-line fact-parser corpus test and docs/evals/2026-08-06-fact-parser.md) up to master. #176 targeted the V-586 code branch, which has since merged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_0117tgnmbgZpHVV3XSNw8Qua

claude added 2 commits 2026-08-06 16:23:54 +02:00

the fact parser gets a corpus, and the LLM arm gets run (V-586) dde556a3d3

V-586 reported 64/91 on the RU routing fixture, unchanged. That number does not
bear on the change: the fixture holds three fact cases and all three miss on
intent, so DefaultFactParser is never reached and any parser edit scores as
"unchanged".

So the parser gets its own corpus, 91 cases, scored against BOTH
implementations — the closed classes that ship and legacyFactParse, a verbatim
copy of the substring parser at 0445693, frozen in the test file so the
comparison reruns. True positives 35/40 to 39/40, misfires rejected 8/15 to
14/15. The rewrite wins every case anyone argued about.

The third case class is the point: 36 sentences a person would plainly say
whose word is in no lexicon set. The old parser caught 3 by accident, the new
one catches 0. "ем суп", "вздремнул", "помылся", "перекур", "i napped". A
silent miss is this parser's worst failure mode and the corpus sizes it.

Two defects recorded rather than fixed, since this branch measures: "допил
воду" misses because the dictionary lemmatises допил to допилить, the same saw
collision drink_verbs carries пил for; and the oblique cases of душ go with the
exact match that keeps the soul out.

The LLM arm the original commit skipped is run here against gemma-4-12b on the
workstation at 192.168.1.105:8080 — it was reachable all along, the failure was
the shell's HTTP_PROXY. cascade+llm 85.7% to 86.8%, one case, same failing set,
variance. Full write-up in docs/evals/2026-08-06-fact-parser.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117tgnmbgZpHVV3XSNw8Qua

Merge pull request 'Measure the fact parser: the closed classes are a floor, not an answer' (#176 ) from task/586-measure-the-fact-parser into task/586-defaultfactparser-uses-hand-written-russ 97e1a44c1a

claude merged commit 6499f6365e into master

2026-08-06 16:25:48 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#181