Maven/docs/evals/2026-08-06-fact-parser.md

The fact parser: closed classes against the substring stems they replaced

Measured 2026-08-06 at 22edc3c and its parent 0445693, on the corpus in internal/router/factparser_corpus_test.go. Dated file: it is not edited after today, and a newer number is a new file.

V-586 rewrote DefaultFactParser off hand-written Russian stems onto six closed classes in internal/lexicon. Its commit message reported 64/91 on the RU routing fixture, unchanged. That number does not bear on the change: the fixture holds three fact cases and all three miss on intent, so the parser is never reached. This file measures the parser directly, and runs the LLM arm the original commit skipped.

The rewrite is better on the utterances it was designed for and no worse on the ones it was not. True positives go 35/40 to 39/40, misfires rejected go 8/15 to 14/15. What neither version has is coverage: of 36 plausible utterances whose word is in no lexicon set, the substring parser caught 3 by accident and the closed-class parser catches 0. That is the honest headline. The word list did not shrink the vocabulary — it never had one — it made the boundary visible.

The corpus

91 cases, three classes. True positives are sentences the owner would say, with the key that must be written. Misfires are sentences the substring parser wrote a fact for and should not have, including the three hard negatives the rewrite was argued on. False negatives are sentences a reasonable person would say whose word is in no set at all; want is the key a human would assign, and the ship parser is expected to miss them. They are the measurement of what a closed class costs, not a bug list.

The old parser is carried in the test file as legacyFactParse, copied verbatim from 0445693, so the comparison reruns:

deps/go/go/bin/go test -run TestFactParserCorpus -v ./internal/router/

Score

	true positives	misfires rejected	false-negative cases recovered
old (substring stems, 0445693)	35/40	8/15	3/36
new (closed classes, 22edc3c)	39/40	14/15	0/36

Sixteen cases disagree. Eleven of them the rewrite wins, three it loses, and two are cases neither gets.

Won. Every misfire the commit message named — душа болит, в комнате душно, это была беда, наша победа, на душе легко — plus водитель пилота ждёт, where two stems in one sentence made the old water arm fire. And five true positives the stems simply did not list: перекусил, передохнул, отдыхаю, пойду спать, i showered. Morphology buys those; a stem list would need a new entry for each.

Lost. допил воду is a real regression and the only failing true positive. The vendored dictionary lemmatises допил to допилить, to finish sawing — exactly the collision drink_verbs already carries пил and пили as surface forms to dodge, left unhandled for the prefixed form. допить is in the set and the sentence still misses. It is flagged broken in the corpus rather than fixed, because this branch measures.

был в душе and после душа полегчало are the price of matching the shower set exactly. The dictionary makes душ and душа one word, so a lemma test cannot tell a shower from a soul; exact matching keeps на душе легко out and loses the oblique cases of the real noun with it. The old parser got both by accident, along with the soul. That trade is right — writing a shower fact when he said his soul feels light is worse than missing one — but it is a trade and the two rows are what it costs.

недоспал is the third loss and the least defensible: the old substring спал caught it, and недоспать is in no set.

Neither. обеденный перерыв отменили — a cancelled lunch break — is a fact for both parsers, meal for the old one off the adjective and break for the new one off перерыв. Nothing in either design reads the cancellation. дрых до обеда is scored meal by both, because the meal arm runs first and обеда is in it, which is not wrong so much as beside the point.

The false-negative surface

This is the half the routing fixture cannot see and the half that decides whether the design holds. 36 cases, 0 recovered:

• water — выпил чаю, глотнул воды, хлебнул воды, выпил стакан, i hydrated, finished my bottle of water. The water arm needs a noun AND a verb, so an elided noun or an unlisted verb drops the whole capture.
• meal — ем суп, съел бутерброд, наелся, пожрал, полдник был, snack, supper, brunch, i eat now. есть is deliberately absent for есть новости по бэкапу, and ем, its most ordinary spoken form, goes with it.
• shower — помылся, сходил в ванную, искупался, i am showering, plus the two oblique cases above.
• break — сделал передышку, перекур, полежал немного, сделал паузу, i took five, resting now.
• sleep — вздремнул, прикорнул, дрых, недоспал, лёг в двенадцать, сон был короткий, i napped, took a nap.

None of these are exotic. They are the second and third word a person reaches for, and every one of them is a fact the owner stated and Maven silently did not record. A silent miss is the worst failure mode this parser has: he said it, she heard it, nothing was written, and nothing told him.

The routing fixture, LLM arm

The arm 22edc3c skipped. MAVEN_LLM_URL points the harness at any llama-server; the previous run reported none reachable, which was the shell's HTTP_PROXY and not the network. Run against gemma-4-12B-it-qat-UD-Q4_K_XL on the workstation at 192.168.1.105:8080, the same box as the 02-08 measurement, with NO_PROXY=192.168.1.105:

NO_PROXY=192.168.1.105 no_proxy=192.168.1.105 \
  make eval-models MAVEN_LLM_URL=http://192.168.1.105:8080

	full	intent-only	p50
llm-only, 0445693	51.6% (47/91)	82.4%	—
llm-only, 22edc3c	52.7% (48/91)	83.5%	341ms
cascade+llm, 0445693	85.7% (78/91)	93.4%	—
cascade+llm, 22edc3c	86.8% (79/91)	94.5%	334ms

One case either way, both directions, and the failing set is identical between the two commits. That is run-to-run variance on a sampling model, not a signal. The parser change is invisible to the routing fixture on the LLM arm for the same reason it is invisible on the classifier arm: the three fact cases miss on intent and the parser is never called. Do not read these rows as evidence about the parser. They are evidence that the fixture cannot answer the question, which is why the corpus above exists.

Verdict

The closed-class rewrite holds up as a rewrite. It is strictly better than what it replaced on both classes anyone argued about, and the one regression (допил) and one bad trade (the oblique душ) are both dictionary collisions rather than design faults.

It does not hold up as an answer. A closed class is the right mechanism for a set that is actually closed — interrogatives, weekdays, cardinals — and "the words a person uses to say he ate" is not that set. The corpus puts a number on it: 36 ordinary sentences, 0 recovered, and every new one costs a lexicon edit by whoever notices. The three mechanisms CLAUDE.md names do not contain the right one for this job. The embedder-topic mechanism is the closest fit and is wrong too, because this is slot extraction rather than aboutness.

This is a case for the V-546 slot-tagging head. Self-care facts are a bounded key space (five keys) over unbounded surface forms, which is exactly what a BIO tagger on e5-small is for: it generalises to вздремнул without anyone adding вздремнуть to a list, and max softmax gives the confidence the parser's hardcoded true does not have. Until it lands, the closed classes are the correct floor and the 36 rows above are the size of the gap they leave.

The corpus, case by case

utterance	class	want	old (substring)	new (closed class)
выпил стакан воды	tp	water	water	water
попил воды	tp	water	water	water
я попил водички	tp	water	water	water
пью воду	tp	water	water	water
воду пил уже	tp	water	water	water
допил воду	tp	water	water	— ≠
запил таблетку водой	tp	water	water	water
drank water	tp	water	water	water
i drank some water	tp	water	water	water
поужинал	tp	meal	meal	meal
я пообедал	tp	meal	meal	meal
позавтракал кашей	tp	meal	meal	meal
перекусил бутербродом	tp	meal	—	meal ≠
покушал	tp	meal	meal	meal
поел супа	tp	meal	meal	meal
обед был в час	tp	meal	meal	meal
ужинать буду позже	tp	meal	meal	meal
i ate	tp	meal	meal	meal
had lunch	tp	meal	meal	meal
dinner done	tp	meal	meal	meal
принял душ	tp	shower	shower	shower
сходил в душ	tp	shower	shower	shower
душ принят	tp	shower	shower	shower
ополоснулся душем	tp	shower	shower	shower
took a shower	tp	shower	shower	shower
i showered	tp	shower	—	shower ≠
сделал перерыв	tp	break	break	break
отдохнул полчаса	tp	break	break	break
передохнул немного	tp	break	—	break ≠
отдыхаю	tp	break	—	break ≠
был перерыв на обед	tp	meal	meal	meal
took a break	tp	break	break	break
спал восемь часов	tp	sleep	sleep	sleep
спала плохо	tp	sleep	sleep	sleep
поспал днём	tp	sleep	sleep	sleep
выспался наконец	tp	sleep	sleep	sleep
проспал будильник	tp	sleep	sleep	sleep
пойду спать	tp	sleep	—	sleep ≠
slept 8 hours	tp	sleep	sleep	sleep
i slept badly	tp	sleep	sleep	sleep
пилот сказал что вылет через час	misfire	—	—	—
водитель уже подъехал	misfire	—	—	—
надо заводить машину	misfire	—	—	—
душа болит	misfire	—	shower	— ≠
в комнате душно	misfire	—	shower	— ≠
это была беда	misfire	—	meal	— ≠
наша победа	misfire	—	meal	— ≠
пила лежит в гараже	misfire	—	—	—
водитель пилота ждёт	misfire	—	water	— ≠
обеденный перерыв отменили	misfire	—	meal	break ≠
есть новости по бэкапу базы	misfire	—	—	—
напоминания на завтра есть	misfire	—	—	—
на душе легко	misfire	—	shower	— ≠
пилил доску весь вечер	misfire	—	—	—
поставь будильник на завтра	misfire	—	—	—
выпил чаю	fn	water	—	—
глотнул воды	fn	water	—	—
хлебнул воды	fn	water	—	—
воды хлебнул из бутылки	fn	water	—	—
выпил стакан	fn	water	—	—
i hydrated	fn	water	—	—
finished my bottle of water	fn	water	—	—
ем суп	fn	meal	—	—
съел бутерброд	fn	meal	—	—
наелся	fn	meal	—	—
пожрал	fn	meal	—	—
полдник был	fn	meal	—	—
i had a snack	fn	meal	—	—
having supper	fn	meal	—	—
brunch was good	fn	meal	—	—
i eat now	fn	meal	—	—
был в душе	fn	shower	shower	— ≠
после душа полегчало	fn	shower	shower	— ≠
помылся	fn	shower	—	—
сходил в ванную	fn	shower	—	—
искупался	fn	shower	—	—
i am showering	fn	shower	—	—
сделал передышку	fn	break	—	—
перекур	fn	break	—	—
полежал немного	fn	break	—	—
сделал паузу	fn	break	—	—
i took five	fn	break	—	—
resting now	fn	break	—	—
вздремнул	fn	sleep	—	—
прикорнул на диване	fn	sleep	—	—
дрых до обеда	fn	sleep	meal	meal
недоспал	fn	sleep	sleep	— ≠
лёг в двенадцать	fn	sleep	—	—
сон был короткий	fn	sleep	—	—
i napped	fn	sleep	—	—
took a nap	fn	sleep	—	—
≠ marks a disagreement. — is no fact written.