Merge pull request 'Measure the fact parser: land the corpus on master (V-586)' (#181) from task/586-defaultfactparser-uses-hand-written-russ into master

2026-08-06 16:25:47 +02:00
parent 1b3af05d0a 97e1a44c1a
commit 6499f6365e
2 changed files with 533 additions and 0 deletions
@@ -0,0 +1,247 @@
 # The fact parser: closed classes against the substring stems they replaced
 Measured 2026-08-06 at 22edc3c and its parent 0445693, on the corpus in
 `internal/router/factparser_corpus_test.go`. Dated file: it is not edited after
 today, and a newer number is a new file.
 V-586 rewrote `DefaultFactParser` off hand-written Russian stems onto six closed
 classes in `internal/lexicon`. Its commit message reported 64/91 on the RU
 routing fixture, unchanged. That number does not bear on the change: the fixture
 holds three fact cases and all three miss on intent, so the parser is never
 reached. This file measures the parser directly, and runs the LLM arm the
 original commit skipped.
 **The rewrite is better on the utterances it was designed for and no worse on
 the ones it was not.** True positives go 35/40 to 39/40, misfires rejected go
 8/15 to 14/15. What neither version has is coverage: of 36 plausible utterances
 whose word is in no lexicon set, the substring parser caught 3 by accident and
 the closed-class parser catches 0. That is the honest headline. The word list
 did not shrink the vocabulary — it never had one — it made the boundary visible.
 ## The corpus
 91 cases, three classes. **True positives** are sentences the owner would say,
 with the key that must be written. **Misfires** are sentences the substring
 parser wrote a fact for and should not have, including the three hard negatives
 the rewrite was argued on. **False negatives** are sentences a reasonable person
 would say whose word is in no set at all; `want` is the key a human would
 assign, and the ship parser is expected to miss them. They are the measurement
 of what a closed class costs, not a bug list.
 The old parser is carried in the test file as `legacyFactParse`, copied verbatim
 from 0445693, so the comparison reruns:
 ```sh
 deps/go/go/bin/go test -run TestFactParserCorpus -v ./internal/router/
 ```
 ## Score
 | | true positives | misfires rejected | false-negative cases recovered |
 |---|---|---|---|
 | old (substring stems, 0445693) | 35/40 | 8/15 | 3/36 |
 | **new (closed classes, 22edc3c)** | **39/40** | **14/15** | **0/36** |
 Sixteen cases disagree. Eleven of them the rewrite wins, three it loses, and two
 are cases neither gets.
 **Won.** Every misfire the commit message named — `душа болит`, `в комнате
 душно`, `это была беда`, `наша победа`, `на душе легко` — plus `водитель пилота
 ждёт`, where two stems in one sentence made the old water arm fire. And five true
 positives the stems simply did not list: `перекусил`, `передохнул`, `отдыхаю`,
 `пойду спать`, `i showered`. Morphology buys those; a stem list would need a new
 entry for each.
 **Lost.** `допил воду` is a real regression and the only failing true positive.
 The vendored dictionary lemmatises `допил` to `допилить`, to finish sawing —
 exactly the collision `drink_verbs` already carries `пил` and `пили` as surface
 forms to dodge, left unhandled for the prefixed form. `допить` is in the set and
 the sentence still misses. It is flagged `broken` in the corpus rather than
 fixed, because this branch measures.
 `был в душе` and `после душа полегчало` are the price of matching the shower set
 exactly. The dictionary makes `душ` and `душа` one word, so a lemma test cannot
 tell a shower from a soul; exact matching keeps `на душе легко` out and loses
 the oblique cases of the real noun with it. The old parser got both by accident,
 along with the soul. That trade is right — writing a shower fact when he said
 his soul feels light is worse than missing one — but it is a trade and the two
 rows are what it costs.
 `недоспал` is the third loss and the least defensible: the old substring `спал`
 caught it, and `недоспать` is in no set.
 **Neither.** `обеденный перерыв отменили` — a cancelled lunch break — is a fact
 for both parsers, `meal` for the old one off the adjective and `break` for the
 new one off `перерыв`. Nothing in either design reads the cancellation.
 `дрых до обеда` is scored `meal` by both, because the meal arm runs first and
 `обеда` is in it, which is not wrong so much as beside the point.
 ## The false-negative surface
 This is the half the routing fixture cannot see and the half that decides
 whether the design holds. 36 cases, 0 recovered:
 - **water** — `выпил чаю`, `глотнул воды`, `хлебнул воды`, `выпил стакан`,
  `i hydrated`, `finished my bottle of water`. The water arm needs a noun AND a
  verb, so an elided noun or an unlisted verb drops the whole capture.
 - **meal** — `ем суп`, `съел бутерброд`, `наелся`, `пожрал`, `полдник был`,
  `snack`, `supper`, `brunch`, `i eat now`. `есть` is deliberately absent for
  `есть новости по бэкапу`, and `ем`, its most ordinary spoken form, goes with it.
 - **shower** — `помылся`, `сходил в ванную`, `искупался`, `i am showering`,
  plus the two oblique cases above.
 - **break** — `сделал передышку`, `перекур`, `полежал немного`, `сделал паузу`,
  `i took five`, `resting now`.
 - **sleep** — `вздремнул`, `прикорнул`, `дрых`, `недоспал`, `лёг в двенадцать`,
  `сон был короткий`, `i napped`, `took a nap`.
 None of these are exotic. They are the second and third word a person reaches
 for, and every one of them is a fact the owner stated and Maven silently did not
 record. A silent miss is the worst failure mode this parser has: he said it, she
 heard it, nothing was written, and nothing told him.
 ## The routing fixture, LLM arm
 The arm 22edc3c skipped. `MAVEN_LLM_URL` points the harness at any llama-server;
 the previous run reported none reachable, which was the shell's `HTTP_PROXY` and
 not the network. Run against **gemma-4-12B-it-qat-UD-Q4_K_XL on the workstation
 at `192.168.1.105:8080`**, the same box as the 02-08 measurement, with
 `NO_PROXY=192.168.1.105`:
 ```sh
 NO_PROXY=192.168.1.105 no_proxy=192.168.1.105 \
  make eval-models MAVEN_LLM_URL=http://192.168.1.105:8080
 ```
 | | full | intent-only | p50 |
 |---|---|---|---|
 | llm-only, 0445693 | 51.6% (47/91) | 82.4% | — |
 | llm-only, 22edc3c | 52.7% (48/91) | 83.5% | 341ms |
 | cascade+llm, 0445693 | 85.7% (78/91) | 93.4% | — |
 | **cascade+llm, 22edc3c** | **86.8% (79/91)** | **94.5%** | 334ms |
 One case either way, both directions, and the failing set is identical between
 the two commits. That is run-to-run variance on a sampling model, not a signal.
 The parser change is invisible to the routing fixture on the LLM arm for the
 same reason it is invisible on the classifier arm: the three fact cases miss on
 intent and the parser is never called. Do not read these rows as evidence about
 the parser. They are evidence that the fixture cannot answer the question, which
 is why the corpus above exists.
 ## Verdict
 The closed-class rewrite holds up as a rewrite. It is strictly better than what
 it replaced on both classes anyone argued about, and the one regression
 (`допил`) and one bad trade (the oblique `душ`) are both dictionary collisions
 rather than design faults.
 It does not hold up as an answer. A closed class is the right mechanism for a
 set that is actually closed — interrogatives, weekdays, cardinals — and
 "the words a person uses to say he ate" is not that set. The corpus puts a
 number on it: 36 ordinary sentences, 0 recovered, and every new one costs a
 lexicon edit by whoever notices. The three mechanisms CLAUDE.md names do not
 contain the right one for this job. The embedder-topic mechanism is the closest
 fit and is wrong too, because this is slot extraction rather than aboutness.
 This is a case for the V-546 slot-tagging head. Self-care facts are a bounded
 key space (five keys) over unbounded surface forms, which is exactly what a BIO
 tagger on e5-small is for: it generalises to `вздремнул` without anyone adding
 `вздремнуть` to a list, and max softmax gives the confidence the parser's
 hardcoded `true` does not have. Until it lands, the closed classes are the
 correct floor and the 36 rows above are the size of the gap they leave.
 ## The corpus, case by case
 | utterance | class | want | old (substring) | new (closed class) |
 |---|---|---|---|---|
 | `выпил стакан воды` | tp | water | water | water |
 | `попил воды` | tp | water | water | water |
 | `я попил водички` | tp | water | water | water |
 | `пью воду` | tp | water | water | water |
 | `воду пил уже` | tp | water | water | water |
 | `допил воду` | tp | water | water | — **≠** |
 | `запил таблетку водой` | tp | water | water | water |
 | `drank water` | tp | water | water | water |
 | `i drank some water` | tp | water | water | water |
 | `поужинал` | tp | meal | meal | meal |
 | `я пообедал` | tp | meal | meal | meal |
 | `позавтракал кашей` | tp | meal | meal | meal |
 | `перекусил бутербродом` | tp | meal | — | meal **≠** |
 | `покушал` | tp | meal | meal | meal |
 | `поел супа` | tp | meal | meal | meal |
 | `обед был в час` | tp | meal | meal | meal |
 | `ужинать буду позже` | tp | meal | meal | meal |
 | `i ate` | tp | meal | meal | meal |
 | `had lunch` | tp | meal | meal | meal |
 | `dinner done` | tp | meal | meal | meal |
 | `принял душ` | tp | shower | shower | shower |
 | `сходил в душ` | tp | shower | shower | shower |
 | `душ принят` | tp | shower | shower | shower |
 | `ополоснулся душем` | tp | shower | shower | shower |
 | `took a shower` | tp | shower | shower | shower |
 | `i showered` | tp | shower | — | shower **≠** |
 | `сделал перерыв` | tp | break | break | break |
 | `отдохнул полчаса` | tp | break | break | break |
 | `передохнул немного` | tp | break | — | break **≠** |
 | `отдыхаю` | tp | break | — | break **≠** |
 | `был перерыв на обед` | tp | meal | meal | meal |
 | `took a break` | tp | break | break | break |
 | `спал восемь часов` | tp | sleep | sleep | sleep |
 | `спала плохо` | tp | sleep | sleep | sleep |
 | `поспал днём` | tp | sleep | sleep | sleep |
 | `выспался наконец` | tp | sleep | sleep | sleep |
 | `проспал будильник` | tp | sleep | sleep | sleep |
 | `пойду спать` | tp | sleep | — | sleep **≠** |
 | `slept 8 hours` | tp | sleep | sleep | sleep |
 | `i slept badly` | tp | sleep | sleep | sleep |
 | `пилот сказал что вылет через час` | misfire | — | — | — |
 | `водитель уже подъехал` | misfire | — | — | — |
 | `надо заводить машину` | misfire | — | — | — |
 | `душа болит` | misfire | — | shower | — **≠** |
 | `в комнате душно` | misfire | — | shower | — **≠** |
 | `это была беда` | misfire | — | meal | — **≠** |
 | `наша победа` | misfire | — | meal | — **≠** |
 | `пила лежит в гараже` | misfire | — | — | — |
 | `водитель пилота ждёт` | misfire | — | water | — **≠** |
 | `обеденный перерыв отменили` | misfire | — | meal | break **≠** |
 | `есть новости по бэкапу базы` | misfire | — | — | — |
 | `напоминания на завтра есть` | misfire | — | — | — |
 | `на душе легко` | misfire | — | shower | — **≠** |
 | `пилил доску весь вечер` | misfire | — | — | — |
 | `поставь будильник на завтра` | misfire | — | — | — |
 | `выпил чаю` | fn | water | — | — |
 | `глотнул воды` | fn | water | — | — |
 | `хлебнул воды` | fn | water | — | — |
 | `воды хлебнул из бутылки` | fn | water | — | — |
 | `выпил стакан` | fn | water | — | — |
 | `i hydrated` | fn | water | — | — |
 | `finished my bottle of water` | fn | water | — | — |
 | `ем суп` | fn | meal | — | — |
 | `съел бутерброд` | fn | meal | — | — |
 | `наелся` | fn | meal | — | — |
 | `пожрал` | fn | meal | — | — |
 | `полдник был` | fn | meal | — | — |
 | `i had a snack` | fn | meal | — | — |
 | `having supper` | fn | meal | — | — |
 | `brunch was good` | fn | meal | — | — |
 | `i eat now` | fn | meal | — | — |
 | `был в душе` | fn | shower | shower | — **≠** |
 | `после душа полегчало` | fn | shower | shower | — **≠** |
 | `помылся` | fn | shower | — | — |
 | `сходил в ванную` | fn | shower | — | — |
 | `искупался` | fn | shower | — | — |
 | `i am showering` | fn | shower | — | — |
 | `сделал передышку` | fn | break | — | — |
 | `перекур` | fn | break | — | — |
 | `полежал немного` | fn | break | — | — |
 | `сделал паузу` | fn | break | — | — |
 | `i took five` | fn | break | — | — |
 | `resting now` | fn | break | — | — |
 | `вздремнул` | fn | sleep | — | — |
 | `прикорнул на диване` | fn | sleep | — | — |
 | `дрых до обеда` | fn | sleep | meal | meal |
 | `недоспал` | fn | sleep | sleep | — **≠** |
 | `лёг в двенадцать` | fn | sleep | — | — |
 | `сон был короткий` | fn | sleep | — | — |
 | `i napped` | fn | sleep | — | — |
 | `took a nap` | fn | sleep | — | — |
 `≠` marks a disagreement. `—` is no fact written.
@@ -0,0 +1,286 @@
 package router
 import (
 	"strconv"
 	"strings"
 	"testing"
 )
 // The fact-parser corpus (V-586). DefaultFactParser is the only thing standing
 // between a spoken sentence and a written self-care fact, and until this file
 // it was covered by a handful of examples chosen by whoever last edited it. The
 // RU routing fixture does not cover it either: it holds three fact cases and
 // all three miss on intent, so the parser is never reached and a parser change
 // scores as "unchanged".
 //
 // So the corpus is here, and it scores BOTH implementations: the closed-class
 // parser that ships, and legacyFactParse below, a faithful copy of the
 // hand-written substring version 22edc3c replaced. The table is the measurement
 // — see docs/evals/2026-08-06-fact-parser.md for the numbers as of that day.
 //
 // Three case classes, and the third is the point:
 //
 //	true positive  — a sentence he would say, with the key it must write.
 //	misfire        — a sentence the substring parser wrote a fact for and
 //	                 should not have. want is "".
 //	false-negative — a sentence he would plausibly say whose word is in NO
 //	                 lexicon set. want is the key a human would assign, and
 //	                 the ship parser is EXPECTED to miss it. These measure the
 //	                 cost of the word-list design, not a bug in it.
 //
 // Nothing here asserts a score. A closed class is a decision about vocabulary
 // and the bar belongs in a dated eval, not in an assertion that turns every
 // vocabulary edit into a red test. What it does fail on is a regression in the
 // two classes that are not judgement calls: a true positive that stops matching
 // and a misfire that starts.
 type factCase struct {
 	utterance string
 	want      string // "" — no fact
 	class     string // "tp", "misfire", "fn"
 	broken    bool   // ship parser is known to get this wrong; see the 06-08 eval
 	note      string
 }
 var factCorpus = []factCase{
 	// ---- water, true positives -------------------------------------------
 	{"выпил стакан воды", "water", "tp", false, ""},
 	{"попил воды", "water", "tp", false, ""},
 	{"я попил водички", "water", "tp", false, ""},
 	{"пью воду", "water", "tp", false, ""},
 	{"воду пил уже", "water", "tp", false, ""},
 	{"допил воду", "water", "tp", true, "REGRESSION: the dictionary lemmatises допил to допилить, to finish sawing — the same saw collision drink_verbs already carries пил for, unfixed for the prefixed form"},
 	{"запил таблетку водой", "water", "tp", false, ""},
 	{"drank water", "water", "tp", false, ""},
 	{"i drank some water", "water", "tp", false, ""},
 	// ---- meal, true positives --------------------------------------------
 	{"поужинал", "meal", "tp", false, ""},
 	{"я пообедал", "meal", "tp", false, ""},
 	{"позавтракал кашей", "meal", "tp", false, ""},
 	{"перекусил бутербродом", "meal", "tp", false, ""},
 	{"покушал", "meal", "tp", false, ""},
 	{"поел супа", "meal", "tp", false, ""},
 	{"обед был в час", "meal", "tp", false, ""},
 	{"ужинать буду позже", "meal", "tp", false, ""},
 	{"i ate", "meal", "tp", false, ""},
 	{"had lunch", "meal", "tp", false, ""},
 	{"dinner done", "meal", "tp", false, ""},
 	// ---- shower, true positives ------------------------------------------
 	{"принял душ", "shower", "tp", false, ""},
 	{"сходил в душ", "shower", "tp", false, ""},
 	{"душ принят", "shower", "tp", false, ""},
 	{"ополоснулся душем", "shower", "tp", false, ""},
 	{"took a shower", "shower", "tp", false, ""},
 	{"i showered", "shower", "tp", false, ""},
 	// ---- break, true positives -------------------------------------------
 	{"сделал перерыв", "break", "tp", false, ""},
 	{"отдохнул полчаса", "break", "tp", false, ""},
 	{"передохнул немного", "break", "tp", false, ""},
 	{"отдыхаю", "break", "tp", false, ""},
 	{"был перерыв на обед", "meal", "tp", false, "meal wins the switch; both keys are true of the sentence"},
 	{"took a break", "break", "tp", false, ""},
 	// ---- sleep, true positives -------------------------------------------
 	{"спал восемь часов", "sleep", "tp", false, ""},
 	{"спала плохо", "sleep", "tp", false, "she may report her own; the parser is speaker-agnostic"},
 	{"поспал днём", "sleep", "tp", false, ""},
 	{"выспался наконец", "sleep", "tp", false, ""},
 	{"проспал будильник", "sleep", "tp", false, ""},
 	{"пойду спать", "sleep", "tp", false, ""},
 	{"slept 8 hours", "sleep", "tp", false, ""},
 	{"i slept badly", "sleep", "tp", false, ""},
 	// ---- misfires 22edc3c set out to reject -------------------------------
 	{"пилот сказал что вылет через час", "", "misfire", false, "substring пил"},
 	{"водитель уже подъехал", "", "misfire", false, "substring вод"},
 	{"надо заводить машину", "", "misfire", false, "substring вод"},
 	{"душа болит", "", "misfire", false, "substring душ"},
 	{"в комнате душно", "", "misfire", false, "substring душ"},
 	{"это была беда", "", "misfire", false, "substring еда"},
 	{"наша победа", "", "misfire", false, "substring еда"},
 	{"пила лежит в гараже", "", "misfire", false, "substring пил"},
 	{"водитель пилота ждёт", "", "misfire", false, "both stems in one sentence — the old water arm fires"},
 	{"обеденный перерыв отменили", "", "misfire", true, "neither parser gets this: the old one writes meal off the adjective, the new one writes break off перерыв. A cancelled break is not a break taken."},
 	// ---- hard negatives that decide the design ----------------------------
 	{"есть новости по бэкапу базы", "", "misfire", false, "есть is the existential, not a meal"},
 	{"напоминания на завтра есть", "", "misfire", false, "завтра carries завтрак as a substring"},
 	{"на душе легко", "", "misfire", false, "душе is the soul, and also the prepositional of душ"},
 	{"пилил доску весь вечер", "", "misfire", false, ""},
 	{"поставь будильник на завтра", "", "misfire", false, "завтра again, no meal"},
 	// ---- false negatives: words in no lexicon set -------------------------
 	// Water.
 	{"выпил чаю", "water", "fn", false, "hydration by any liquid; чай is in no set"},
 	{"глотнул воды", "water", "fn", false, "глотнуть is not a drink verb"},
 	{"хлебнул воды", "water", "fn", false, "хлебнуть is not a drink verb"},
 	{"воды хлебнул из бутылки", "water", "fn", false, ""},
 	{"выпил стакан", "water", "fn", false, "the noun is elided; he says this"},
 	{"i hydrated", "water", "fn", false, "hydrate is in no set"},
 	{"finished my bottle of water", "water", "fn", false, "no drink verb in the English set"},
 	// Meal.
 	{"ем суп", "meal", "fn", false, "есть is deliberately absent, and this is the cost"},
 	{"съел бутерброд", "meal", "fn", false, "съесть is in no set"},
 	{"наелся", "meal", "fn", false, "наесться is in no set"},
 	{"пожрал", "meal", "fn", false, "coarse but spoken"},
 	{"полдник был", "meal", "fn", false, "полдник is in no set"},
 	{"i had a snack", "meal", "fn", false, "snack is in no set"},
 	{"having supper", "meal", "fn", false, "supper is in no set"},
 	{"brunch was good", "meal", "fn", false, "brunch is in no set"},
 	{"i eat now", "meal", "fn", false, "eat is in no set — only ate is"},
 	// Shower.
 	{"был в душе", "shower", "fn", false, "prepositional; anyExact cannot take it without taking the soul"},
 	{"после душа полегчало", "shower", "fn", false, "genitive, same collision"},
 	{"помылся", "shower", "fn", false, "помыться is in no set"},
 	{"сходил в ванную", "shower", "fn", false, "ванная is in no set"},
 	{"искупался", "shower", "fn", false, "искупаться is in no set"},
 	{"i am showering", "shower", "fn", false, "showering is not a member and the set is exact-matched"},
 	// Break.
 	{"сделал передышку", "break", "fn", false, "передышка is in no set"},
 	{"перекур", "break", "fn", false, "перекур is in no set"},
 	{"полежал немного", "break", "fn", false, "полежать is in no set"},
 	{"сделал паузу", "break", "fn", false, "пауза is in no set"},
 	{"i took five", "break", "fn", false, "idiomatic, in no set"},
 	{"resting now", "break", "fn", false, "resting is not a member and rest is matched by lemma only"},
 	// Sleep.
 	{"вздремнул", "sleep", "fn", false, "вздремнуть is in no set"},
 	{"прикорнул на диване", "sleep", "fn", false, "прикорнуть is in no set"},
 	{"дрых до обеда", "sleep", "fn", false, "дрыхнуть is in no set — and обед makes this a MEAL for both parsers"},
 	{"недоспал", "sleep", "fn", false, "недоспать is in no set"},
 	{"лёг в двенадцать", "sleep", "fn", false, "лечь is in no set"},
 	{"сон был короткий", "sleep", "fn", false, "сон deliberately absent"},
 	{"i napped", "sleep", "fn", false, "nap is in no set"},
 	{"took a nap", "sleep", "fn", false, "break matches nothing here either"},
 }
 // legacyFactParse — DefaultFactParser exactly as it stood at 22edc3c's parent
 // (0445693), copied here so the corpus scores the trade rather than describing
 // it. Do not fix it. It is a frozen baseline, and the day it stops being worth
 // comparing against, delete it and the two-column table with it.
 func legacyFactParse(utterance string) (string, string, bool) {
 	s := strings.ToLower(strings.TrimSpace(utterance))
 	hasRoot := func(root string) bool { return strings.Contains(s, root) }
 	containsWord := func(w string) bool {
 		for _, tok := range strings.Fields(s) {
 			if tok == w {
 				return true
 			}
 		}
 		return false
 	}
 	switch {
 	case (containsWord("water") && containsWord("drank")) ||
 		(hasRoot("вод") && (hasRoot("пил") || hasRoot("пью") || hasRoot("пей"))):
 		return "water", `"drank"`, true
 	case containsWord("meal") || (containsWord("ate") && !containsWord("backup")) || containsWord("lunch") || containsWord("dinner") ||
 		hasRoot("поел") || hasRoot("поесть") || hasRoot("куша") || hasRoot("обед") || hasRoot("ужин") || hasRoot("завтрак") || hasRoot("еда"):
 		return "meal", `"ate"`, true
 	case containsWord("shower") || hasRoot("душ"):
 		return "shower", `"took"`, true
 	case containsWord("break") || hasRoot("перерыв") || hasRoot("отдох"):
 		return "break", `"took"`, true
 	case containsWord("slept") || containsWord("sleep") ||
 		hasRoot("спал") || hasRoot("выспал"):
 		if v, ok := parseDurationValue(afterWord(s, "slept")); ok {
 			return "sleep", strconv.Quote(v), true
 		}
 		return "sleep", `"slept"`, true
 	}
 	return "", "", false
 }
 // TestFactParserCorpus scores both parsers over the corpus and prints the
 // side-by-side. Run it with -v; the table is the output.
 func TestFactParserCorpus(t *testing.T) {
 	type tally struct{ tpHit, tpMiss, misfire, misfireOK, fnHit, fnMiss int }
 	var now, old tally
 	var rows []string
 	rows = append(rows, "| utterance | class | want | old (substring) | new (closed class) |")
 	rows = append(rows, "|---|---|---|---|---|")
 	score := func(key string, ok bool, c factCase, tl *tally) string {
 		got := ""
 		if ok {
 			got = key
 		}
 		switch c.class {
 		case "tp":
 			if got == c.want {
 				tl.tpHit++
 			} else {
 				tl.tpMiss++
 			}
 		case "misfire":
 			if got == c.want {
 				tl.misfireOK++
 			} else {
 				tl.misfire++
 			}
 		case "fn":
 			if got == c.want {
 				tl.fnHit++
 			} else {
 				tl.fnMiss++
 			}
 		}
 		if got == "" {
 			return "—"
 		}
 		return got
 	}
 	for _, c := range factCorpus {
 		nk, _, nok := DefaultFactParser{}.Parse(c.utterance)
 		ok, _, ook := legacyFactParse(c.utterance)
 		gotNew := score(nk, nok, c, &now)
 		gotOld := score(ok, ook, c, &old)
 		want := c.want
 		if want == "" {
 			want = "—"
 		}
 		mark := ""
 		if gotOld != gotNew {
 			mark = " **≠**"
 		}
 		rows = append(rows, "| `"+c.utterance+"` | "+c.class+" | "+want+" | "+gotOld+" | "+gotNew+mark+" |")
 	}
 	line := func(name string, tl tally) string {
 		return name +
 			": true positives " + strconv.Itoa(tl.tpHit) + "/" + strconv.Itoa(tl.tpHit+tl.tpMiss) +
 			", misfires rejected " + strconv.Itoa(tl.misfireOK) + "/" + strconv.Itoa(tl.misfireOK+tl.misfire) +
 			", false-negative cases recovered " + strconv.Itoa(tl.fnHit) + "/" + strconv.Itoa(tl.fnHit+tl.fnMiss)
 	}
 	t.Log("\n" + strings.Join(rows, "\n") + "\n\n" + line("old (substring)", old) + "\n" + line("new (closed class)", now))
 	// The two regressions that are not judgement calls.
 	for _, c := range factCorpus {
 		k, _, ok := DefaultFactParser{}.Parse(c.utterance)
 		got := ""
 		if ok {
 			got = k
 		}
 		if c.class == "fn" {
 			continue
 		}
 		if c.broken {
 			// Recorded as wrong on 06-08. If it starts passing, someone fixed
 			// it and the flag is now a lie — say so rather than staying green.
 			if got == c.want {
 				t.Errorf("%q now returns %q as wanted — drop its broken flag", c.utterance, got)
 			}
 			continue
 		}
 		if got != c.want {
 			t.Errorf("%s %q: got %q, want %q (%s)", c.class, c.utterance, got, c.want, c.note)
 		}
 	}
 }