diff --git a/docs/evals/2026-08-06-fact-parser.md b/docs/evals/2026-08-06-fact-parser.md new file mode 100644 index 0000000..40c99ad --- /dev/null +++ b/docs/evals/2026-08-06-fact-parser.md @@ -0,0 +1,247 @@ +# The fact parser: closed classes against the substring stems they replaced + +Measured 2026-08-06 at 22edc3c and its parent 0445693, on the corpus in +`internal/router/factparser_corpus_test.go`. Dated file: it is not edited after +today, and a newer number is a new file. + +V-586 rewrote `DefaultFactParser` off hand-written Russian stems onto six closed +classes in `internal/lexicon`. Its commit message reported 64/91 on the RU +routing fixture, unchanged. That number does not bear on the change: the fixture +holds three fact cases and all three miss on intent, so the parser is never +reached. This file measures the parser directly, and runs the LLM arm the +original commit skipped. + +**The rewrite is better on the utterances it was designed for and no worse on +the ones it was not.** True positives go 35/40 to 39/40, misfires rejected go +8/15 to 14/15. What neither version has is coverage: of 36 plausible utterances +whose word is in no lexicon set, the substring parser caught 3 by accident and +the closed-class parser catches 0. That is the honest headline. The word list +did not shrink the vocabulary — it never had one — it made the boundary visible. + +## The corpus + +91 cases, three classes. **True positives** are sentences the owner would say, +with the key that must be written. **Misfires** are sentences the substring +parser wrote a fact for and should not have, including the three hard negatives +the rewrite was argued on. **False negatives** are sentences a reasonable person +would say whose word is in no set at all; `want` is the key a human would +assign, and the ship parser is expected to miss them. They are the measurement +of what a closed class costs, not a bug list. + +The old parser is carried in the test file as `legacyFactParse`, copied verbatim +from 0445693, so the comparison reruns: + +```sh +deps/go/go/bin/go test -run TestFactParserCorpus -v ./internal/router/ +``` + +## Score + +| | true positives | misfires rejected | false-negative cases recovered | +|---|---|---|---| +| old (substring stems, 0445693) | 35/40 | 8/15 | 3/36 | +| **new (closed classes, 22edc3c)** | **39/40** | **14/15** | **0/36** | + +Sixteen cases disagree. Eleven of them the rewrite wins, three it loses, and two +are cases neither gets. + +**Won.** Every misfire the commit message named — `душа болит`, `в комнате +душно`, `это была беда`, `наша победа`, `на душе легко` — plus `водитель пилота +ждёт`, where two stems in one sentence made the old water arm fire. And five true +positives the stems simply did not list: `перекусил`, `передохнул`, `отдыхаю`, +`пойду спать`, `i showered`. Morphology buys those; a stem list would need a new +entry for each. + +**Lost.** `допил воду` is a real regression and the only failing true positive. +The vendored dictionary lemmatises `допил` to `допилить`, to finish sawing — +exactly the collision `drink_verbs` already carries `пил` and `пили` as surface +forms to dodge, left unhandled for the prefixed form. `допить` is in the set and +the sentence still misses. It is flagged `broken` in the corpus rather than +fixed, because this branch measures. + +`был в душе` and `после душа полегчало` are the price of matching the shower set +exactly. The dictionary makes `душ` and `душа` one word, so a lemma test cannot +tell a shower from a soul; exact matching keeps `на душе легко` out and loses +the oblique cases of the real noun with it. The old parser got both by accident, +along with the soul. That trade is right — writing a shower fact when he said +his soul feels light is worse than missing one — but it is a trade and the two +rows are what it costs. + +`недоспал` is the third loss and the least defensible: the old substring `спал` +caught it, and `недоспать` is in no set. + +**Neither.** `обеденный перерыв отменили` — a cancelled lunch break — is a fact +for both parsers, `meal` for the old one off the adjective and `break` for the +new one off `перерыв`. Nothing in either design reads the cancellation. +`дрых до обеда` is scored `meal` by both, because the meal arm runs first and +`обеда` is in it, which is not wrong so much as beside the point. + +## The false-negative surface + +This is the half the routing fixture cannot see and the half that decides +whether the design holds. 36 cases, 0 recovered: + +- **water** — `выпил чаю`, `глотнул воды`, `хлебнул воды`, `выпил стакан`, + `i hydrated`, `finished my bottle of water`. The water arm needs a noun AND a + verb, so an elided noun or an unlisted verb drops the whole capture. +- **meal** — `ем суп`, `съел бутерброд`, `наелся`, `пожрал`, `полдник был`, + `snack`, `supper`, `brunch`, `i eat now`. `есть` is deliberately absent for + `есть новости по бэкапу`, and `ем`, its most ordinary spoken form, goes with it. +- **shower** — `помылся`, `сходил в ванную`, `искупался`, `i am showering`, + plus the two oblique cases above. +- **break** — `сделал передышку`, `перекур`, `полежал немного`, `сделал паузу`, + `i took five`, `resting now`. +- **sleep** — `вздремнул`, `прикорнул`, `дрых`, `недоспал`, `лёг в двенадцать`, + `сон был короткий`, `i napped`, `took a nap`. + +None of these are exotic. They are the second and third word a person reaches +for, and every one of them is a fact the owner stated and Maven silently did not +record. A silent miss is the worst failure mode this parser has: he said it, she +heard it, nothing was written, and nothing told him. + +## The routing fixture, LLM arm + +The arm 22edc3c skipped. `MAVEN_LLM_URL` points the harness at any llama-server; +the previous run reported none reachable, which was the shell's `HTTP_PROXY` and +not the network. Run against **gemma-4-12B-it-qat-UD-Q4_K_XL on the workstation +at `192.168.1.105:8080`**, the same box as the 02-08 measurement, with +`NO_PROXY=192.168.1.105`: + +```sh +NO_PROXY=192.168.1.105 no_proxy=192.168.1.105 \ + make eval-models MAVEN_LLM_URL=http://192.168.1.105:8080 +``` + +| | full | intent-only | p50 | +|---|---|---|---| +| llm-only, 0445693 | 51.6% (47/91) | 82.4% | — | +| llm-only, 22edc3c | 52.7% (48/91) | 83.5% | 341ms | +| cascade+llm, 0445693 | 85.7% (78/91) | 93.4% | — | +| **cascade+llm, 22edc3c** | **86.8% (79/91)** | **94.5%** | 334ms | + +One case either way, both directions, and the failing set is identical between +the two commits. That is run-to-run variance on a sampling model, not a signal. +The parser change is invisible to the routing fixture on the LLM arm for the +same reason it is invisible on the classifier arm: the three fact cases miss on +intent and the parser is never called. Do not read these rows as evidence about +the parser. They are evidence that the fixture cannot answer the question, which +is why the corpus above exists. + +## Verdict + +The closed-class rewrite holds up as a rewrite. It is strictly better than what +it replaced on both classes anyone argued about, and the one regression +(`допил`) and one bad trade (the oblique `душ`) are both dictionary collisions +rather than design faults. + +It does not hold up as an answer. A closed class is the right mechanism for a +set that is actually closed — interrogatives, weekdays, cardinals — and +"the words a person uses to say he ate" is not that set. The corpus puts a +number on it: 36 ordinary sentences, 0 recovered, and every new one costs a +lexicon edit by whoever notices. The three mechanisms CLAUDE.md names do not +contain the right one for this job. The embedder-topic mechanism is the closest +fit and is wrong too, because this is slot extraction rather than aboutness. + +This is a case for the V-546 slot-tagging head. Self-care facts are a bounded +key space (five keys) over unbounded surface forms, which is exactly what a BIO +tagger on e5-small is for: it generalises to `вздремнул` without anyone adding +`вздремнуть` to a list, and max softmax gives the confidence the parser's +hardcoded `true` does not have. Until it lands, the closed classes are the +correct floor and the 36 rows above are the size of the gap they leave. + +## The corpus, case by case + +| utterance | class | want | old (substring) | new (closed class) | +|---|---|---|---|---| +| `выпил стакан воды` | tp | water | water | water | +| `попил воды` | tp | water | water | water | +| `я попил водички` | tp | water | water | water | +| `пью воду` | tp | water | water | water | +| `воду пил уже` | tp | water | water | water | +| `допил воду` | tp | water | water | — **≠** | +| `запил таблетку водой` | tp | water | water | water | +| `drank water` | tp | water | water | water | +| `i drank some water` | tp | water | water | water | +| `поужинал` | tp | meal | meal | meal | +| `я пообедал` | tp | meal | meal | meal | +| `позавтракал кашей` | tp | meal | meal | meal | +| `перекусил бутербродом` | tp | meal | — | meal **≠** | +| `покушал` | tp | meal | meal | meal | +| `поел супа` | tp | meal | meal | meal | +| `обед был в час` | tp | meal | meal | meal | +| `ужинать буду позже` | tp | meal | meal | meal | +| `i ate` | tp | meal | meal | meal | +| `had lunch` | tp | meal | meal | meal | +| `dinner done` | tp | meal | meal | meal | +| `принял душ` | tp | shower | shower | shower | +| `сходил в душ` | tp | shower | shower | shower | +| `душ принят` | tp | shower | shower | shower | +| `ополоснулся душем` | tp | shower | shower | shower | +| `took a shower` | tp | shower | shower | shower | +| `i showered` | tp | shower | — | shower **≠** | +| `сделал перерыв` | tp | break | break | break | +| `отдохнул полчаса` | tp | break | break | break | +| `передохнул немного` | tp | break | — | break **≠** | +| `отдыхаю` | tp | break | — | break **≠** | +| `был перерыв на обед` | tp | meal | meal | meal | +| `took a break` | tp | break | break | break | +| `спал восемь часов` | tp | sleep | sleep | sleep | +| `спала плохо` | tp | sleep | sleep | sleep | +| `поспал днём` | tp | sleep | sleep | sleep | +| `выспался наконец` | tp | sleep | sleep | sleep | +| `проспал будильник` | tp | sleep | sleep | sleep | +| `пойду спать` | tp | sleep | — | sleep **≠** | +| `slept 8 hours` | tp | sleep | sleep | sleep | +| `i slept badly` | tp | sleep | sleep | sleep | +| `пилот сказал что вылет через час` | misfire | — | — | — | +| `водитель уже подъехал` | misfire | — | — | — | +| `надо заводить машину` | misfire | — | — | — | +| `душа болит` | misfire | — | shower | — **≠** | +| `в комнате душно` | misfire | — | shower | — **≠** | +| `это была беда` | misfire | — | meal | — **≠** | +| `наша победа` | misfire | — | meal | — **≠** | +| `пила лежит в гараже` | misfire | — | — | — | +| `водитель пилота ждёт` | misfire | — | water | — **≠** | +| `обеденный перерыв отменили` | misfire | — | meal | break **≠** | +| `есть новости по бэкапу базы` | misfire | — | — | — | +| `напоминания на завтра есть` | misfire | — | — | — | +| `на душе легко` | misfire | — | shower | — **≠** | +| `пилил доску весь вечер` | misfire | — | — | — | +| `поставь будильник на завтра` | misfire | — | — | — | +| `выпил чаю` | fn | water | — | — | +| `глотнул воды` | fn | water | — | — | +| `хлебнул воды` | fn | water | — | — | +| `воды хлебнул из бутылки` | fn | water | — | — | +| `выпил стакан` | fn | water | — | — | +| `i hydrated` | fn | water | — | — | +| `finished my bottle of water` | fn | water | — | — | +| `ем суп` | fn | meal | — | — | +| `съел бутерброд` | fn | meal | — | — | +| `наелся` | fn | meal | — | — | +| `пожрал` | fn | meal | — | — | +| `полдник был` | fn | meal | — | — | +| `i had a snack` | fn | meal | — | — | +| `having supper` | fn | meal | — | — | +| `brunch was good` | fn | meal | — | — | +| `i eat now` | fn | meal | — | — | +| `был в душе` | fn | shower | shower | — **≠** | +| `после душа полегчало` | fn | shower | shower | — **≠** | +| `помылся` | fn | shower | — | — | +| `сходил в ванную` | fn | shower | — | — | +| `искупался` | fn | shower | — | — | +| `i am showering` | fn | shower | — | — | +| `сделал передышку` | fn | break | — | — | +| `перекур` | fn | break | — | — | +| `полежал немного` | fn | break | — | — | +| `сделал паузу` | fn | break | — | — | +| `i took five` | fn | break | — | — | +| `resting now` | fn | break | — | — | +| `вздремнул` | fn | sleep | — | — | +| `прикорнул на диване` | fn | sleep | — | — | +| `дрых до обеда` | fn | sleep | meal | meal | +| `недоспал` | fn | sleep | sleep | — **≠** | +| `лёг в двенадцать` | fn | sleep | — | — | +| `сон был короткий` | fn | sleep | — | — | +| `i napped` | fn | sleep | — | — | +| `took a nap` | fn | sleep | — | — | +`≠` marks a disagreement. `—` is no fact written. diff --git a/internal/router/factparser_corpus_test.go b/internal/router/factparser_corpus_test.go new file mode 100644 index 0000000..68ae694 --- /dev/null +++ b/internal/router/factparser_corpus_test.go @@ -0,0 +1,286 @@ +package router + +import ( + "strconv" + "strings" + "testing" +) + +// The fact-parser corpus (V-586). DefaultFactParser is the only thing standing +// between a spoken sentence and a written self-care fact, and until this file +// it was covered by a handful of examples chosen by whoever last edited it. The +// RU routing fixture does not cover it either: it holds three fact cases and +// all three miss on intent, so the parser is never reached and a parser change +// scores as "unchanged". +// +// So the corpus is here, and it scores BOTH implementations: the closed-class +// parser that ships, and legacyFactParse below, a faithful copy of the +// hand-written substring version 22edc3c replaced. The table is the measurement +// — see docs/evals/2026-08-06-fact-parser.md for the numbers as of that day. +// +// Three case classes, and the third is the point: +// +// true positive — a sentence he would say, with the key it must write. +// misfire — a sentence the substring parser wrote a fact for and +// should not have. want is "". +// false-negative — a sentence he would plausibly say whose word is in NO +// lexicon set. want is the key a human would assign, and +// the ship parser is EXPECTED to miss it. These measure the +// cost of the word-list design, not a bug in it. +// +// Nothing here asserts a score. A closed class is a decision about vocabulary +// and the bar belongs in a dated eval, not in an assertion that turns every +// vocabulary edit into a red test. What it does fail on is a regression in the +// two classes that are not judgement calls: a true positive that stops matching +// and a misfire that starts. + +type factCase struct { + utterance string + want string // "" — no fact + class string // "tp", "misfire", "fn" + broken bool // ship parser is known to get this wrong; see the 06-08 eval + note string +} + +var factCorpus = []factCase{ + // ---- water, true positives ------------------------------------------- + {"выпил стакан воды", "water", "tp", false, ""}, + {"попил воды", "water", "tp", false, ""}, + {"я попил водички", "water", "tp", false, ""}, + {"пью воду", "water", "tp", false, ""}, + {"воду пил уже", "water", "tp", false, ""}, + {"допил воду", "water", "tp", true, "REGRESSION: the dictionary lemmatises допил to допилить, to finish sawing — the same saw collision drink_verbs already carries пил for, unfixed for the prefixed form"}, + {"запил таблетку водой", "water", "tp", false, ""}, + {"drank water", "water", "tp", false, ""}, + {"i drank some water", "water", "tp", false, ""}, + + // ---- meal, true positives -------------------------------------------- + {"поужинал", "meal", "tp", false, ""}, + {"я пообедал", "meal", "tp", false, ""}, + {"позавтракал кашей", "meal", "tp", false, ""}, + {"перекусил бутербродом", "meal", "tp", false, ""}, + {"покушал", "meal", "tp", false, ""}, + {"поел супа", "meal", "tp", false, ""}, + {"обед был в час", "meal", "tp", false, ""}, + {"ужинать буду позже", "meal", "tp", false, ""}, + {"i ate", "meal", "tp", false, ""}, + {"had lunch", "meal", "tp", false, ""}, + {"dinner done", "meal", "tp", false, ""}, + + // ---- shower, true positives ------------------------------------------ + {"принял душ", "shower", "tp", false, ""}, + {"сходил в душ", "shower", "tp", false, ""}, + {"душ принят", "shower", "tp", false, ""}, + {"ополоснулся душем", "shower", "tp", false, ""}, + {"took a shower", "shower", "tp", false, ""}, + {"i showered", "shower", "tp", false, ""}, + + // ---- break, true positives ------------------------------------------- + {"сделал перерыв", "break", "tp", false, ""}, + {"отдохнул полчаса", "break", "tp", false, ""}, + {"передохнул немного", "break", "tp", false, ""}, + {"отдыхаю", "break", "tp", false, ""}, + {"был перерыв на обед", "meal", "tp", false, "meal wins the switch; both keys are true of the sentence"}, + {"took a break", "break", "tp", false, ""}, + + // ---- sleep, true positives ------------------------------------------- + {"спал восемь часов", "sleep", "tp", false, ""}, + {"спала плохо", "sleep", "tp", false, "she may report her own; the parser is speaker-agnostic"}, + {"поспал днём", "sleep", "tp", false, ""}, + {"выспался наконец", "sleep", "tp", false, ""}, + {"проспал будильник", "sleep", "tp", false, ""}, + {"пойду спать", "sleep", "tp", false, ""}, + {"slept 8 hours", "sleep", "tp", false, ""}, + {"i slept badly", "sleep", "tp", false, ""}, + + // ---- misfires 22edc3c set out to reject ------------------------------- + {"пилот сказал что вылет через час", "", "misfire", false, "substring пил"}, + {"водитель уже подъехал", "", "misfire", false, "substring вод"}, + {"надо заводить машину", "", "misfire", false, "substring вод"}, + {"душа болит", "", "misfire", false, "substring душ"}, + {"в комнате душно", "", "misfire", false, "substring душ"}, + {"это была беда", "", "misfire", false, "substring еда"}, + {"наша победа", "", "misfire", false, "substring еда"}, + {"пила лежит в гараже", "", "misfire", false, "substring пил"}, + {"водитель пилота ждёт", "", "misfire", false, "both stems in one sentence — the old water arm fires"}, + {"обеденный перерыв отменили", "", "misfire", true, "neither parser gets this: the old one writes meal off the adjective, the new one writes break off перерыв. A cancelled break is not a break taken."}, + + // ---- hard negatives that decide the design ---------------------------- + {"есть новости по бэкапу базы", "", "misfire", false, "есть is the existential, not a meal"}, + {"напоминания на завтра есть", "", "misfire", false, "завтра carries завтрак as a substring"}, + {"на душе легко", "", "misfire", false, "душе is the soul, and also the prepositional of душ"}, + {"пилил доску весь вечер", "", "misfire", false, ""}, + {"поставь будильник на завтра", "", "misfire", false, "завтра again, no meal"}, + + // ---- false negatives: words in no lexicon set ------------------------- + // Water. + {"выпил чаю", "water", "fn", false, "hydration by any liquid; чай is in no set"}, + {"глотнул воды", "water", "fn", false, "глотнуть is not a drink verb"}, + {"хлебнул воды", "water", "fn", false, "хлебнуть is not a drink verb"}, + {"воды хлебнул из бутылки", "water", "fn", false, ""}, + {"выпил стакан", "water", "fn", false, "the noun is elided; he says this"}, + {"i hydrated", "water", "fn", false, "hydrate is in no set"}, + {"finished my bottle of water", "water", "fn", false, "no drink verb in the English set"}, + + // Meal. + {"ем суп", "meal", "fn", false, "есть is deliberately absent, and this is the cost"}, + {"съел бутерброд", "meal", "fn", false, "съесть is in no set"}, + {"наелся", "meal", "fn", false, "наесться is in no set"}, + {"пожрал", "meal", "fn", false, "coarse but spoken"}, + {"полдник был", "meal", "fn", false, "полдник is in no set"}, + {"i had a snack", "meal", "fn", false, "snack is in no set"}, + {"having supper", "meal", "fn", false, "supper is in no set"}, + {"brunch was good", "meal", "fn", false, "brunch is in no set"}, + {"i eat now", "meal", "fn", false, "eat is in no set — only ate is"}, + + // Shower. + {"был в душе", "shower", "fn", false, "prepositional; anyExact cannot take it without taking the soul"}, + {"после душа полегчало", "shower", "fn", false, "genitive, same collision"}, + {"помылся", "shower", "fn", false, "помыться is in no set"}, + {"сходил в ванную", "shower", "fn", false, "ванная is in no set"}, + {"искупался", "shower", "fn", false, "искупаться is in no set"}, + {"i am showering", "shower", "fn", false, "showering is not a member and the set is exact-matched"}, + + // Break. + {"сделал передышку", "break", "fn", false, "передышка is in no set"}, + {"перекур", "break", "fn", false, "перекур is in no set"}, + {"полежал немного", "break", "fn", false, "полежать is in no set"}, + {"сделал паузу", "break", "fn", false, "пауза is in no set"}, + {"i took five", "break", "fn", false, "idiomatic, in no set"}, + {"resting now", "break", "fn", false, "resting is not a member and rest is matched by lemma only"}, + + // Sleep. + {"вздремнул", "sleep", "fn", false, "вздремнуть is in no set"}, + {"прикорнул на диване", "sleep", "fn", false, "прикорнуть is in no set"}, + {"дрых до обеда", "sleep", "fn", false, "дрыхнуть is in no set — and обед makes this a MEAL for both parsers"}, + {"недоспал", "sleep", "fn", false, "недоспать is in no set"}, + {"лёг в двенадцать", "sleep", "fn", false, "лечь is in no set"}, + {"сон был короткий", "sleep", "fn", false, "сон deliberately absent"}, + {"i napped", "sleep", "fn", false, "nap is in no set"}, + {"took a nap", "sleep", "fn", false, "break matches nothing here either"}, +} + +// legacyFactParse — DefaultFactParser exactly as it stood at 22edc3c's parent +// (0445693), copied here so the corpus scores the trade rather than describing +// it. Do not fix it. It is a frozen baseline, and the day it stops being worth +// comparing against, delete it and the two-column table with it. +func legacyFactParse(utterance string) (string, string, bool) { + s := strings.ToLower(strings.TrimSpace(utterance)) + hasRoot := func(root string) bool { return strings.Contains(s, root) } + containsWord := func(w string) bool { + for _, tok := range strings.Fields(s) { + if tok == w { + return true + } + } + return false + } + switch { + case (containsWord("water") && containsWord("drank")) || + (hasRoot("вод") && (hasRoot("пил") || hasRoot("пью") || hasRoot("пей"))): + return "water", `"drank"`, true + case containsWord("meal") || (containsWord("ate") && !containsWord("backup")) || containsWord("lunch") || containsWord("dinner") || + hasRoot("поел") || hasRoot("поесть") || hasRoot("куша") || hasRoot("обед") || hasRoot("ужин") || hasRoot("завтрак") || hasRoot("еда"): + return "meal", `"ate"`, true + case containsWord("shower") || hasRoot("душ"): + return "shower", `"took"`, true + case containsWord("break") || hasRoot("перерыв") || hasRoot("отдох"): + return "break", `"took"`, true + case containsWord("slept") || containsWord("sleep") || + hasRoot("спал") || hasRoot("выспал"): + if v, ok := parseDurationValue(afterWord(s, "slept")); ok { + return "sleep", strconv.Quote(v), true + } + return "sleep", `"slept"`, true + } + return "", "", false +} + +// TestFactParserCorpus scores both parsers over the corpus and prints the +// side-by-side. Run it with -v; the table is the output. +func TestFactParserCorpus(t *testing.T) { + type tally struct{ tpHit, tpMiss, misfire, misfireOK, fnHit, fnMiss int } + var now, old tally + + var rows []string + rows = append(rows, "| utterance | class | want | old (substring) | new (closed class) |") + rows = append(rows, "|---|---|---|---|---|") + + score := func(key string, ok bool, c factCase, tl *tally) string { + got := "" + if ok { + got = key + } + switch c.class { + case "tp": + if got == c.want { + tl.tpHit++ + } else { + tl.tpMiss++ + } + case "misfire": + if got == c.want { + tl.misfireOK++ + } else { + tl.misfire++ + } + case "fn": + if got == c.want { + tl.fnHit++ + } else { + tl.fnMiss++ + } + } + if got == "" { + return "—" + } + return got + } + + for _, c := range factCorpus { + nk, _, nok := DefaultFactParser{}.Parse(c.utterance) + ok, _, ook := legacyFactParse(c.utterance) + gotNew := score(nk, nok, c, &now) + gotOld := score(ok, ook, c, &old) + want := c.want + if want == "" { + want = "—" + } + mark := "" + if gotOld != gotNew { + mark = " **≠**" + } + rows = append(rows, "| `"+c.utterance+"` | "+c.class+" | "+want+" | "+gotOld+" | "+gotNew+mark+" |") + } + + line := func(name string, tl tally) string { + return name + + ": true positives " + strconv.Itoa(tl.tpHit) + "/" + strconv.Itoa(tl.tpHit+tl.tpMiss) + + ", misfires rejected " + strconv.Itoa(tl.misfireOK) + "/" + strconv.Itoa(tl.misfireOK+tl.misfire) + + ", false-negative cases recovered " + strconv.Itoa(tl.fnHit) + "/" + strconv.Itoa(tl.fnHit+tl.fnMiss) + } + t.Log("\n" + strings.Join(rows, "\n") + "\n\n" + line("old (substring)", old) + "\n" + line("new (closed class)", now)) + + // The two regressions that are not judgement calls. + for _, c := range factCorpus { + k, _, ok := DefaultFactParser{}.Parse(c.utterance) + got := "" + if ok { + got = k + } + if c.class == "fn" { + continue + } + if c.broken { + // Recorded as wrong on 06-08. If it starts passing, someone fixed + // it and the flag is now a lie — say so rather than staying green. + if got == c.want { + t.Errorf("%q now returns %q as wanted — drop its broken flag", c.utterance, got) + } + continue + } + if got != c.want { + t.Errorf("%s %q: got %q, want %q (%s)", c.class, c.utterance, got, c.want, c.note) + } + } +}