Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 claude
|
4f6dec0cf2 | ||
|
claude
|
23ad5c0247 |
@@ -139,9 +139,9 @@ func (h *reactiveHandler) handlePraxisAct(ctx context.Context, dec router.Decisi
|
||||
// praxisItemAction is the shared shape of the item-lifecycle capabilities: take
|
||||
// an item id from the value slot, call one Praxis endpoint, trace the result.
|
||||
type praxisItemAction struct {
|
||||
verbs []string
|
||||
ask string // reply when no item id was given
|
||||
op string // trace + log name of the operation
|
||||
verbs []string
|
||||
ask string // reply when no item id was given
|
||||
op string // trace + log name of the operation
|
||||
// failure is the first half of the reply when the Praxis call errors: which
|
||||
// operation did not happen. ecosystemGap supplies the second half, which
|
||||
// names Praxis and splits a refused token from an outage — those two used to
|
||||
|
||||
@@ -1,247 +0,0 @@
|
||||
# The fact parser: closed classes against the substring stems they replaced
|
||||
|
||||
Measured 2026-08-06 at 22edc3c and its parent 0445693, on the corpus in
|
||||
`internal/router/factparser_corpus_test.go`. Dated file: it is not edited after
|
||||
today, and a newer number is a new file.
|
||||
|
||||
V-586 rewrote `DefaultFactParser` off hand-written Russian stems onto six closed
|
||||
classes in `internal/lexicon`. Its commit message reported 64/91 on the RU
|
||||
routing fixture, unchanged. That number does not bear on the change: the fixture
|
||||
holds three fact cases and all three miss on intent, so the parser is never
|
||||
reached. This file measures the parser directly, and runs the LLM arm the
|
||||
original commit skipped.
|
||||
|
||||
**The rewrite is better on the utterances it was designed for and no worse on
|
||||
the ones it was not.** True positives go 35/40 to 39/40, misfires rejected go
|
||||
8/15 to 14/15. What neither version has is coverage: of 36 plausible utterances
|
||||
whose word is in no lexicon set, the substring parser caught 3 by accident and
|
||||
the closed-class parser catches 0. That is the honest headline. The word list
|
||||
did not shrink the vocabulary — it never had one — it made the boundary visible.
|
||||
|
||||
## The corpus
|
||||
|
||||
91 cases, three classes. **True positives** are sentences the owner would say,
|
||||
with the key that must be written. **Misfires** are sentences the substring
|
||||
parser wrote a fact for and should not have, including the three hard negatives
|
||||
the rewrite was argued on. **False negatives** are sentences a reasonable person
|
||||
would say whose word is in no set at all; `want` is the key a human would
|
||||
assign, and the ship parser is expected to miss them. They are the measurement
|
||||
of what a closed class costs, not a bug list.
|
||||
|
||||
The old parser is carried in the test file as `legacyFactParse`, copied verbatim
|
||||
from 0445693, so the comparison reruns:
|
||||
|
||||
```sh
|
||||
deps/go/go/bin/go test -run TestFactParserCorpus -v ./internal/router/
|
||||
```
|
||||
|
||||
## Score
|
||||
|
||||
| | true positives | misfires rejected | false-negative cases recovered |
|
||||
|---|---|---|---|
|
||||
| old (substring stems, 0445693) | 35/40 | 8/15 | 3/36 |
|
||||
| **new (closed classes, 22edc3c)** | **39/40** | **14/15** | **0/36** |
|
||||
|
||||
Sixteen cases disagree. Eleven of them the rewrite wins, three it loses, and two
|
||||
are cases neither gets.
|
||||
|
||||
**Won.** Every misfire the commit message named — `душа болит`, `в комнате
|
||||
душно`, `это была беда`, `наша победа`, `на душе легко` — plus `водитель пилота
|
||||
ждёт`, where two stems in one sentence made the old water arm fire. And five true
|
||||
positives the stems simply did not list: `перекусил`, `передохнул`, `отдыхаю`,
|
||||
`пойду спать`, `i showered`. Morphology buys those; a stem list would need a new
|
||||
entry for each.
|
||||
|
||||
**Lost.** `допил воду` is a real regression and the only failing true positive.
|
||||
The vendored dictionary lemmatises `допил` to `допилить`, to finish sawing —
|
||||
exactly the collision `drink_verbs` already carries `пил` and `пили` as surface
|
||||
forms to dodge, left unhandled for the prefixed form. `допить` is in the set and
|
||||
the sentence still misses. It is flagged `broken` in the corpus rather than
|
||||
fixed, because this branch measures.
|
||||
|
||||
`был в душе` and `после душа полегчало` are the price of matching the shower set
|
||||
exactly. The dictionary makes `душ` and `душа` one word, so a lemma test cannot
|
||||
tell a shower from a soul; exact matching keeps `на душе легко` out and loses
|
||||
the oblique cases of the real noun with it. The old parser got both by accident,
|
||||
along with the soul. That trade is right — writing a shower fact when he said
|
||||
his soul feels light is worse than missing one — but it is a trade and the two
|
||||
rows are what it costs.
|
||||
|
||||
`недоспал` is the third loss and the least defensible: the old substring `спал`
|
||||
caught it, and `недоспать` is in no set.
|
||||
|
||||
**Neither.** `обеденный перерыв отменили` — a cancelled lunch break — is a fact
|
||||
for both parsers, `meal` for the old one off the adjective and `break` for the
|
||||
new one off `перерыв`. Nothing in either design reads the cancellation.
|
||||
`дрых до обеда` is scored `meal` by both, because the meal arm runs first and
|
||||
`обеда` is in it, which is not wrong so much as beside the point.
|
||||
|
||||
## The false-negative surface
|
||||
|
||||
This is the half the routing fixture cannot see and the half that decides
|
||||
whether the design holds. 36 cases, 0 recovered:
|
||||
|
||||
- **water** — `выпил чаю`, `глотнул воды`, `хлебнул воды`, `выпил стакан`,
|
||||
`i hydrated`, `finished my bottle of water`. The water arm needs a noun AND a
|
||||
verb, so an elided noun or an unlisted verb drops the whole capture.
|
||||
- **meal** — `ем суп`, `съел бутерброд`, `наелся`, `пожрал`, `полдник был`,
|
||||
`snack`, `supper`, `brunch`, `i eat now`. `есть` is deliberately absent for
|
||||
`есть новости по бэкапу`, and `ем`, its most ordinary spoken form, goes with it.
|
||||
- **shower** — `помылся`, `сходил в ванную`, `искупался`, `i am showering`,
|
||||
plus the two oblique cases above.
|
||||
- **break** — `сделал передышку`, `перекур`, `полежал немного`, `сделал паузу`,
|
||||
`i took five`, `resting now`.
|
||||
- **sleep** — `вздремнул`, `прикорнул`, `дрых`, `недоспал`, `лёг в двенадцать`,
|
||||
`сон был короткий`, `i napped`, `took a nap`.
|
||||
|
||||
None of these are exotic. They are the second and third word a person reaches
|
||||
for, and every one of them is a fact the owner stated and Maven silently did not
|
||||
record. A silent miss is the worst failure mode this parser has: he said it, she
|
||||
heard it, nothing was written, and nothing told him.
|
||||
|
||||
## The routing fixture, LLM arm
|
||||
|
||||
The arm 22edc3c skipped. `MAVEN_LLM_URL` points the harness at any llama-server;
|
||||
the previous run reported none reachable, which was the shell's `HTTP_PROXY` and
|
||||
not the network. Run against **gemma-4-12B-it-qat-UD-Q4_K_XL on the workstation
|
||||
at `192.168.1.105:8080`**, the same box as the 02-08 measurement, with
|
||||
`NO_PROXY=192.168.1.105`:
|
||||
|
||||
```sh
|
||||
NO_PROXY=192.168.1.105 no_proxy=192.168.1.105 \
|
||||
make eval-models MAVEN_LLM_URL=http://192.168.1.105:8080
|
||||
```
|
||||
|
||||
| | full | intent-only | p50 |
|
||||
|---|---|---|---|
|
||||
| llm-only, 0445693 | 51.6% (47/91) | 82.4% | — |
|
||||
| llm-only, 22edc3c | 52.7% (48/91) | 83.5% | 341ms |
|
||||
| cascade+llm, 0445693 | 85.7% (78/91) | 93.4% | — |
|
||||
| **cascade+llm, 22edc3c** | **86.8% (79/91)** | **94.5%** | 334ms |
|
||||
|
||||
One case either way, both directions, and the failing set is identical between
|
||||
the two commits. That is run-to-run variance on a sampling model, not a signal.
|
||||
The parser change is invisible to the routing fixture on the LLM arm for the
|
||||
same reason it is invisible on the classifier arm: the three fact cases miss on
|
||||
intent and the parser is never called. Do not read these rows as evidence about
|
||||
the parser. They are evidence that the fixture cannot answer the question, which
|
||||
is why the corpus above exists.
|
||||
|
||||
## Verdict
|
||||
|
||||
The closed-class rewrite holds up as a rewrite. It is strictly better than what
|
||||
it replaced on both classes anyone argued about, and the one regression
|
||||
(`допил`) and one bad trade (the oblique `душ`) are both dictionary collisions
|
||||
rather than design faults.
|
||||
|
||||
It does not hold up as an answer. A closed class is the right mechanism for a
|
||||
set that is actually closed — interrogatives, weekdays, cardinals — and
|
||||
"the words a person uses to say he ate" is not that set. The corpus puts a
|
||||
number on it: 36 ordinary sentences, 0 recovered, and every new one costs a
|
||||
lexicon edit by whoever notices. The three mechanisms CLAUDE.md names do not
|
||||
contain the right one for this job. The embedder-topic mechanism is the closest
|
||||
fit and is wrong too, because this is slot extraction rather than aboutness.
|
||||
|
||||
This is a case for the V-546 slot-tagging head. Self-care facts are a bounded
|
||||
key space (five keys) over unbounded surface forms, which is exactly what a BIO
|
||||
tagger on e5-small is for: it generalises to `вздремнул` without anyone adding
|
||||
`вздремнуть` to a list, and max softmax gives the confidence the parser's
|
||||
hardcoded `true` does not have. Until it lands, the closed classes are the
|
||||
correct floor and the 36 rows above are the size of the gap they leave.
|
||||
|
||||
## The corpus, case by case
|
||||
|
||||
| utterance | class | want | old (substring) | new (closed class) |
|
||||
|---|---|---|---|---|
|
||||
| `выпил стакан воды` | tp | water | water | water |
|
||||
| `попил воды` | tp | water | water | water |
|
||||
| `я попил водички` | tp | water | water | water |
|
||||
| `пью воду` | tp | water | water | water |
|
||||
| `воду пил уже` | tp | water | water | water |
|
||||
| `допил воду` | tp | water | water | — **≠** |
|
||||
| `запил таблетку водой` | tp | water | water | water |
|
||||
| `drank water` | tp | water | water | water |
|
||||
| `i drank some water` | tp | water | water | water |
|
||||
| `поужинал` | tp | meal | meal | meal |
|
||||
| `я пообедал` | tp | meal | meal | meal |
|
||||
| `позавтракал кашей` | tp | meal | meal | meal |
|
||||
| `перекусил бутербродом` | tp | meal | — | meal **≠** |
|
||||
| `покушал` | tp | meal | meal | meal |
|
||||
| `поел супа` | tp | meal | meal | meal |
|
||||
| `обед был в час` | tp | meal | meal | meal |
|
||||
| `ужинать буду позже` | tp | meal | meal | meal |
|
||||
| `i ate` | tp | meal | meal | meal |
|
||||
| `had lunch` | tp | meal | meal | meal |
|
||||
| `dinner done` | tp | meal | meal | meal |
|
||||
| `принял душ` | tp | shower | shower | shower |
|
||||
| `сходил в душ` | tp | shower | shower | shower |
|
||||
| `душ принят` | tp | shower | shower | shower |
|
||||
| `ополоснулся душем` | tp | shower | shower | shower |
|
||||
| `took a shower` | tp | shower | shower | shower |
|
||||
| `i showered` | tp | shower | — | shower **≠** |
|
||||
| `сделал перерыв` | tp | break | break | break |
|
||||
| `отдохнул полчаса` | tp | break | break | break |
|
||||
| `передохнул немного` | tp | break | — | break **≠** |
|
||||
| `отдыхаю` | tp | break | — | break **≠** |
|
||||
| `был перерыв на обед` | tp | meal | meal | meal |
|
||||
| `took a break` | tp | break | break | break |
|
||||
| `спал восемь часов` | tp | sleep | sleep | sleep |
|
||||
| `спала плохо` | tp | sleep | sleep | sleep |
|
||||
| `поспал днём` | tp | sleep | sleep | sleep |
|
||||
| `выспался наконец` | tp | sleep | sleep | sleep |
|
||||
| `проспал будильник` | tp | sleep | sleep | sleep |
|
||||
| `пойду спать` | tp | sleep | — | sleep **≠** |
|
||||
| `slept 8 hours` | tp | sleep | sleep | sleep |
|
||||
| `i slept badly` | tp | sleep | sleep | sleep |
|
||||
| `пилот сказал что вылет через час` | misfire | — | — | — |
|
||||
| `водитель уже подъехал` | misfire | — | — | — |
|
||||
| `надо заводить машину` | misfire | — | — | — |
|
||||
| `душа болит` | misfire | — | shower | — **≠** |
|
||||
| `в комнате душно` | misfire | — | shower | — **≠** |
|
||||
| `это была беда` | misfire | — | meal | — **≠** |
|
||||
| `наша победа` | misfire | — | meal | — **≠** |
|
||||
| `пила лежит в гараже` | misfire | — | — | — |
|
||||
| `водитель пилота ждёт` | misfire | — | water | — **≠** |
|
||||
| `обеденный перерыв отменили` | misfire | — | meal | break **≠** |
|
||||
| `есть новости по бэкапу базы` | misfire | — | — | — |
|
||||
| `напоминания на завтра есть` | misfire | — | — | — |
|
||||
| `на душе легко` | misfire | — | shower | — **≠** |
|
||||
| `пилил доску весь вечер` | misfire | — | — | — |
|
||||
| `поставь будильник на завтра` | misfire | — | — | — |
|
||||
| `выпил чаю` | fn | water | — | — |
|
||||
| `глотнул воды` | fn | water | — | — |
|
||||
| `хлебнул воды` | fn | water | — | — |
|
||||
| `воды хлебнул из бутылки` | fn | water | — | — |
|
||||
| `выпил стакан` | fn | water | — | — |
|
||||
| `i hydrated` | fn | water | — | — |
|
||||
| `finished my bottle of water` | fn | water | — | — |
|
||||
| `ем суп` | fn | meal | — | — |
|
||||
| `съел бутерброд` | fn | meal | — | — |
|
||||
| `наелся` | fn | meal | — | — |
|
||||
| `пожрал` | fn | meal | — | — |
|
||||
| `полдник был` | fn | meal | — | — |
|
||||
| `i had a snack` | fn | meal | — | — |
|
||||
| `having supper` | fn | meal | — | — |
|
||||
| `brunch was good` | fn | meal | — | — |
|
||||
| `i eat now` | fn | meal | — | — |
|
||||
| `был в душе` | fn | shower | shower | — **≠** |
|
||||
| `после душа полегчало` | fn | shower | shower | — **≠** |
|
||||
| `помылся` | fn | shower | — | — |
|
||||
| `сходил в ванную` | fn | shower | — | — |
|
||||
| `искупался` | fn | shower | — | — |
|
||||
| `i am showering` | fn | shower | — | — |
|
||||
| `сделал передышку` | fn | break | — | — |
|
||||
| `перекур` | fn | break | — | — |
|
||||
| `полежал немного` | fn | break | — | — |
|
||||
| `сделал паузу` | fn | break | — | — |
|
||||
| `i took five` | fn | break | — | — |
|
||||
| `resting now` | fn | break | — | — |
|
||||
| `вздремнул` | fn | sleep | — | — |
|
||||
| `прикорнул на диване` | fn | sleep | — | — |
|
||||
| `дрых до обеда` | fn | sleep | meal | meal |
|
||||
| `недоспал` | fn | sleep | sleep | — **≠** |
|
||||
| `лёг в двенадцать` | fn | sleep | — | — |
|
||||
| `сон был короткий` | fn | sleep | — | — |
|
||||
| `i napped` | fn | sleep | — | — |
|
||||
| `took a nap` | fn | sleep | — | — |
|
||||
`≠` marks a disagreement. `—` is no fact written.
|
||||
@@ -135,30 +135,6 @@ func ConfirmNo() []string { return words("confirm_no") }
|
||||
// TaskDropWords — see TaskDoneWords.
|
||||
func TaskDropWords() []string { return words("task_drop_words") }
|
||||
|
||||
// WaterNouns and DrinkVerbs are the two halves of a water fact: he has to name
|
||||
// the drink and the drinking, because "вода" alone is a word about water and
|
||||
// "выпил" alone does not say what. The other four self-care sets need only one
|
||||
// word each. All six are matched over tokens by lemma — except ShowerWords, see
|
||||
// its own note.
|
||||
func WaterNouns() []string { return words("water_nouns") }
|
||||
|
||||
// DrinkVerbs — see WaterNouns.
|
||||
func DrinkVerbs() []string { return words("drink_verbs") }
|
||||
|
||||
// MealWords returns the nouns and verbs of having eaten.
|
||||
func MealWords() []string { return words("meal_words") }
|
||||
|
||||
// ShowerWords returns the shower noun. Match these EXACTLY and not by lemma:
|
||||
// the dictionary makes "душ" and "душа" one word, and only one of them is a
|
||||
// shower. The set's note says why exact matching costs nothing here.
|
||||
func ShowerWords() []string { return words("shower_words") }
|
||||
|
||||
// BreakWords returns the noun and verbs of taking a break.
|
||||
func BreakWords() []string { return words("break_words") }
|
||||
|
||||
// SleepWords returns the verbs of having slept.
|
||||
func SleepWords() []string { return words("sleep_words") }
|
||||
|
||||
// SlotValueFrame returns the words that can surround a bare slot value without
|
||||
// making the utterance a request of its own. A caller strips these (along with
|
||||
// the numbers and the other closed time sets) to see whether an utterance
|
||||
|
||||
@@ -252,50 +252,6 @@
|
||||
"yes", "yeah", "yep", "yup", "ok", "okay", "sure", "confirm", "affirmative"
|
||||
]
|
||||
},
|
||||
"water_nouns": {
|
||||
"note": "The water noun, in the forms he drinks it in, plus English (V-586). Closed because it is one noun: Russian gives it six cases and two numbers and that is the whole list. Both \"вода\" and \"водой\" are listed even though one declension covers both, because the vendored dictionary lemmatises \"воды\" to \"вод\" and \"водой\" to \"вода\" — two lemmas for one noun, so the set has to name both or a caller matching by lemma misses half of them. Matched over tokens with morph.SameWord, never as a substring: the \"вод\" this replaced fired on \"водитель\" and \"заводить\".",
|
||||
"words": [
|
||||
"вода", "водой", "водичка", "water"
|
||||
]
|
||||
},
|
||||
"drink_verbs": {
|
||||
"note": "Drinking, in the aspects and prefixes he speaks (V-586). Closed in the sense that matters: these are the verbs that make a water noun a water FACT, and the list is her vocabulary rather than a discovery about Russian. \"пил\" and \"пили\" are listed as surface forms because the dictionary lemmatises them to \"пила\", the saw; the perfective forms lemmatise correctly and one member each covers them. Whole tokens only — the substring \"пил\" this replaced fired on \"пилот\".",
|
||||
"words": [
|
||||
"пить", "пил", "пили", "пей", "выпить", "попить", "допить", "запить",
|
||||
"drink", "drank", "drinking"
|
||||
]
|
||||
},
|
||||
"meal_words": {
|
||||
"note": "Eating: the meal nouns and the verbs of having one (V-586). Closed the same way capture_verbs is — these are the words that write a meal fact, decided here. The verbs are listed in the infinitive because that is the lemma the dictionary returns, so \"поужинал\" and \"позавтракал\" match without their own entries. \"есть\" and \"ел\" are deliberately ABSENT: \"есть\" is also the existential, and \"есть новости по бэкапу\" is a question rather than a meal. The English \"ate\" carried a guard against \"backup\" when this was a substring test; over tokens the guard is unnecessary.",
|
||||
"words": [
|
||||
"обед", "обедать", "пообедать",
|
||||
"ужин", "ужинать", "поужинать",
|
||||
"завтрак", "завтракать", "позавтракать",
|
||||
"еда", "перекус", "перекусить",
|
||||
"поесть", "кушать", "покушать",
|
||||
"meal", "ate", "lunch", "dinner", "breakfast"
|
||||
]
|
||||
},
|
||||
"shower_words": {
|
||||
"note": "The shower, and the one set here matched EXACTLY rather than by lemma (V-586). The dictionary lemmatises \"душ\" to \"душа\", so a lemma test cannot tell a shower from a soul, and \"на душе легко\" is not a fact about washing. The accusative of an inanimate noun is its nominative, so \"принял душ\" and \"сходил в душ\" are both the bare form and exact matching loses nothing he actually says. The substring this replaced also fired on \"душно\".",
|
||||
"words": [
|
||||
"душ", "душем", "shower", "showered"
|
||||
]
|
||||
},
|
||||
"break_words": {
|
||||
"note": "Taking a break, noun and verb (V-586). Closed like meal_words and for the same reason. \"отдых\" and \"отдыхать\" are both listed because the noun and the verb are separate lemmas; the perfective \"отдохнул\" lemmatises to \"отдохнуть\".",
|
||||
"words": [
|
||||
"перерыв", "отдых", "отдыхать", "отдохнуть", "передохнуть",
|
||||
"break", "rest"
|
||||
]
|
||||
},
|
||||
"sleep_words": {
|
||||
"note": "Sleeping (V-586). The imperfective surface forms \"спал\" and \"спала\" are listed because the dictionary lemmatises them to \"спасть\", a different verb, and one entry for the pair is the honest fix; the prefixed forms lemmatise consistently and their infinitives cover them. \"сон\" is absent: the noun names a dream as readily as a night's sleep, and it was not in the pattern this replaces either.",
|
||||
"words": [
|
||||
"спать", "спал", "поспать", "поспал", "проспать", "выспаться",
|
||||
"sleep", "slept", "sleeping"
|
||||
]
|
||||
},
|
||||
"confirm_no": {
|
||||
"note": "The answers that decline a parked confirm. Same matching rule as confirm_yes and the same reason. The multi-word members are here rather than assembled by a caller because \"надо\" alone is not an answer and \"не надо\" is the opposite of one: the two must land on opposite sides, and only the phrase says which. \"не\" on its own is NOT a member — \"не забудь купить хлеб\" is a reminder, not a refusal.",
|
||||
"words": [
|
||||
|
||||
@@ -662,7 +662,7 @@ func (t *emptyFrameTransport) Call(ctx context.Context, req *rpcRequest) (*rpcRe
|
||||
}
|
||||
|
||||
func (t *emptyFrameTransport) Notify(context.Context, string, any) error { return nil }
|
||||
func (t *emptyFrameTransport) Close() error { return nil }
|
||||
func (t *emptyFrameTransport) Close() error { return nil }
|
||||
|
||||
func TestResultlessResponseIsNotSuccess(t *testing.T) {
|
||||
c := newClient("empty", &emptyFrameTransport{})
|
||||
|
||||
@@ -1,286 +0,0 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The fact-parser corpus (V-586). DefaultFactParser is the only thing standing
|
||||
// between a spoken sentence and a written self-care fact, and until this file
|
||||
// it was covered by a handful of examples chosen by whoever last edited it. The
|
||||
// RU routing fixture does not cover it either: it holds three fact cases and
|
||||
// all three miss on intent, so the parser is never reached and a parser change
|
||||
// scores as "unchanged".
|
||||
//
|
||||
// So the corpus is here, and it scores BOTH implementations: the closed-class
|
||||
// parser that ships, and legacyFactParse below, a faithful copy of the
|
||||
// hand-written substring version 22edc3c replaced. The table is the measurement
|
||||
// — see docs/evals/2026-08-06-fact-parser.md for the numbers as of that day.
|
||||
//
|
||||
// Three case classes, and the third is the point:
|
||||
//
|
||||
// true positive — a sentence he would say, with the key it must write.
|
||||
// misfire — a sentence the substring parser wrote a fact for and
|
||||
// should not have. want is "".
|
||||
// false-negative — a sentence he would plausibly say whose word is in NO
|
||||
// lexicon set. want is the key a human would assign, and
|
||||
// the ship parser is EXPECTED to miss it. These measure the
|
||||
// cost of the word-list design, not a bug in it.
|
||||
//
|
||||
// Nothing here asserts a score. A closed class is a decision about vocabulary
|
||||
// and the bar belongs in a dated eval, not in an assertion that turns every
|
||||
// vocabulary edit into a red test. What it does fail on is a regression in the
|
||||
// two classes that are not judgement calls: a true positive that stops matching
|
||||
// and a misfire that starts.
|
||||
|
||||
type factCase struct {
|
||||
utterance string
|
||||
want string // "" — no fact
|
||||
class string // "tp", "misfire", "fn"
|
||||
broken bool // ship parser is known to get this wrong; see the 06-08 eval
|
||||
note string
|
||||
}
|
||||
|
||||
var factCorpus = []factCase{
|
||||
// ---- water, true positives -------------------------------------------
|
||||
{"выпил стакан воды", "water", "tp", false, ""},
|
||||
{"попил воды", "water", "tp", false, ""},
|
||||
{"я попил водички", "water", "tp", false, ""},
|
||||
{"пью воду", "water", "tp", false, ""},
|
||||
{"воду пил уже", "water", "tp", false, ""},
|
||||
{"допил воду", "water", "tp", true, "REGRESSION: the dictionary lemmatises допил to допилить, to finish sawing — the same saw collision drink_verbs already carries пил for, unfixed for the prefixed form"},
|
||||
{"запил таблетку водой", "water", "tp", false, ""},
|
||||
{"drank water", "water", "tp", false, ""},
|
||||
{"i drank some water", "water", "tp", false, ""},
|
||||
|
||||
// ---- meal, true positives --------------------------------------------
|
||||
{"поужинал", "meal", "tp", false, ""},
|
||||
{"я пообедал", "meal", "tp", false, ""},
|
||||
{"позавтракал кашей", "meal", "tp", false, ""},
|
||||
{"перекусил бутербродом", "meal", "tp", false, ""},
|
||||
{"покушал", "meal", "tp", false, ""},
|
||||
{"поел супа", "meal", "tp", false, ""},
|
||||
{"обед был в час", "meal", "tp", false, ""},
|
||||
{"ужинать буду позже", "meal", "tp", false, ""},
|
||||
{"i ate", "meal", "tp", false, ""},
|
||||
{"had lunch", "meal", "tp", false, ""},
|
||||
{"dinner done", "meal", "tp", false, ""},
|
||||
|
||||
// ---- shower, true positives ------------------------------------------
|
||||
{"принял душ", "shower", "tp", false, ""},
|
||||
{"сходил в душ", "shower", "tp", false, ""},
|
||||
{"душ принят", "shower", "tp", false, ""},
|
||||
{"ополоснулся душем", "shower", "tp", false, ""},
|
||||
{"took a shower", "shower", "tp", false, ""},
|
||||
{"i showered", "shower", "tp", false, ""},
|
||||
|
||||
// ---- break, true positives -------------------------------------------
|
||||
{"сделал перерыв", "break", "tp", false, ""},
|
||||
{"отдохнул полчаса", "break", "tp", false, ""},
|
||||
{"передохнул немного", "break", "tp", false, ""},
|
||||
{"отдыхаю", "break", "tp", false, ""},
|
||||
{"был перерыв на обед", "meal", "tp", false, "meal wins the switch; both keys are true of the sentence"},
|
||||
{"took a break", "break", "tp", false, ""},
|
||||
|
||||
// ---- sleep, true positives -------------------------------------------
|
||||
{"спал восемь часов", "sleep", "tp", false, ""},
|
||||
{"спала плохо", "sleep", "tp", false, "she may report her own; the parser is speaker-agnostic"},
|
||||
{"поспал днём", "sleep", "tp", false, ""},
|
||||
{"выспался наконец", "sleep", "tp", false, ""},
|
||||
{"проспал будильник", "sleep", "tp", false, ""},
|
||||
{"пойду спать", "sleep", "tp", false, ""},
|
||||
{"slept 8 hours", "sleep", "tp", false, ""},
|
||||
{"i slept badly", "sleep", "tp", false, ""},
|
||||
|
||||
// ---- misfires 22edc3c set out to reject -------------------------------
|
||||
{"пилот сказал что вылет через час", "", "misfire", false, "substring пил"},
|
||||
{"водитель уже подъехал", "", "misfire", false, "substring вод"},
|
||||
{"надо заводить машину", "", "misfire", false, "substring вод"},
|
||||
{"душа болит", "", "misfire", false, "substring душ"},
|
||||
{"в комнате душно", "", "misfire", false, "substring душ"},
|
||||
{"это была беда", "", "misfire", false, "substring еда"},
|
||||
{"наша победа", "", "misfire", false, "substring еда"},
|
||||
{"пила лежит в гараже", "", "misfire", false, "substring пил"},
|
||||
{"водитель пилота ждёт", "", "misfire", false, "both stems in one sentence — the old water arm fires"},
|
||||
{"обеденный перерыв отменили", "", "misfire", true, "neither parser gets this: the old one writes meal off the adjective, the new one writes break off перерыв. A cancelled break is not a break taken."},
|
||||
|
||||
// ---- hard negatives that decide the design ----------------------------
|
||||
{"есть новости по бэкапу базы", "", "misfire", false, "есть is the existential, not a meal"},
|
||||
{"напоминания на завтра есть", "", "misfire", false, "завтра carries завтрак as a substring"},
|
||||
{"на душе легко", "", "misfire", false, "душе is the soul, and also the prepositional of душ"},
|
||||
{"пилил доску весь вечер", "", "misfire", false, ""},
|
||||
{"поставь будильник на завтра", "", "misfire", false, "завтра again, no meal"},
|
||||
|
||||
// ---- false negatives: words in no lexicon set -------------------------
|
||||
// Water.
|
||||
{"выпил чаю", "water", "fn", false, "hydration by any liquid; чай is in no set"},
|
||||
{"глотнул воды", "water", "fn", false, "глотнуть is not a drink verb"},
|
||||
{"хлебнул воды", "water", "fn", false, "хлебнуть is not a drink verb"},
|
||||
{"воды хлебнул из бутылки", "water", "fn", false, ""},
|
||||
{"выпил стакан", "water", "fn", false, "the noun is elided; he says this"},
|
||||
{"i hydrated", "water", "fn", false, "hydrate is in no set"},
|
||||
{"finished my bottle of water", "water", "fn", false, "no drink verb in the English set"},
|
||||
|
||||
// Meal.
|
||||
{"ем суп", "meal", "fn", false, "есть is deliberately absent, and this is the cost"},
|
||||
{"съел бутерброд", "meal", "fn", false, "съесть is in no set"},
|
||||
{"наелся", "meal", "fn", false, "наесться is in no set"},
|
||||
{"пожрал", "meal", "fn", false, "coarse but spoken"},
|
||||
{"полдник был", "meal", "fn", false, "полдник is in no set"},
|
||||
{"i had a snack", "meal", "fn", false, "snack is in no set"},
|
||||
{"having supper", "meal", "fn", false, "supper is in no set"},
|
||||
{"brunch was good", "meal", "fn", false, "brunch is in no set"},
|
||||
{"i eat now", "meal", "fn", false, "eat is in no set — only ate is"},
|
||||
|
||||
// Shower.
|
||||
{"был в душе", "shower", "fn", false, "prepositional; anyExact cannot take it without taking the soul"},
|
||||
{"после душа полегчало", "shower", "fn", false, "genitive, same collision"},
|
||||
{"помылся", "shower", "fn", false, "помыться is in no set"},
|
||||
{"сходил в ванную", "shower", "fn", false, "ванная is in no set"},
|
||||
{"искупался", "shower", "fn", false, "искупаться is in no set"},
|
||||
{"i am showering", "shower", "fn", false, "showering is not a member and the set is exact-matched"},
|
||||
|
||||
// Break.
|
||||
{"сделал передышку", "break", "fn", false, "передышка is in no set"},
|
||||
{"перекур", "break", "fn", false, "перекур is in no set"},
|
||||
{"полежал немного", "break", "fn", false, "полежать is in no set"},
|
||||
{"сделал паузу", "break", "fn", false, "пауза is in no set"},
|
||||
{"i took five", "break", "fn", false, "idiomatic, in no set"},
|
||||
{"resting now", "break", "fn", false, "resting is not a member and rest is matched by lemma only"},
|
||||
|
||||
// Sleep.
|
||||
{"вздремнул", "sleep", "fn", false, "вздремнуть is in no set"},
|
||||
{"прикорнул на диване", "sleep", "fn", false, "прикорнуть is in no set"},
|
||||
{"дрых до обеда", "sleep", "fn", false, "дрыхнуть is in no set — and обед makes this a MEAL for both parsers"},
|
||||
{"недоспал", "sleep", "fn", false, "недоспать is in no set"},
|
||||
{"лёг в двенадцать", "sleep", "fn", false, "лечь is in no set"},
|
||||
{"сон был короткий", "sleep", "fn", false, "сон deliberately absent"},
|
||||
{"i napped", "sleep", "fn", false, "nap is in no set"},
|
||||
{"took a nap", "sleep", "fn", false, "break matches nothing here either"},
|
||||
}
|
||||
|
||||
// legacyFactParse — DefaultFactParser exactly as it stood at 22edc3c's parent
|
||||
// (0445693), copied here so the corpus scores the trade rather than describing
|
||||
// it. Do not fix it. It is a frozen baseline, and the day it stops being worth
|
||||
// comparing against, delete it and the two-column table with it.
|
||||
func legacyFactParse(utterance string) (string, string, bool) {
|
||||
s := strings.ToLower(strings.TrimSpace(utterance))
|
||||
hasRoot := func(root string) bool { return strings.Contains(s, root) }
|
||||
containsWord := func(w string) bool {
|
||||
for _, tok := range strings.Fields(s) {
|
||||
if tok == w {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
switch {
|
||||
case (containsWord("water") && containsWord("drank")) ||
|
||||
(hasRoot("вод") && (hasRoot("пил") || hasRoot("пью") || hasRoot("пей"))):
|
||||
return "water", `"drank"`, true
|
||||
case containsWord("meal") || (containsWord("ate") && !containsWord("backup")) || containsWord("lunch") || containsWord("dinner") ||
|
||||
hasRoot("поел") || hasRoot("поесть") || hasRoot("куша") || hasRoot("обед") || hasRoot("ужин") || hasRoot("завтрак") || hasRoot("еда"):
|
||||
return "meal", `"ate"`, true
|
||||
case containsWord("shower") || hasRoot("душ"):
|
||||
return "shower", `"took"`, true
|
||||
case containsWord("break") || hasRoot("перерыв") || hasRoot("отдох"):
|
||||
return "break", `"took"`, true
|
||||
case containsWord("slept") || containsWord("sleep") ||
|
||||
hasRoot("спал") || hasRoot("выспал"):
|
||||
if v, ok := parseDurationValue(afterWord(s, "slept")); ok {
|
||||
return "sleep", strconv.Quote(v), true
|
||||
}
|
||||
return "sleep", `"slept"`, true
|
||||
}
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// TestFactParserCorpus scores both parsers over the corpus and prints the
|
||||
// side-by-side. Run it with -v; the table is the output.
|
||||
func TestFactParserCorpus(t *testing.T) {
|
||||
type tally struct{ tpHit, tpMiss, misfire, misfireOK, fnHit, fnMiss int }
|
||||
var now, old tally
|
||||
|
||||
var rows []string
|
||||
rows = append(rows, "| utterance | class | want | old (substring) | new (closed class) |")
|
||||
rows = append(rows, "|---|---|---|---|---|")
|
||||
|
||||
score := func(key string, ok bool, c factCase, tl *tally) string {
|
||||
got := ""
|
||||
if ok {
|
||||
got = key
|
||||
}
|
||||
switch c.class {
|
||||
case "tp":
|
||||
if got == c.want {
|
||||
tl.tpHit++
|
||||
} else {
|
||||
tl.tpMiss++
|
||||
}
|
||||
case "misfire":
|
||||
if got == c.want {
|
||||
tl.misfireOK++
|
||||
} else {
|
||||
tl.misfire++
|
||||
}
|
||||
case "fn":
|
||||
if got == c.want {
|
||||
tl.fnHit++
|
||||
} else {
|
||||
tl.fnMiss++
|
||||
}
|
||||
}
|
||||
if got == "" {
|
||||
return "—"
|
||||
}
|
||||
return got
|
||||
}
|
||||
|
||||
for _, c := range factCorpus {
|
||||
nk, _, nok := DefaultFactParser{}.Parse(c.utterance)
|
||||
ok, _, ook := legacyFactParse(c.utterance)
|
||||
gotNew := score(nk, nok, c, &now)
|
||||
gotOld := score(ok, ook, c, &old)
|
||||
want := c.want
|
||||
if want == "" {
|
||||
want = "—"
|
||||
}
|
||||
mark := ""
|
||||
if gotOld != gotNew {
|
||||
mark = " **≠**"
|
||||
}
|
||||
rows = append(rows, "| `"+c.utterance+"` | "+c.class+" | "+want+" | "+gotOld+" | "+gotNew+mark+" |")
|
||||
}
|
||||
|
||||
line := func(name string, tl tally) string {
|
||||
return name +
|
||||
": true positives " + strconv.Itoa(tl.tpHit) + "/" + strconv.Itoa(tl.tpHit+tl.tpMiss) +
|
||||
", misfires rejected " + strconv.Itoa(tl.misfireOK) + "/" + strconv.Itoa(tl.misfireOK+tl.misfire) +
|
||||
", false-negative cases recovered " + strconv.Itoa(tl.fnHit) + "/" + strconv.Itoa(tl.fnHit+tl.fnMiss)
|
||||
}
|
||||
t.Log("\n" + strings.Join(rows, "\n") + "\n\n" + line("old (substring)", old) + "\n" + line("new (closed class)", now))
|
||||
|
||||
// The two regressions that are not judgement calls.
|
||||
for _, c := range factCorpus {
|
||||
k, _, ok := DefaultFactParser{}.Parse(c.utterance)
|
||||
got := ""
|
||||
if ok {
|
||||
got = k
|
||||
}
|
||||
if c.class == "fn" {
|
||||
continue
|
||||
}
|
||||
if c.broken {
|
||||
// Recorded as wrong on 06-08. If it starts passing, someone fixed
|
||||
// it and the flag is now a lie — say so rather than staying green.
|
||||
if got == c.want {
|
||||
t.Errorf("%q now returns %q as wanted — drop its broken flag", c.utterance, got)
|
||||
}
|
||||
continue
|
||||
}
|
||||
if got != c.want {
|
||||
t.Errorf("%s %q: got %q, want %q (%s)", c.class, c.utterance, got, c.want, c.note)
|
||||
}
|
||||
}
|
||||
}
|
||||
+23
-36
@@ -7,7 +7,6 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/kami/maven/internal/lexicon"
|
||||
"github.com/kami/maven/internal/morph"
|
||||
)
|
||||
|
||||
// DateTimeParser — resolves relative→absolute AT CAPTURE ("in 4h" → now+4h),
|
||||
@@ -124,22 +123,23 @@ type DefaultFactParser struct{}
|
||||
|
||||
func (DefaultFactParser) Parse(utterance string) (string, string, bool) {
|
||||
s := strings.ToLower(strings.TrimSpace(utterance))
|
||||
toks := strings.Fields(s)
|
||||
// Maven is ru-first (voice, tts), and Russian inflects, so the words each
|
||||
// case reads are closed classes in internal/lexicon and the inflection is
|
||||
// morph's job (V-586). What stood here was a fourth mechanism: hand-written
|
||||
// stems matched as substrings, so "пилот" was drinking, "водитель" was
|
||||
// water, "душно" was a shower and "победа" was a meal.
|
||||
// Maven is ru-first (voice, tts). Each case carries the English tokens AND
|
||||
// Russian stems — matched by prefix (hasStem) because Russian inflects
|
||||
// (воды/воду/вода share "вод"), so exact-token matching would miss most
|
||||
// real utterances and silently drop the capture.
|
||||
switch {
|
||||
case anyLemma(toks, lexicon.WaterNouns()) && anyLemma(toks, lexicon.DrinkVerbs()):
|
||||
case (containsWord(s, "water") && containsWord(s, "drank")) ||
|
||||
(hasRoot(s, "вод") && (hasRoot(s, "пил") || hasRoot(s, "пью") || hasRoot(s, "пей"))):
|
||||
return "water", `"drank"`, true
|
||||
case anyLemma(toks, lexicon.MealWords()):
|
||||
case containsWord(s, "meal") || (containsWord(s, "ate") && !containsWord(s, "backup")) || containsWord(s, "lunch") || containsWord(s, "dinner") ||
|
||||
hasRoot(s, "поел") || hasRoot(s, "поесть") || hasRoot(s, "куша") || hasRoot(s, "обед") || hasRoot(s, "ужин") || hasRoot(s, "завтрак") || hasRoot(s, "еда"):
|
||||
return "meal", `"ate"`, true
|
||||
case anyExact(toks, lexicon.ShowerWords()):
|
||||
case containsWord(s, "shower") || hasRoot(s, "душ"):
|
||||
return "shower", `"took"`, true
|
||||
case anyLemma(toks, lexicon.BreakWords()):
|
||||
case containsWord(s, "break") || hasRoot(s, "перерыв") || hasRoot(s, "отдох"):
|
||||
return "break", `"took"`, true
|
||||
case anyLemma(toks, lexicon.SleepWords()):
|
||||
case containsWord(s, "slept") || containsWord(s, "sleep") ||
|
||||
hasRoot(s, "спал") || hasRoot(s, "выспал"):
|
||||
if v, ok := parseDurationValue(afterWord(s, "slept")); ok {
|
||||
return "sleep", strconv.Quote(v), true
|
||||
}
|
||||
@@ -148,31 +148,18 @@ func (DefaultFactParser) Parse(utterance string) (string, string, bool) {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
// anyLemma reports whether any token is one of the set's words, in any case or
|
||||
// tense. Exact equality first because morph falls back to it without the
|
||||
// dictionary, and because a member the dictionary lemmatises oddly is carried
|
||||
// in the set as its surface form.
|
||||
func anyLemma(toks []string, set []string) bool {
|
||||
for _, tok := range toks {
|
||||
t := cleanWord(tok)
|
||||
for _, w := range set {
|
||||
if t == w || morph.SameWord(t, w) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
// hasRoot — substring match on the whole utterance. Russian inflects with BOTH
|
||||
// prefixes and suffixes (вы-пил, по-пил, пил-и), so a prefix test misses the
|
||||
// verb; the root as a substring catches all forms. A rare over-match (пил in
|
||||
// пилот) is fine at this floor. ponytail: substring roots over a morphology lib
|
||||
// until misfires actually bite.
|
||||
func hasRoot(s, root string) bool { return strings.Contains(s, root) }
|
||||
|
||||
// anyExact is anyLemma without the grammar, for a set whose members share a
|
||||
// lemma with a word that means something else. Only ShowerWords needs it.
|
||||
func anyExact(toks []string, set []string) bool {
|
||||
for _, tok := range toks {
|
||||
t := cleanWord(tok)
|
||||
for _, w := range set {
|
||||
if t == w {
|
||||
return true
|
||||
}
|
||||
// containsWord — whole-token membership (avoids "breakfast" matching "break").
|
||||
func containsWord(s, w string) bool {
|
||||
for _, tok := range strings.Fields(s) {
|
||||
if tok == w {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
|
||||
@@ -23,24 +23,6 @@ func TestDefaultFactParserRU(t *testing.T) {
|
||||
{"немного отдохнул", "break", true},
|
||||
{"поспал шесть часов", "sleep", true},
|
||||
{"перезапусти nginx", "", false}, // an act, not a fact
|
||||
// Inflections the substring stems caught and a lemma test must keep.
|
||||
{"только что выпил кружку воды", "water", true},
|
||||
{"воды попил наконец", "water", true},
|
||||
{"пью воду", "water", true},
|
||||
{"поужинал", "meal", true},
|
||||
{"отметь что я позавтракал овсянкой", "meal", true},
|
||||
{"сходил в душ", "shower", true},
|
||||
{"отдыхал час", "break", true},
|
||||
{"выспался", "sleep", true},
|
||||
// Misfires the substring stems accepted. Rejecting them is the point of
|
||||
// V-586: none of these is a fact about his day.
|
||||
{"я пилот", "", false},
|
||||
{"водитель приехал", "", false},
|
||||
{"надо заводить машину", "", false},
|
||||
{"у меня душа болит", "", false},
|
||||
{"в комнате душно", "", false},
|
||||
{"это была беда", "", false},
|
||||
{"победа наша", "", false},
|
||||
}
|
||||
for _, c := range cases {
|
||||
k, _, ok := p.Parse(c.utterance)
|
||||
|
||||
Reference in New Issue
Block a user