Harden semantic boundaries and repair dialogue state
Replace nearest-neighbour personal routing with a frozen class-balanced linear head measured on historical, stratified, cross-validation, holdout, and fresh challenge gates (V-702). Close the four repair handoff holes, preserve nested clarification flows, and route Russian possession statements through structural grammar rather than lexical exceptions (V-573). Owner explicitly requested direct commits to master.
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
# A class-balanced linear head closes every held-out personal-boundary cell
|
||||
|
||||
Measured 2026-08-13 on homesrv from `master` at `56254a5` plus the V-702
|
||||
working tree. Embedder: quantized multilingual-e5-small through ONNX Runtime
|
||||
1.26.0. This supersedes
|
||||
`2026-08-13-personal-boundary-neighbourhood.md`.
|
||||
|
||||
Commands:
|
||||
|
||||
```sh
|
||||
MAVEN_ONNX_LIB="$PWD/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so.1.26.0" \
|
||||
go test ./cmd/mavend -run '^TestONNXPersonalBoundary$' -count=1 -v
|
||||
MAVEN_ONNX_LIB="$PWD/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so.1.26.0" \
|
||||
MAVEN_EVAL_PERSONAL_BOUNDARY=1 \
|
||||
go test ./cmd/mavend -run '^TestONNXPersonalBoundaryStratified$' -count=1 -v
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
The prior three-neighbour scorer passed the 29-case regression table but
|
||||
missed 11 of a new 72-case fixture. The fixture is balanced across Russian and
|
||||
English, expected personal and world classes, and six sentence shapes:
|
||||
remembered speech, possession, narrative, first-person preamble, current
|
||||
advice/information and public proper nouns. Its hygiene test rejects copied
|
||||
training phrases and repeated Baikal variants.
|
||||
|
||||
Production uses a binary logistic head fitted over 132 bilingual examples.
|
||||
Each class has total sample weight 0.5 even though the class sizes differ. The
|
||||
optimiser is deterministic: 5,000 full-batch steps, decaying learning rate from
|
||||
10, L2 coefficient 0.0003. The fitted 384 weights are frozen into the binary
|
||||
and pinned to the embedder model ID; a model-backed test retrains from the text
|
||||
corpus and rejects any drift. An unknown embedder ID falls back to fitting its
|
||||
own corpus rather than applying weights from another vector space.
|
||||
|
||||
No evaluation utterance is in the training corpus. Ordinary CI checks class
|
||||
balancing, mixed dimensions, exact regression-table leakage, and generalisation
|
||||
across synthetic semantic directions. The model-backed gate also runs
|
||||
four-fold cross-validation over the expanded corpus, stratified inside each
|
||||
sentence shape and class; the original compact corpus remains in every fold as
|
||||
the historical floor.
|
||||
|
||||
The six shapes of the training corpus were chosen after the first 72-case
|
||||
matrix exposed their gaps, so 72/72 is a regression gate, not independent proof
|
||||
of generalisation. The first independently authored 24-case challenge then
|
||||
exposed one remaining miss, `на каком порту я решил поднять тестовый сервис?`,
|
||||
at personal probability 0.4666. It became model-selection data at that point
|
||||
and cannot honestly remain the independent result.
|
||||
|
||||
A comparison rejected shrinkage LDA (24/24 challenge but 69/72 stratified) and
|
||||
an LDA/logistic blend (24/24 but 71/72). Increasing only the balanced logistic
|
||||
head's L2 coefficient from 0.0001 to 0.0003 kept the corpus and stratified gates
|
||||
intact while improving evidence that did not contain that sentence: four-fold
|
||||
cross-validation rose from 97/104 to 99/104 and whole-semantic-group holdout
|
||||
rose from 91/104 to 92/104. It also moved the original challenge to 24/24,
|
||||
although its +0.0001 minimum signed probability margin is correctly treated as
|
||||
a regression, not fresh proof.
|
||||
|
||||
Three checks now supply the honest evidence beyond the 72-case regression:
|
||||
|
||||
- Four-fold within-shape cross-validation scores 99/104.
|
||||
- Whole-group holdout scores 92/104 in aggregate.
|
||||
- A second 24-case challenge was written after the coefficient and frozen head
|
||||
were fixed. It returns to private configuration, commitments and stored
|
||||
choices versus public technical facts, compatibility and maintenance with
|
||||
new subjects. It scores 24/24 with minimum signed probability margin +0.1718.
|
||||
Its rows remain outside the training corpus and no result from it was used to
|
||||
retune this revision.
|
||||
|
||||
Whole-semantic-group holdout is intentionally diagnostic rather than claimed
|
||||
as perfect: remembered speech 14/16, possession 19/22, narrative 16/16,
|
||||
first-person preamble 15/16, advice/current information 13/16, and public proper
|
||||
nouns 15/18. This shows the embedder has a shared boundary direction but some
|
||||
ambiguities genuinely need shape coverage.
|
||||
|
||||
## Result
|
||||
|
||||
| Gate | three-neighbour | linear head |
|
||||
| --- | ---: | ---: |
|
||||
| historical regression | 29/29 | **29/29** |
|
||||
| stratified fixture | 61/72 | **72/72** |
|
||||
| Russian | 30/36 | **36/36** |
|
||||
| English | 31/36 | **36/36** |
|
||||
| expected personal | 28/36 | **36/36** |
|
||||
| expected world | 33/36 | **36/36** |
|
||||
|
||||
Every one of the six strata is 12/12. The minimum signed probability margin is
|
||||
+0.0522 after the independently supported regularisation change. The decision
|
||||
remains personal probability greater than 0.5; neither a lexical exception nor
|
||||
a shifted privacy prior was introduced.
|
||||
|
||||
Cold first-boundary load plus score is 15.179µs; steady inference is 1.112µs per
|
||||
score over 100,000 iterations. The previous runtime corpus fit took 5.618s on
|
||||
the same box, which is why fitting moved to the model-backed build gate.
|
||||
The unconfigured `HashEmbedder` floor still fits its own 1024-dimensional head
|
||||
because these ONNX weights do not belong in that space; its sparse-vector fit
|
||||
uses 400 steps and measures 93.818ms on first use.
|
||||
|
||||
The ONNX Runtime wrapper is process-global today, so these model-backed tests
|
||||
must be invoked in separate `go test` processes until V-716 repairs the harness;
|
||||
otherwise only the first test runs and the rest self-skip. The figures above
|
||||
were all collected as separate commands.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Three neighbours keep a public noun from opening the personal boundary
|
||||
|
||||
Measured 2026-08-13 on homesrv from `master` at `56254a5` plus the V-702
|
||||
working tree. Embedder: quantized multilingual-e5-small through ONNX Runtime
|
||||
1.26.0. Command:
|
||||
|
||||
```sh
|
||||
make t PKG=./cmd/mavend/ RUN=TestONNXPersonalBoundary V=1 RACE=0
|
||||
make t PKG=./cmd/mavend/ RUN=TestONNXTopics V=1 RACE=0
|
||||
```
|
||||
|
||||
This supersedes `2026-08-03-personal-boundary.md` for the boundary score. That
|
||||
file remains the measurement of why possession markers were replaced.
|
||||
|
||||
## Defect
|
||||
|
||||
The boundary used the best cosine on each side: one-nearest-neighbour
|
||||
classification. Its world class correctly includes narrative questions such
|
||||
as `расскажи про байкал`, while its personal class includes remembered-speech
|
||||
questions. The held-out `я рассказывал тебе про байкал?` therefore scored
|
||||
personal 0.9068 against world 0.9413. The public subject outweighed the question
|
||||
about what the owner had previously told Maven, and the boundary opened toward
|
||||
SearXNG.
|
||||
|
||||
Adding two plausible personal paraphrases did not move either score. Candidate
|
||||
similarities to the failing turn ranged from 0.8373 to 0.9022, still below the
|
||||
world proper-noun neighbour. This ruled out a missing synonym and argued against
|
||||
putting the held-out sentence or another Baikal-shaped exception into the seed
|
||||
set.
|
||||
|
||||
## Change
|
||||
|
||||
Each class now scores as the mean of its three nearest seeds. The same `k=3`
|
||||
applies on both sides, so the larger world class cannot win merely by having
|
||||
more coverage. Only the local neighbourhood is averaged because the personal
|
||||
class has two modes—possession and remembered speech—and a whole-class centroid
|
||||
would dilute them into each other.
|
||||
|
||||
This is a scoring change, not a seed or a phrase rule. The Baikal utterance
|
||||
remains held out.
|
||||
|
||||
## Result
|
||||
|
||||
| Gate | one neighbour | three neighbours |
|
||||
| --- | ---: | ---: |
|
||||
| personal boundary | 28/29 | **29/29** |
|
||||
| Baikal margin | -0.0345 | **+0.0147** |
|
||||
| topic recognisers | 43/43 | **43/43** |
|
||||
|
||||
The nearest personal true positive is `когда мой следующий отпуск` at +0.0110.
|
||||
The nearest world true negative is `расскажи про эверест` at -0.0135. No extra
|
||||
threshold is justified between them. The gate remains the sign of the class
|
||||
difference, favouring local refusal over sending owner-related text outward.
|
||||
@@ -52,7 +52,10 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the
|
||||
|
||||
| measurement | state |
|
||||
| --- | --- |
|
||||
| [Personal boundary, seed scoring vs possession markers](2026-08-03-personal-boundary.md) | live |
|
||||
| [Five turns retain one referent across fact, query and chat routes](2026-08-13-conversation-continuity.md) | live |
|
||||
| [Personal boundary, seed scoring vs possession markers](2026-08-03-personal-boundary.md) | superseded |
|
||||
| [Three neighbours keep a public noun from opening the personal boundary](2026-08-13-personal-boundary-neighbourhood.md) | superseded |
|
||||
| [A class-balanced linear head closes every held-out personal-boundary cell](2026-08-13-personal-boundary-linear-head.md) | live |
|
||||
| [Half-past and quarter-to hours](2026-08-05-half-past-hours.md) | live |
|
||||
| [Praxis reach at stage 0](2026-08-05-praxis-reach.md) | live |
|
||||
| [Alarm verbs reach stage 0](2026-08-06-alarm-verbs-reach-stage-0.md) | live |
|
||||
|
||||
@@ -135,6 +135,13 @@ second getting it right. Every rule was added against a measurement.
|
||||
and "сколько будет 17 на 23". Wired after the agenda rules and **before** the
|
||||
feed and list rules. "что такое лента" is a definition question, and the feed
|
||||
rule would take it on the noun alone.
|
||||
- **Russian possession statements** (`PossessionStatementGrammar`, V-577,
|
||||
2026-08-13). The closed grammatical frame `у меня …` anchors a plain
|
||||
declaration as `IntentNote`; its remainder is open and contains no noun list.
|
||||
Questions, captures, reminders and narrative requests keep their narrower
|
||||
routes. Hyphenated indefinite pronouns such as `что-то`, `кто-нибудь` and
|
||||
`кое-что` remain statements—the interrogative root inside that closed
|
||||
morphology is not a question by itself.
|
||||
|
||||
`calendar-query` and `event-time-query` name the calendar as the destination.
|
||||
The possessive agenda rules deliberately do not. "что у меня в списке покупок"
|
||||
@@ -357,6 +364,32 @@ at all. It was measured on the box 2026-08-07
|
||||
TCP?" and "сколько будет 17 на 23?" with "для какого города?". The feed answered
|
||||
"какой у меня любимый язык?" with kernel headlines.
|
||||
|
||||
### How the personal boundary scores
|
||||
|
||||
The boundary was fitted from a frozen bilingual training corpus for two classes:
|
||||
questions that require the owner's records, and questions an outside knowledge
|
||||
source can answer. The model-backed gate fits one class-balanced logistic head
|
||||
over those vectors, verifies it, and production loads its model-ID-pinned frozen
|
||||
weights. The head reads the whole semantic vector; it does not let the one
|
||||
training sentence with the nearest public noun decide.
|
||||
|
||||
The corpus covers remembered speech, possession, narrative, first-person
|
||||
preambles, current advice/information and public proper nouns on both sides.
|
||||
Each class contributes total training weight 0.5, so adding coverage to the
|
||||
larger world class cannot shift the prior merely by class size. The corpus is
|
||||
separate from both evaluation sets, and deterministic CI checks dimensions,
|
||||
class balancing, leakage, and leave-one-shape-out generalisation.
|
||||
|
||||
The measured result is 29/29 historical regressions, 72/72 on a balanced
|
||||
Russian/English stratified fixture, 99/104 four-fold training-corpus
|
||||
cross-validation, and 24/24 on a challenge authored after the final
|
||||
regularisation was frozen (V-702,
|
||||
`docs/evals/2026-08-13-personal-boundary-linear-head.md`).
|
||||
|
||||
The decision threshold remains exactly personal > world. This boundary is
|
||||
asymmetric: a false local claim costs an honest “не знаю”, while a false world
|
||||
claim can send a question about the owner upstream.
|
||||
|
||||
### Who may drop the personal boundary
|
||||
|
||||
The personal boundary guesses, so naming `SourceWorld` drops it. That is what
|
||||
@@ -480,3 +513,24 @@ All three reaches offer it as of 2026-08-06:
|
||||
|
||||
The turn source is still `tap:text` for both telegram and the web. So provenance
|
||||
cannot tell a chat turn from a typed one.
|
||||
|
||||
### Repair and a parked question
|
||||
|
||||
An explicit spoken correction gets first refusal before `clarify-answer`
|
||||
(V-573). It is narrower evidence than the mere presence of a parked question,
|
||||
so it must not be scored as bad slot material or spend that question's retry.
|
||||
The question is suspended and repeated after the repair reply. If redoing the
|
||||
corrected turn exposes a missing slot of its own, that new question is pushed
|
||||
on the dialogue stack; completing or cancelling it pops only the top and makes
|
||||
the older flow audible again.
|
||||
|
||||
Completed clarifications and repairs deliberately enter different wrappers.
|
||||
`finishClarified` acts on a decision whose parked gaps were already checked;
|
||||
`finishRepaired` first validates the rebuilt decision against the current
|
||||
required-slot schema. Both record the action as the next correctable turn.
|
||||
|
||||
The repair pointer is claimed atomically only after its window and target are
|
||||
valid. A stale correction changes no state. A correction naming the intent
|
||||
already used is acknowledged without redoing the action, and retains the
|
||||
pointer for a genuine correction that follows. This avoids routing correction
|
||||
prose as a fresh note and overwriting the referent it was meant to preserve.
|
||||
|
||||
Reference in New Issue
Block a user