Harden semantic boundaries and repair dialogue state
Replace nearest-neighbour personal routing with a frozen class-balanced linear head measured on historical, stratified, cross-validation, holdout, and fresh challenge gates (V-702). Close the four repair handoff holes, preserve nested clarification flows, and route Russian possession statements through structural grammar rather than lexical exceptions (V-573). Owner explicitly requested direct commits to master.
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
# A class-balanced linear head closes every held-out personal-boundary cell
|
||||
|
||||
Measured 2026-08-13 on homesrv from `master` at `56254a5` plus the V-702
|
||||
working tree. Embedder: quantized multilingual-e5-small through ONNX Runtime
|
||||
1.26.0. This supersedes
|
||||
`2026-08-13-personal-boundary-neighbourhood.md`.
|
||||
|
||||
Commands:
|
||||
|
||||
```sh
|
||||
MAVEN_ONNX_LIB="$PWD/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so.1.26.0" \
|
||||
go test ./cmd/mavend -run '^TestONNXPersonalBoundary$' -count=1 -v
|
||||
MAVEN_ONNX_LIB="$PWD/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so.1.26.0" \
|
||||
MAVEN_EVAL_PERSONAL_BOUNDARY=1 \
|
||||
go test ./cmd/mavend -run '^TestONNXPersonalBoundaryStratified$' -count=1 -v
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
The prior three-neighbour scorer passed the 29-case regression table but
|
||||
missed 11 of a new 72-case fixture. The fixture is balanced across Russian and
|
||||
English, expected personal and world classes, and six sentence shapes:
|
||||
remembered speech, possession, narrative, first-person preamble, current
|
||||
advice/information and public proper nouns. Its hygiene test rejects copied
|
||||
training phrases and repeated Baikal variants.
|
||||
|
||||
Production uses a binary logistic head fitted over 132 bilingual examples.
|
||||
Each class has total sample weight 0.5 even though the class sizes differ. The
|
||||
optimiser is deterministic: 5,000 full-batch steps, decaying learning rate from
|
||||
10, L2 coefficient 0.0003. The fitted 384 weights are frozen into the binary
|
||||
and pinned to the embedder model ID; a model-backed test retrains from the text
|
||||
corpus and rejects any drift. An unknown embedder ID falls back to fitting its
|
||||
own corpus rather than applying weights from another vector space.
|
||||
|
||||
No evaluation utterance is in the training corpus. Ordinary CI checks class
|
||||
balancing, mixed dimensions, exact regression-table leakage, and generalisation
|
||||
across synthetic semantic directions. The model-backed gate also runs
|
||||
four-fold cross-validation over the expanded corpus, stratified inside each
|
||||
sentence shape and class; the original compact corpus remains in every fold as
|
||||
the historical floor.
|
||||
|
||||
The six shapes of the training corpus were chosen after the first 72-case
|
||||
matrix exposed their gaps, so 72/72 is a regression gate, not independent proof
|
||||
of generalisation. The first independently authored 24-case challenge then
|
||||
exposed one remaining miss, `на каком порту я решил поднять тестовый сервис?`,
|
||||
at personal probability 0.4666. It became model-selection data at that point
|
||||
and cannot honestly remain the independent result.
|
||||
|
||||
A comparison rejected shrinkage LDA (24/24 challenge but 69/72 stratified) and
|
||||
an LDA/logistic blend (24/24 but 71/72). Increasing only the balanced logistic
|
||||
head's L2 coefficient from 0.0001 to 0.0003 kept the corpus and stratified gates
|
||||
intact while improving evidence that did not contain that sentence: four-fold
|
||||
cross-validation rose from 97/104 to 99/104 and whole-semantic-group holdout
|
||||
rose from 91/104 to 92/104. It also moved the original challenge to 24/24,
|
||||
although its +0.0001 minimum signed probability margin is correctly treated as
|
||||
a regression, not fresh proof.
|
||||
|
||||
Three checks now supply the honest evidence beyond the 72-case regression:
|
||||
|
||||
- Four-fold within-shape cross-validation scores 99/104.
|
||||
- Whole-group holdout scores 92/104 in aggregate.
|
||||
- A second 24-case challenge was written after the coefficient and frozen head
|
||||
were fixed. It returns to private configuration, commitments and stored
|
||||
choices versus public technical facts, compatibility and maintenance with
|
||||
new subjects. It scores 24/24 with minimum signed probability margin +0.1718.
|
||||
Its rows remain outside the training corpus and no result from it was used to
|
||||
retune this revision.
|
||||
|
||||
Whole-semantic-group holdout is intentionally diagnostic rather than claimed
|
||||
as perfect: remembered speech 14/16, possession 19/22, narrative 16/16,
|
||||
first-person preamble 15/16, advice/current information 13/16, and public proper
|
||||
nouns 15/18. This shows the embedder has a shared boundary direction but some
|
||||
ambiguities genuinely need shape coverage.
|
||||
|
||||
## Result
|
||||
|
||||
| Gate | three-neighbour | linear head |
|
||||
| --- | ---: | ---: |
|
||||
| historical regression | 29/29 | **29/29** |
|
||||
| stratified fixture | 61/72 | **72/72** |
|
||||
| Russian | 30/36 | **36/36** |
|
||||
| English | 31/36 | **36/36** |
|
||||
| expected personal | 28/36 | **36/36** |
|
||||
| expected world | 33/36 | **36/36** |
|
||||
|
||||
Every one of the six strata is 12/12. The minimum signed probability margin is
|
||||
+0.0522 after the independently supported regularisation change. The decision
|
||||
remains personal probability greater than 0.5; neither a lexical exception nor
|
||||
a shifted privacy prior was introduced.
|
||||
|
||||
Cold first-boundary load plus score is 15.179µs; steady inference is 1.112µs per
|
||||
score over 100,000 iterations. The previous runtime corpus fit took 5.618s on
|
||||
the same box, which is why fitting moved to the model-backed build gate.
|
||||
The unconfigured `HashEmbedder` floor still fits its own 1024-dimensional head
|
||||
because these ONNX weights do not belong in that space; its sparse-vector fit
|
||||
uses 400 steps and measures 93.818ms on first use.
|
||||
|
||||
The ONNX Runtime wrapper is process-global today, so these model-backed tests
|
||||
must be invoked in separate `go test` processes until V-716 repairs the harness;
|
||||
otherwise only the first test runs and the rest self-skip. The figures above
|
||||
were all collected as separate commands.
|
||||
@@ -0,0 +1,53 @@
|
||||
# Three neighbours keep a public noun from opening the personal boundary
|
||||
|
||||
Measured 2026-08-13 on homesrv from `master` at `56254a5` plus the V-702
|
||||
working tree. Embedder: quantized multilingual-e5-small through ONNX Runtime
|
||||
1.26.0. Command:
|
||||
|
||||
```sh
|
||||
make t PKG=./cmd/mavend/ RUN=TestONNXPersonalBoundary V=1 RACE=0
|
||||
make t PKG=./cmd/mavend/ RUN=TestONNXTopics V=1 RACE=0
|
||||
```
|
||||
|
||||
This supersedes `2026-08-03-personal-boundary.md` for the boundary score. That
|
||||
file remains the measurement of why possession markers were replaced.
|
||||
|
||||
## Defect
|
||||
|
||||
The boundary used the best cosine on each side: one-nearest-neighbour
|
||||
classification. Its world class correctly includes narrative questions such
|
||||
as `расскажи про байкал`, while its personal class includes remembered-speech
|
||||
questions. The held-out `я рассказывал тебе про байкал?` therefore scored
|
||||
personal 0.9068 against world 0.9413. The public subject outweighed the question
|
||||
about what the owner had previously told Maven, and the boundary opened toward
|
||||
SearXNG.
|
||||
|
||||
Adding two plausible personal paraphrases did not move either score. Candidate
|
||||
similarities to the failing turn ranged from 0.8373 to 0.9022, still below the
|
||||
world proper-noun neighbour. This ruled out a missing synonym and argued against
|
||||
putting the held-out sentence or another Baikal-shaped exception into the seed
|
||||
set.
|
||||
|
||||
## Change
|
||||
|
||||
Each class now scores as the mean of its three nearest seeds. The same `k=3`
|
||||
applies on both sides, so the larger world class cannot win merely by having
|
||||
more coverage. Only the local neighbourhood is averaged because the personal
|
||||
class has two modes—possession and remembered speech—and a whole-class centroid
|
||||
would dilute them into each other.
|
||||
|
||||
This is a scoring change, not a seed or a phrase rule. The Baikal utterance
|
||||
remains held out.
|
||||
|
||||
## Result
|
||||
|
||||
| Gate | one neighbour | three neighbours |
|
||||
| --- | ---: | ---: |
|
||||
| personal boundary | 28/29 | **29/29** |
|
||||
| Baikal margin | -0.0345 | **+0.0147** |
|
||||
| topic recognisers | 43/43 | **43/43** |
|
||||
|
||||
The nearest personal true positive is `когда мой следующий отпуск` at +0.0110.
|
||||
The nearest world true negative is `расскажи про эверест` at -0.0135. No extra
|
||||
threshold is justified between them. The gate remains the sign of the class
|
||||
difference, favouring local refusal over sending owner-related text outward.
|
||||
@@ -52,7 +52,10 @@ A pair in `docs/routing.md` went stale unnoticed. Its source predated the
|
||||
|
||||
| measurement | state |
|
||||
| --- | --- |
|
||||
| [Personal boundary, seed scoring vs possession markers](2026-08-03-personal-boundary.md) | live |
|
||||
| [Five turns retain one referent across fact, query and chat routes](2026-08-13-conversation-continuity.md) | live |
|
||||
| [Personal boundary, seed scoring vs possession markers](2026-08-03-personal-boundary.md) | superseded |
|
||||
| [Three neighbours keep a public noun from opening the personal boundary](2026-08-13-personal-boundary-neighbourhood.md) | superseded |
|
||||
| [A class-balanced linear head closes every held-out personal-boundary cell](2026-08-13-personal-boundary-linear-head.md) | live |
|
||||
| [Half-past and quarter-to hours](2026-08-05-half-past-hours.md) | live |
|
||||
| [Praxis reach at stage 0](2026-08-05-praxis-reach.md) | live |
|
||||
| [Alarm verbs reach stage 0](2026-08-06-alarm-verbs-reach-stage-0.md) | live |
|
||||
|
||||
Reference in New Issue
Block a user