claude
8015fdbb79
Replace nearest-neighbour personal routing with a frozen class-balanced linear head measured on historical, stratified, cross-validation, holdout, and fresh challenge gates (V-702). Close the four repair handoff holes, preserve nested clarification flows, and route Russian possession statements through structural grammar rather than lexical exceptions (V-573). Owner explicitly requested direct commits to master.
102 lines
5.2 KiB
Markdown
102 lines
5.2 KiB
Markdown
# A class-balanced linear head closes every held-out personal-boundary cell
|
|
|
|
Measured 2026-08-13 on homesrv from `master` at `56254a5` plus the V-702
|
|
working tree. Embedder: quantized multilingual-e5-small through ONNX Runtime
|
|
1.26.0. This supersedes
|
|
`2026-08-13-personal-boundary-neighbourhood.md`.
|
|
|
|
Commands:
|
|
|
|
```sh
|
|
MAVEN_ONNX_LIB="$PWD/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so.1.26.0" \
|
|
go test ./cmd/mavend -run '^TestONNXPersonalBoundary$' -count=1 -v
|
|
MAVEN_ONNX_LIB="$PWD/deps/onnxruntime-linux-x64-1.26.0/lib/libonnxruntime.so.1.26.0" \
|
|
MAVEN_EVAL_PERSONAL_BOUNDARY=1 \
|
|
go test ./cmd/mavend -run '^TestONNXPersonalBoundaryStratified$' -count=1 -v
|
|
```
|
|
|
|
## Setup
|
|
|
|
The prior three-neighbour scorer passed the 29-case regression table but
|
|
missed 11 of a new 72-case fixture. The fixture is balanced across Russian and
|
|
English, expected personal and world classes, and six sentence shapes:
|
|
remembered speech, possession, narrative, first-person preamble, current
|
|
advice/information and public proper nouns. Its hygiene test rejects copied
|
|
training phrases and repeated Baikal variants.
|
|
|
|
Production uses a binary logistic head fitted over 132 bilingual examples.
|
|
Each class has total sample weight 0.5 even though the class sizes differ. The
|
|
optimiser is deterministic: 5,000 full-batch steps, decaying learning rate from
|
|
10, L2 coefficient 0.0003. The fitted 384 weights are frozen into the binary
|
|
and pinned to the embedder model ID; a model-backed test retrains from the text
|
|
corpus and rejects any drift. An unknown embedder ID falls back to fitting its
|
|
own corpus rather than applying weights from another vector space.
|
|
|
|
No evaluation utterance is in the training corpus. Ordinary CI checks class
|
|
balancing, mixed dimensions, exact regression-table leakage, and generalisation
|
|
across synthetic semantic directions. The model-backed gate also runs
|
|
four-fold cross-validation over the expanded corpus, stratified inside each
|
|
sentence shape and class; the original compact corpus remains in every fold as
|
|
the historical floor.
|
|
|
|
The six shapes of the training corpus were chosen after the first 72-case
|
|
matrix exposed their gaps, so 72/72 is a regression gate, not independent proof
|
|
of generalisation. The first independently authored 24-case challenge then
|
|
exposed one remaining miss, `на каком порту я решил поднять тестовый сервис?`,
|
|
at personal probability 0.4666. It became model-selection data at that point
|
|
and cannot honestly remain the independent result.
|
|
|
|
A comparison rejected shrinkage LDA (24/24 challenge but 69/72 stratified) and
|
|
an LDA/logistic blend (24/24 but 71/72). Increasing only the balanced logistic
|
|
head's L2 coefficient from 0.0001 to 0.0003 kept the corpus and stratified gates
|
|
intact while improving evidence that did not contain that sentence: four-fold
|
|
cross-validation rose from 97/104 to 99/104 and whole-semantic-group holdout
|
|
rose from 91/104 to 92/104. It also moved the original challenge to 24/24,
|
|
although its +0.0001 minimum signed probability margin is correctly treated as
|
|
a regression, not fresh proof.
|
|
|
|
Three checks now supply the honest evidence beyond the 72-case regression:
|
|
|
|
- Four-fold within-shape cross-validation scores 99/104.
|
|
- Whole-group holdout scores 92/104 in aggregate.
|
|
- A second 24-case challenge was written after the coefficient and frozen head
|
|
were fixed. It returns to private configuration, commitments and stored
|
|
choices versus public technical facts, compatibility and maintenance with
|
|
new subjects. It scores 24/24 with minimum signed probability margin +0.1718.
|
|
Its rows remain outside the training corpus and no result from it was used to
|
|
retune this revision.
|
|
|
|
Whole-semantic-group holdout is intentionally diagnostic rather than claimed
|
|
as perfect: remembered speech 14/16, possession 19/22, narrative 16/16,
|
|
first-person preamble 15/16, advice/current information 13/16, and public proper
|
|
nouns 15/18. This shows the embedder has a shared boundary direction but some
|
|
ambiguities genuinely need shape coverage.
|
|
|
|
## Result
|
|
|
|
| Gate | three-neighbour | linear head |
|
|
| --- | ---: | ---: |
|
|
| historical regression | 29/29 | **29/29** |
|
|
| stratified fixture | 61/72 | **72/72** |
|
|
| Russian | 30/36 | **36/36** |
|
|
| English | 31/36 | **36/36** |
|
|
| expected personal | 28/36 | **36/36** |
|
|
| expected world | 33/36 | **36/36** |
|
|
|
|
Every one of the six strata is 12/12. The minimum signed probability margin is
|
|
+0.0522 after the independently supported regularisation change. The decision
|
|
remains personal probability greater than 0.5; neither a lexical exception nor
|
|
a shifted privacy prior was introduced.
|
|
|
|
Cold first-boundary load plus score is 15.179µs; steady inference is 1.112µs per
|
|
score over 100,000 iterations. The previous runtime corpus fit took 5.618s on
|
|
the same box, which is why fitting moved to the model-backed build gate.
|
|
The unconfigured `HashEmbedder` floor still fits its own 1024-dimensional head
|
|
because these ONNX weights do not belong in that space; its sparse-vector fit
|
|
uses 400 steps and measures 93.818ms on first use.
|
|
|
|
The ONNX Runtime wrapper is process-global today, so these model-backed tests
|
|
must be invoked in separate `go test` processes until V-716 repairs the harness;
|
|
otherwise only the first test runs and the rest self-skip. The figures above
|
|
were all collected as separate commands.
|