docs: record the ecosystem reach measurement (V-405)
Praxis reach is zero on all twelve cases under both embedders, and it is structurally impossible rather than merely weak: handlePraxisAct dispatches on fn equality, and no praxis alias can ever enter the fn slot, because that slot is filled from the deployment's tool allowlist. Hexis reach is 9/10. All three services are up and answer; both praxis feeds are empty, so the gap is entirely on Maven's side of the wire.
This commit is contained in:
@@ -0,0 +1,100 @@
|
||||
# Ecosystem reach, measured — 2026-08-04
|
||||
|
||||
Vikunja #405. Measured at `86dcd99` on the 30-case held-out fixture
|
||||
`internal/router/eval/ru_ecosystem_v1.json`, scored by `make eval-reach`.
|
||||
|
||||
**Praxis reach is zero. Not low — zero, on all twelve cases, under both embedders.**
|
||||
|
||||
## What was measured
|
||||
|
||||
Reach is where an utterance *arrives*, not what it achieves. The derivation is in
|
||||
`internal/router/eval/reach.go` and mirrors `actionAct` in `cmd/mavend/actions_act.go`:
|
||||
|
||||
- **Praxis** needs `IntentAct` and a fn slot whose value is one of the aliases in
|
||||
`praxisCapabilities`.
|
||||
- **Hexis** needs `IntentAct` and non-empty `Slots.Text`. It also fires from
|
||||
`hexisBeforeClarify`, so a clarified act with text reaches Hexis before the clarify
|
||||
question is ever asked.
|
||||
- Everything else stays inside Maven.
|
||||
|
||||
The fixture holds 10 Hexis cases, 12 Praxis cases and 8 negatives. The negatives carry
|
||||
the expensive direction: an utterance that reaches a mutating path it had no business
|
||||
reaching is worse than one that never arrives, because he never gets asked about it.
|
||||
|
||||
## The numbers
|
||||
|
||||
| Path | Reached the right place | Missed | Overreach | Wrong service | p50 |
|
||||
|---|---|---|---|---|---|
|
||||
| classifier + hash embedder | 7/30 (23.3%) | 22 | 1 | 0 | 14.8µs |
|
||||
| classifier + e5-small (deployed) | 16/30 (53.3%) | 11 | 1 | 2 | 23.8ms |
|
||||
|
||||
Split by target, on the deployed embedder:
|
||||
|
||||
| Want | Passed |
|
||||
|---|---|
|
||||
| hexis | 9/10 |
|
||||
| none | 7/8 |
|
||||
| **praxis** | **0/12** |
|
||||
|
||||
## The finding
|
||||
|
||||
Hexis reach is fine. Nine of ten act-shaped home and homelab utterances land on the
|
||||
entity resolver, which is what the gate was built to do: it only needs the intent and
|
||||
some text, and both the act grammar and the embedder produce those.
|
||||
|
||||
Praxis reach is structurally impossible from free Russian, and the fixture makes that
|
||||
visible for the first time. `handlePraxisAct` dispatches on exact equality between
|
||||
`Slots.Fn` and a capability alias. The fn slot is filled by `DefaultActMatcher`, whose
|
||||
allowlist is the deployment's enabled tool names — `перезапусти`, `выключи`, and so on.
|
||||
No Praxis alias is in that list, so no utterance can ever put one in the slot. The
|
||||
Russian aliases in `praxisCapabilities` (`готово`, `принято`, `игнорировать`) read as if
|
||||
they match speech and they do not: they are compared against a fn slot, never against
|
||||
the utterance.
|
||||
|
||||
That means the whole lifecycle half of the Praxis contract — acknowledge, resolve,
|
||||
ignore, pin — has no voice path at all. Attention and changes have none either.
|
||||
|
||||
Three of the twelve got as far as the wrong place, which is the same defect seen from the
|
||||
other side: `"готово, закрывай"` and `"как дела у праксиса"` route to act with text, so
|
||||
they fall past the Praxis check into the Hexis one and go to entity resolution instead.
|
||||
|
||||
## The live services
|
||||
|
||||
All three are up and answer. With `no_proxy` set for the loopback (the host's `http_proxy`
|
||||
answers 503 for 127.0.0.1, which is the same trap `llmrouter_test.go` documents):
|
||||
|
||||
```
|
||||
127.0.0.1:8989/health -> {"status":"ok"} praxis
|
||||
127.0.0.1:9740/health -> {"status":"ok"} nexus
|
||||
127.0.0.1:9741/health -> {"status":"ok"} hexis
|
||||
127.0.0.1:8989/api/v1/tools/attention?limit=3 -> []
|
||||
127.0.0.1:8989/api/v1/tools/changes?limit=3 -> []
|
||||
```
|
||||
|
||||
So the boundary is not the problem, and an end-to-end run today would add nothing: both
|
||||
Praxis feeds are empty, so even a perfect reach score would produce "ничего не требует
|
||||
внимания". The gap is entirely on Maven's side of the wire.
|
||||
|
||||
## Not measured
|
||||
|
||||
**The resident model.** The LLM router is the deployed default, and these numbers are the
|
||||
classifier only. `mavend` spawns its llama-server on a container-local port
|
||||
(127.0.0.1:40063 inside `maven-mavend-1`), unreachable from the host, and the workstation
|
||||
at 192.168.1.105:8080 was down. The classifier is the failure floor and it is what always
|
||||
answers, so the floor is worth knowing on its own — but the LLM router could fill the fn
|
||||
slot with a literal `list_attention`, since the grammar lets it emit any string. Whether
|
||||
it does is the open question, and the fixture is ready for it.
|
||||
|
||||
## What this argues for
|
||||
|
||||
Not a new intent. The seven are frozen by prompt parity with the training workspace.
|
||||
|
||||
The cheap fix is stage 0: a grammar per Praxis capability that sets `Slots.Fn` to the
|
||||
canonical arm name, the same trick `AgendaQueryGrammars` used to take agenda questions off
|
||||
the model. It costs one regex per capability on every turn and it is deterministic, which
|
||||
for a lifecycle verb is the right trade — "отметь это как сделанное" should never be a
|
||||
similarity guess.
|
||||
|
||||
The second fix is smaller and separate: `entity_attention` aliases to grammar names only,
|
||||
so scoped attention ("что там с нексусом") needs a grammar before it can be reached at
|
||||
all.
|
||||
Reference in New Issue
Block a user