From d773f1f72b300d3dc04a7b85ddc33216ebf2f412 Mon Sep 17 00:00:00 2001 From: claude Date: Tue, 4 Aug 2026 06:22:02 +0400 Subject: [PATCH] docs: record the ecosystem reach measurement (V-405) Praxis reach is zero on all twelve cases under both embedders, and it is structurally impossible rather than merely weak: handlePraxisAct dispatches on fn equality, and no praxis alias can ever enter the fn slot, because that slot is filled from the deployment's tool allowlist. Hexis reach is 9/10. All three services are up and answer; both praxis feeds are empty, so the gap is entirely on Maven's side of the wire. --- docs/evals/2026-08-04-ecosystem-reach.md | 100 +++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 docs/evals/2026-08-04-ecosystem-reach.md diff --git a/docs/evals/2026-08-04-ecosystem-reach.md b/docs/evals/2026-08-04-ecosystem-reach.md new file mode 100644 index 0000000..7a8fad3 --- /dev/null +++ b/docs/evals/2026-08-04-ecosystem-reach.md @@ -0,0 +1,100 @@ +# Ecosystem reach, measured — 2026-08-04 + +Vikunja #405. Measured at `86dcd99` on the 30-case held-out fixture +`internal/router/eval/ru_ecosystem_v1.json`, scored by `make eval-reach`. + +**Praxis reach is zero. Not low — zero, on all twelve cases, under both embedders.** + +## What was measured + +Reach is where an utterance *arrives*, not what it achieves. The derivation is in +`internal/router/eval/reach.go` and mirrors `actionAct` in `cmd/mavend/actions_act.go`: + +- **Praxis** needs `IntentAct` and a fn slot whose value is one of the aliases in + `praxisCapabilities`. +- **Hexis** needs `IntentAct` and non-empty `Slots.Text`. It also fires from + `hexisBeforeClarify`, so a clarified act with text reaches Hexis before the clarify + question is ever asked. +- Everything else stays inside Maven. + +The fixture holds 10 Hexis cases, 12 Praxis cases and 8 negatives. The negatives carry +the expensive direction: an utterance that reaches a mutating path it had no business +reaching is worse than one that never arrives, because he never gets asked about it. + +## The numbers + +| Path | Reached the right place | Missed | Overreach | Wrong service | p50 | +|---|---|---|---|---|---| +| classifier + hash embedder | 7/30 (23.3%) | 22 | 1 | 0 | 14.8µs | +| classifier + e5-small (deployed) | 16/30 (53.3%) | 11 | 1 | 2 | 23.8ms | + +Split by target, on the deployed embedder: + +| Want | Passed | +|---|---| +| hexis | 9/10 | +| none | 7/8 | +| **praxis** | **0/12** | + +## The finding + +Hexis reach is fine. Nine of ten act-shaped home and homelab utterances land on the +entity resolver, which is what the gate was built to do: it only needs the intent and +some text, and both the act grammar and the embedder produce those. + +Praxis reach is structurally impossible from free Russian, and the fixture makes that +visible for the first time. `handlePraxisAct` dispatches on exact equality between +`Slots.Fn` and a capability alias. The fn slot is filled by `DefaultActMatcher`, whose +allowlist is the deployment's enabled tool names — `перезапусти`, `выключи`, and so on. +No Praxis alias is in that list, so no utterance can ever put one in the slot. The +Russian aliases in `praxisCapabilities` (`готово`, `принято`, `игнорировать`) read as if +they match speech and they do not: they are compared against a fn slot, never against +the utterance. + +That means the whole lifecycle half of the Praxis contract — acknowledge, resolve, +ignore, pin — has no voice path at all. Attention and changes have none either. + +Three of the twelve got as far as the wrong place, which is the same defect seen from the +other side: `"готово, закрывай"` and `"как дела у праксиса"` route to act with text, so +they fall past the Praxis check into the Hexis one and go to entity resolution instead. + +## The live services + +All three are up and answer. With `no_proxy` set for the loopback (the host's `http_proxy` +answers 503 for 127.0.0.1, which is the same trap `llmrouter_test.go` documents): + +``` +127.0.0.1:8989/health -> {"status":"ok"} praxis +127.0.0.1:9740/health -> {"status":"ok"} nexus +127.0.0.1:9741/health -> {"status":"ok"} hexis +127.0.0.1:8989/api/v1/tools/attention?limit=3 -> [] +127.0.0.1:8989/api/v1/tools/changes?limit=3 -> [] +``` + +So the boundary is not the problem, and an end-to-end run today would add nothing: both +Praxis feeds are empty, so even a perfect reach score would produce "ничего не требует +внимания". The gap is entirely on Maven's side of the wire. + +## Not measured + +**The resident model.** The LLM router is the deployed default, and these numbers are the +classifier only. `mavend` spawns its llama-server on a container-local port +(127.0.0.1:40063 inside `maven-mavend-1`), unreachable from the host, and the workstation +at 192.168.1.105:8080 was down. The classifier is the failure floor and it is what always +answers, so the floor is worth knowing on its own — but the LLM router could fill the fn +slot with a literal `list_attention`, since the grammar lets it emit any string. Whether +it does is the open question, and the fixture is ready for it. + +## What this argues for + +Not a new intent. The seven are frozen by prompt parity with the training workspace. + +The cheap fix is stage 0: a grammar per Praxis capability that sets `Slots.Fn` to the +canonical arm name, the same trick `AgendaQueryGrammars` used to take agenda questions off +the model. It costs one regex per capability on every turn and it is deterministic, which +for a lifecycle verb is the right trade — "отметь это как сделанное" should never be a +similarity guess. + +The second fix is smaller and separate: `entity_attention` aliases to grammar names only, +so scoped attention ("что там с нексусом") needs a grammar before it can be reached at +all.