Maven/docs/evals/2026-08-04-ecosystem-reach.md

Ecosystem reach, measured — 2026-08-04

Vikunja #405. Measured at 86dcd99 on the 30-case held-out fixture internal/router/eval/ru_ecosystem_v1.json, scored by make eval-reach.

Praxis reach is zero. Not low — zero, on all twelve cases, under both embedders.

What was measured

Reach is where an utterance arrives, not what it achieves. The derivation is in internal/router/eval/reach.go and mirrors actionAct in cmd/mavend/actions_act.go:

• Praxis needs IntentAct and a fn slot whose value is one of the aliases in praxisCapabilities.
• Hexis needs IntentAct and non-empty Slots.Text. It also fires from hexisBeforeClarify, so a clarified act with text reaches Hexis before the clarify question is ever asked.
• Everything else stays inside Maven.

The fixture holds 10 Hexis cases, 12 Praxis cases and 8 negatives. The negatives carry the expensive direction: an utterance that reaches a mutating path it had no business reaching is worse than one that never arrives, because he never gets asked about it.

The numbers

Path	Reached the right place	Missed	Overreach	Wrong service	p50
classifier + hash embedder	7/30 (23.3%)	22	1	0	14.8µs
classifier + e5-small (deployed)	16/30 (53.3%)	11	1	2	23.8ms

Split by target, on the deployed embedder:

Want	Passed
hexis	9/10
none	7/8
praxis	0/12

The finding

Hexis reach is fine. Nine of ten act-shaped home and homelab utterances land on the entity resolver, which is what the gate was built to do: it only needs the intent and some text, and both the act grammar and the embedder produce those.

Praxis reach is structurally impossible from free Russian, and the fixture makes that visible for the first time. handlePraxisAct dispatches on exact equality between Slots.Fn and a capability alias. The fn slot is filled by DefaultActMatcher, whose allowlist is the deployment's enabled tool names — перезапусти, выключи, and so on. No Praxis alias is in that list, so no utterance can ever put one in the slot. The Russian aliases in praxisCapabilities (готово, принято, игнорировать) read as if they match speech and they do not: they are compared against a fn slot, never against the utterance.

That means the whole lifecycle half of the Praxis contract — acknowledge, resolve, ignore, pin — has no voice path at all. Attention and changes have none either.

Three of the twelve got as far as the wrong place, which is the same defect seen from the other side: "готово, закрывай" and "как дела у праксиса" route to act with text, so they fall past the Praxis check into the Hexis one and go to entity resolution instead.

The live services

All three are up and answer. With no_proxy set for the loopback (the host's http_proxy answers 503 for 127.0.0.1, which is the same trap llmrouter_test.go documents):

127.0.0.1:8989/health -> {"status":"ok"}   praxis
127.0.0.1:9740/health -> {"status":"ok"}   nexus
127.0.0.1:9741/health -> {"status":"ok"}   hexis
127.0.0.1:8989/api/v1/tools/attention?limit=3 -> []
127.0.0.1:8989/api/v1/tools/changes?limit=3   -> []

So the boundary is not the problem, and an end-to-end run today would add nothing: both Praxis feeds are empty, so even a perfect reach score would produce "ничего не требует внимания". The gap is entirely on Maven's side of the wire.

Not measured

The resident model. The LLM router is the deployed default, and these numbers are the classifier only. mavend spawns its llama-server on a container-local port (127.0.0.1:40063 inside maven-mavend-1), unreachable from the host, and the workstation at 192.168.1.105:8080 was down. The classifier is the failure floor and it is what always answers, so the floor is worth knowing on its own — but the LLM router could fill the fn slot with a literal list_attention, since the grammar lets it emit any string. Whether it does is the open question, and the fixture is ready for it.

What this argues for

Not a new intent. The seven are frozen by prompt parity with the training workspace.

The cheap fix is stage 0: a grammar per Praxis capability that sets Slots.Fn to the canonical arm name, the same trick AgendaQueryGrammars used to take agenda questions off the model. It costs one regex per capability on every turn and it is deterministic, which for a lifecycle verb is the right trade — "отметь это как сделанное" should never be a similarity guess.

The second fix is smaller and separate: entity_attention aliases to grammar names only, so scoped attention ("что там с нексусом") needs a grammar before it can be reached at all.