2 Commits

Author SHA1 Message Date
kami 968cbfa973 fix(context,tools): context hygiene + tool ergonomics from freestyle QA
Found while live-QAing freestyle_planning on a 12B local model:

- list_dir tool: recursive, .gitignore-aware listing so weak models stop
  flooding context with `ls -R` over node_modules/build/dist. Wired into
  the fileRead toggle + advertised to the planner (architect_freestyle).
- ContextClassifier: assistantToolCall turns are STRUCTURED, so the token
  pruner never shreds the model's own tool-call history — that was causing
  amnesia loops (re-issuing calls it had already made).
- Retire instruction-doc LLMLingua pruning (DOC_SOURCE_TYPES emptied): it
  fused load-bearing procedural text into unparseable soup. The static
  block stays small by dropping CLAUDE.md at the loader instead.
- AgentInstructionsLoader: load only AGENTS.md, not CLAUDE.md — the latter
  targets the outer assistant and polluted the agent's stage context.
- DefaultSessionReducer: WorkflowFailed flips session status to FAILED (was
  stuck ACTIVE forever, so clients/approval loops never saw a terminal).
- ShellTool: run shell command lines (cd/&&/pipes) via `sh -c`; unrunnable
  program is recoverable instead of an uncaught IOException killing the
  stage; malformed argv (non-string/collapsed-array) rejected with guidance.
- llmlingua sidecar: cap force_tokens to max_force_token (big docs blew the
  assert and 500'd, so doc pruning silently failed open).

Tests added/updated across all of the above.
2026-07-02 00:44:04 +04:00
kami 047e2a4070 feat(context,infra): compression pipeline stages 4-5 — token pruning + relevance + ToMe
- build() is now suspend: pipeline runs all stages in fixed order, each gated by level
- TOKEN_PRUNE (level 3): TokenPruner interface + LLMLingua-2 sidecar (sidecars/llmlingua)
  + HttpTokenPruner adapter (fails open if sidecar down); prunes freeform, preserves
  protected spans, skips tier-0 turns when TIER_SPLIT on
- TOME_MERGE (level 8): ToMeMerger collapses near-duplicate freeform turns (Jaccard)
- Stage 5 selection: RelevanceScorer + EmbeddingRelevanceScorer (cosine over Embedder);
  query-conditioned reorder so least-relevant freeform drops first under budget
- [orchestration] compression_level + token_pruner_url config, wired in Main
- suspend ripple fixed across builder callers/stubs
2026-07-01 14:29:56 +04:00