kami
5fe8f228c1
Add a read-only /ecosystem page that consumes the sibling services' JSON APIs (Nexus entities, Praxis attention, Hexis capabilities), fetched concurrently with honest per-panel error states. Siblings stay headless — mavweb is their human surface (arch §16). Wired via mavweb -nexus/-praxis/-hexis flags; mavweb joins the ecosystem compose network. Fix mobile horizontal overflow across all pages: .content is a flex child with default min-width:auto, so it refused to shrink below the tables' intrinsic width. min-width:0 lets wide tables pan inside .scroll instead of dragging the page sideways. Verified via CDP geometry check (scrollWidth === clientWidth at 430px). Also includes in-progress Ethos UI redesign, ecosystem deploy compose, and planning docs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
28 lines
2.2 KiB
Markdown
28 lines
2.2 KiB
Markdown
# Plan: Vision — Image Understanding Capability
|
|
|
|
**Goal:** Maven can "see" — accept images (from mavweb upload, Telegram, or filesystem paths), run vision inference via a local or remote multimodal model, and answer questions about the image content or extract structured information.
|
|
|
|
**Done when:**
|
|
- Vision model backend is configurable: local multimodal LLM (e.g., LLaVA, Qwen-VL via `llama-server` mmproj) or remote API
|
|
- `internal/vision/` package handles image preprocessing, model inference, result parsing
|
|
- Voice/text commands like "что на картинке?" or "прочитай текст с экрана" route to the vision handler
|
|
- Extracted information can be written as facts/notes through `ipc.CoreAPI`
|
|
- Telegram image messages are processed through the same pipeline
|
|
|
|
**Scope:**
|
|
- New `internal/vision/` package — image loader (Go stdlib `image` + `golang.org/x/image`), inference client
|
|
- New config block: `voice.vision` in `config.Config` — `{enabled, provider, model_path, mmproj_path, remote_url}`
|
|
- Router intent extension: new `IntentVision` or reuse `IntentQuery` with a vision flag
|
|
- Reuses `internal/llm.Client` for API-compatible backends (OpenAI-compatible vision API)
|
|
- Reuses `internal/ipc.CoreAPI` for writing extracted data
|
|
|
|
**Steps:**
|
|
1. Create `internal/vision/provider.go` — `Provider` interface with `Describe(image []byte, prompt string) (string, error)` and `ExtractText(image []byte) (string, error)`
|
|
2. Implement `LocalProvider` — spawns `llama-server` with mmproj, sends multimodal chat completion requests
|
|
3. Implement `RemoteProvider` — calls an OpenAI-compatible vision API endpoint, reuses `internal/llm.Client`
|
|
4. Create `internal/vision/processor.go` — image preprocessing (resize, format conversion to JPEG/PNG, base64 encoding)
|
|
5. Wire vision into `cmd/mavend/voice.go:reactiveHandler` — detect vision intent from router (new `IntentVision` or a `Slots.HasImage` flag)
|
|
6. Add IPC method `MethodDescribeImage` for programmatic access (mavweb upload, telegram bot)
|
|
7. Add vision config block to `config.Config` and wire in `cmd/mavend/main.go`
|
|
8. Test with a local multimodal model: send an image via mavweb, verify description and text extraction
|