Store and describe images through a shared media intake (#252)
Vision needs a second model this box does not have, so the shipped half is the part that works without one: an image arrives, is sniffed, is stored content-addressed, and is prepared for inference. The describing half is written and tested against a fake server, and refuses any endpoint that is not on this box. internal/media is the intake all three senses share — hearing and speaker recognition store their audio in the same place under the same retention. Blobs stay out of the sqlite store; only the derived text becomes a note, and only when the caller asks. Retention is enforced by an hourly prune loop rather than by a comment. The plan's RemoteProvider step is refused: no cloud model, inference stays on the box, and vision.NewLocal validates that at construction.
This commit is contained in:
@@ -176,6 +176,51 @@ type IngestMailResp struct {
|
||||
Skipped bool `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
// DescribeImageReq — one image handed to core to look at (Vikunja #252).
|
||||
//
|
||||
// Data is the raw image file as received (png / jpeg / gif). Core sniffs it and
|
||||
// refuses anything else; a declared content type is not part of this request
|
||||
// because the sender's claim about its own bytes is not evidence. Base64 on the
|
||||
// wire via the usual JSON marshal of []byte.
|
||||
//
|
||||
// Question is what he asked about the picture ("что тут написано?"). Empty ⇒
|
||||
// core uses its configured default prompt.
|
||||
//
|
||||
// Source is provenance recorded on the stored blob: "telegram", "web:upload".
|
||||
//
|
||||
// Exactly one of Data or ID is set. ID re-describes an image core already has —
|
||||
// a different question, or the first attempt that succeeds after a vision model
|
||||
// finally lands on disk.
|
||||
//
|
||||
// The method exists only when core has both a media store and an enabled vision
|
||||
// block; otherwise it answers ErrUnknownMethod, which is what "off unless
|
||||
// configured" looks like at the wire. A surface cannot make Maven look at
|
||||
// pictures by merely sending one.
|
||||
type DescribeImageReq struct {
|
||||
Data []byte `json:"data,omitempty"`
|
||||
ID string `json:"id,omitempty"`
|
||||
Source string `json:"source,omitempty"`
|
||||
Question string `json:"question,omitempty"`
|
||||
// SaveNote — also write the description as a note (source
|
||||
// "media:image:<id-prefix>") so it is recallable later. Default false: a
|
||||
// glance at a screenshot is not automatically a memory.
|
||||
SaveNote bool `json:"save_note,omitempty"`
|
||||
}
|
||||
|
||||
// DescribeImageResp — what she saw. ID is the stored blob's content address, and
|
||||
// it is set even when Description is empty because the description failed: the
|
||||
// bytes are on disk and the same id can be retried. NoteID is non-zero only when
|
||||
// SaveNote was set and the write succeeded.
|
||||
//
|
||||
// The image itself is never echoed back.
|
||||
type DescribeImageResp struct {
|
||||
ID string `json:"id"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Width int `json:"width,omitempty"`
|
||||
Height int `json:"height,omitempty"`
|
||||
NoteID int64 `json:"note_id,omitempty"`
|
||||
}
|
||||
|
||||
// SwapModelReq — load another resident model without restarting the daemon
|
||||
// (Vikunja #250). ModelPath must be one of the paths in phraser.swap_models;
|
||||
// anything else is ErrForbidden, and an unconfigured allowlist makes the whole
|
||||
|
||||
Reference in New Issue
Block a user