kami
1e47eaca5a
The embedder moved from paraphrase-multilingual-MiniLM-L12-v2 to multilingual-e5-small. Both are 384-dimensional, so nothing in the code noticed: cosine between an old stored vector and a new query vector is noise, and recall degrades silently. So the DB now records the embedder that wrote its vectors. One value for the whole DB (migration #11, a small `meta` key/value table) rather than a column on every vector row: the backfill re-embeds every note and fact in one pass, so a per-row marker would hold the same string in every row and cost a column on two tables for nothing. The identity comes from the embedder itself via a new optional ID() method ("multilingual-e5-small@384", model file name plus dimension), so pointing the config at another model changes the string without anyone editing a constant. mavend logs a loud WARNING at startup naming both the stored and the configured embedder when they differ. Detection only — recall behaviour is unchanged. TODO(#378) in store.CheckEmbedder marks where the backfill will hook in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CGeSZxh1DCtRxmFVSYVGvJ
25 lines
760 B
Go
25 lines
760 B
Go
package router
|
|
|
|
import "testing"
|
|
|
|
func TestEmbedderIDFromModelPath(t *testing.T) {
|
|
got := modelIDFromPath("/opt/maven/models/embedder/multilingual-e5-small.onnx")
|
|
if got != "multilingual-e5-small@384" {
|
|
t.Fatalf("modelIDFromPath = %q", got)
|
|
}
|
|
// A different model file must produce a different id, even at 384 dim.
|
|
old := modelIDFromPath("/opt/maven/models/embedder/paraphrase-multilingual-MiniLM-L12-v2.onnx")
|
|
if old == got {
|
|
t.Fatal("two different models share one id")
|
|
}
|
|
}
|
|
|
|
func TestEmbedderIDIncludesDim(t *testing.T) {
|
|
if id := EmbedderID(NewHashEmbedder(1024)); id != "hash@1024" {
|
|
t.Fatalf("EmbedderID = %q", id)
|
|
}
|
|
if EmbedderID(NewHashEmbedder(1024)) == EmbedderID(NewHashEmbedder(384)) {
|
|
t.Fatal("dimension not part of the id")
|
|
}
|
|
}
|