1182 lines
54 KiB
Markdown
1182 lines
54 KiB
Markdown
# Recommendation & Identity v2
|
||
|
||
This document retires the v1 recommendation engine; it does not tune it.
|
||
Where the old `05-recommendation-spec.md` shipped a single CTE inside
|
||
`db.service.ts:getNextVibeChunk` (line ~715) that simultaneously did
|
||
candidate generation, scoring, evaluation, exploration policy, and
|
||
session composition, v2 splits that loading into five cooperating
|
||
systems. The CTE stays untouched (and buggy) until System D lands,
|
||
then is deleted.
|
||
|
||
The two philosophy docs — *Music Intelligence System* and *Discovery
|
||
Pipeline / Session Director* — describe the shape. This doc is the
|
||
engineering plan: schema, write paths, read paths, acceptance, and an
|
||
explicit *replaces* list per system.
|
||
|
||
### What survives from the old v2 plan
|
||
|
||
- **Phase 4 — Image candidates** (`image_candidates` table). Preserved
|
||
verbatim in §F below. Orthogonal to recommendation; the bad-image
|
||
problem is a provenance problem, unrelated to the engine.
|
||
- **§1.1's intent** (the engine must learn from plays, not from a Keep
|
||
button the user does not press) — re-implemented under System B as
|
||
evidence rows feeding per-profile beliefs, not as a
|
||
`feedback(action='promoted')` row.
|
||
|
||
### What is retired by this plan
|
||
|
||
The old v2 **Phases 1, 2, 3, 5** are subsumed and displaced:
|
||
|
||
| Old phase | Retired by | Why |
|
||
|---|---|---|
|
||
| Phase 1 (engine tuning: recency, overplay, same-art, cap) | System D | All four are symptoms of doing the session director's job inside a scorer. In D they stop being tuning constants and become structural consequences of fatigue + budgets. |
|
||
| Phase 1 §1.1 (implicit promote) | System B | `feedback(action='promoted')` is the wrong shape; evidence rows under per-profile beliefs are the right shape. |
|
||
| Phase 2 (`album_artists` junction) | System A | A single probabilistic claims graph subsumes album ownership, MB credit, and identity groups as predicate triples. No separate `album_artists` table. |
|
||
| Phase 3 (MB authoritative credit, re-credit pass) | System A | MB is the structural *spine* (it seeds high-trust claims), not the *truth*. Re-credit pass becomes "fetch MB claims into the graph"; no destructive overwrite of `track_artists`. |
|
||
| Phase 5 (`artist_groups`) | System A | `alias_of` is a continuous belief (`P(DOOM ≡ Madvillain)`), not a curated flag table. |
|
||
|
||
### Architectural principles
|
||
|
||
1. **Music is a graph of entities + probabilistic claims**, not a
|
||
folder of files or a table of flat similarity rows.
|
||
2. **Truth is probabilistic fusion.** Every claim is evidence, not
|
||
fact. Conflicts coexist; resolution happens at read time, weighted by
|
||
source trust and recency.
|
||
3. **MusicBrainz is the structural spine** (MBIDs + credit bands as
|
||
high-trust claim seeds), never the truth by decree. When MB and tags
|
||
disagree, both claims live in the graph with different trust weights.
|
||
4. **Aliasing is continuous** and evolves with listening behavior. A
|
||
`alias_of` claim is a belief with a confidence value, reinforced
|
||
when the listener plays both aliases back-to-back in a session.
|
||
5. **Listener identity is multidimensional**, keyed on `user_id` from
|
||
the start (single user today, multi-user tomorrow — no retrofit).
|
||
Multiple profiles coexist: long-term, current obsession, discovery,
|
||
negative, forgotten, contextual.
|
||
6. **Sessions are directed, not scored.** The objective is the best
|
||
next *hour*, not the best next *track*. Fatigue, diversity budgets,
|
||
arcs, surprise, callbacks, and an entropy target all live in a
|
||
planner that re-plans continuously.
|
||
7. **Discovery is autonomous and separate from playback.** Acquisition
|
||
writes candidates into the graph; probation is a state on those
|
||
candidates; the session director consumes probation-tracked tracks
|
||
without knowing they are probation.
|
||
8. **Everything decays unless reinforced.** Beliefs, claims' confidence,
|
||
and fatigue all weaken over time. This prevents permanent historical
|
||
bias and keeps the system learning the *current* listener, not the
|
||
listener of two years ago.
|
||
|
||
---
|
||
|
||
## The five systems
|
||
|
||
```
|
||
A Knowledge graph (probabilistic fusion)
|
||
│
|
||
┌────┴────┐
|
||
B E
|
||
Listener Acquisition
|
||
model pipeline
|
||
│
|
||
C Candidate generators
|
||
│
|
||
D Session director
|
||
```
|
||
|
||
- **A** is the foundation; no other system can run without it.
|
||
- **B** and **E** depend only on A and may be built in parallel.
|
||
- **C** depends on A (graph traversal) and reads B (profiles inform
|
||
generator selection, e.g. revival generator reads the forgotten
|
||
profile, discovery generator reads the discovery profile).
|
||
- **D** depends on C (needs generators to populate the plan) and B
|
||
(needs listener state from beliefs); it is the final piece.
|
||
- **Phase 4 (image candidates)** ships any time, independent of A–E.
|
||
|
||
Critical path: **A → {B, E} → C → D**.
|
||
|
||
---
|
||
|
||
## System A — Knowledge graph (probabilistic fusion)
|
||
|
||
Maintains the connected entity graph with probabilistic relationships.
|
||
Never contains user-specific knowledge (objective claims carry
|
||
`user_id = NULL`); listener-behavior-derived claims carry `user_id`
|
||
and fuse into the per-user graph view at read time.
|
||
|
||
### A.1 Schema
|
||
|
||
```sql
|
||
-- Source trust weights. One row per source of claims. Tunable.
|
||
CREATE TABLE source_trust (
|
||
key TEXT PRIMARY KEY, -- 'mb' | 'discogs' | 'lastfm' | 'tag' | 'listener_behavior' | 'curated'
|
||
trust REAL NOT NULL CHECK (trust >= 0 AND trust <= 1.0),
|
||
description TEXT NOT NULL
|
||
);
|
||
|
||
-- Claims: the spine of the graph. One row per (subject, predicate, object, source).
|
||
-- user_id is NULL for objective claims (MB, Discogs, tags), non-NULL for
|
||
-- listener-behavior-derived claims (e.g. weak alias_of from adjacent plays).
|
||
CREATE TABLE claims (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
user_id UUID, -- NULL = objective
|
||
subject_type TEXT NOT NULL, -- 'artist'|'album'|'track'|'label'|'genre'|'scene'
|
||
subject_id UUID NOT NULL,
|
||
predicate TEXT NOT NULL, -- see A.2
|
||
object_type TEXT NOT NULL,
|
||
object_id UUID NOT NULL,
|
||
source TEXT NOT NULL REFERENCES source_trust(key),
|
||
confidence REAL NOT NULL DEFAULT 1.0 CHECK (confidence >= 0 AND confidence <= 1.0),
|
||
evidence_at TIMESTAMPTZ NOT NULL, -- when the source asserted this
|
||
last_reinforced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
raw JSONB, -- original payload for audit
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
UNIQUE (subject_type, subject_id, predicate, object_type, object_id, source, user_id)
|
||
);
|
||
CREATE INDEX claims_subject_idx ON claims (subject_type, subject_id, predicate);
|
||
CREATE INDEX claims_object_idx ON claims (object_type, object_id, predicate);
|
||
CREATE INDEX claims_user_idx ON claims (user_id) WHERE user_id IS NOT NULL;
|
||
|
||
-- Every entity that participates in the graph carries an optional MBID as
|
||
-- the structural spine anchor. These columns already exist on artists/albums
|
||
-- today; we add to tracks. MBID presence raises the entity's identity
|
||
-- resolution priority (see A.5).
|
||
ALTER TABLE tracks ADD COLUMN IF NOT EXISTS recording_mbid UUID;
|
||
CREATE INDEX IF NOT EXISTS tracks_recording_mbid_idx ON tracks (recording_mbid) WHERE recording_mbid IS NOT NULL;
|
||
```
|
||
|
||
### A.2 Predicates
|
||
|
||
The enumerated set. Adding a predicate is a code change (a generator
|
||
or fusion view that reads it), not a schema migration — predicates
|
||
live in the `claims.predicate` free-text column, validated in code.
|
||
|
||
| Predicate | subject → object | Meaning |
|
||
|---|---|---|
|
||
| `credited_main_on` | artist → track | artist is the main credit on the recording |
|
||
| `featured_on` | artist → track | artist is a featured performer on the track |
|
||
| `credited_main_on_album` | artist → album | artist is the main credit on the album as a whole |
|
||
| `featured_on_album` | artist → album | artist is a co-owner / featured credit on the album |
|
||
| `alias_of` | artist → artist | subject is an alias of object (directional; confidence = belief) |
|
||
| `member_of` | artist → artist | subject is a member of the group object |
|
||
| `produced` | artist → track | subject produced the track |
|
||
| `composed` | artist → track | subject composed the track |
|
||
| `same_label_as` | artist → artist | both artists release on the same label |
|
||
| `same_scene_as` | artist → artist | both artists belong to the same scene |
|
||
| `influences` | artist → artist | subject influenced object |
|
||
| `remix_of` | track → track | subject is a remix of object |
|
||
| `cover_of` | track → track | subject is a cover of object |
|
||
| `soundtrack_contrib` | artist → franchise | subject contributed to a soundtrack (anime/game/film) |
|
||
| `belongs_to_genre` | track/artist → genre | subject belongs to genre (weighted; replaces exact-id match) |
|
||
|
||
### A.3 Source trust seed
|
||
|
||
```sql
|
||
INSERT INTO source_trust (key, trust, description) VALUES
|
||
('curated', 1.00, 'Manual / human-curated claim. Never decayed.'),
|
||
('mb', 0.90, 'MusicBrainz structural spine. High-trust seed; not infallible (re-credits disagree with tags).'),
|
||
('cover_art_archive',0.85,'Cover Art Archive, MB-backed.'),
|
||
('discogs', 0.75, 'Discogs release/artist credits. Strong for releases, weaker for person aliases.'),
|
||
('lastfm', 0.50, 'Last.fm tags + similar. Noisy; used as weak signal.'),
|
||
('listener_behavior',0.40, 'Derived from observed play patterns (e.g. back-to-back play → weak alias_of). User-keyed.'),
|
||
('tag', 0.30, 'File-tag-derived via scanner heuristic. Lowest trust; the &-split fallback.')
|
||
ON CONFLICT (key) DO NOTHING;
|
||
```
|
||
|
||
### A.4 Fusion read path
|
||
|
||
Truth is resolved at read time as a weighted vote across all claims
|
||
for a given `(subject, predicate, object)`. The fusion formula:
|
||
|
||
```
|
||
fused(subject, pred, object, user_id) =
|
||
Σ over claims c with matching (subject, pred, object)
|
||
where c.user_id IS NULL OR c.user_id = $user_id
|
||
of source_trust(c.source) * c.confidence * recency(c)
|
||
```
|
||
|
||
where `recency(c) = clamp(0.1, 1.0, days_since(c.last_reinforced_at) / 180)`,
|
||
so a claim never reinforced for 180+ days contributes at 10% floor.
|
||
|
||
A materialised view `claim_fusion` exposes the per-(subject, pred,
|
||
object, user) fused value. **View shims** over `claim_fusion` provide
|
||
the v1 shapes so existing reads survive the transition without rewrite:
|
||
|
||
```sql
|
||
CREATE OR REPLACE VIEW track_artists_v2 AS
|
||
SELECT t.id AS track_id,
|
||
a.id AS artist_id,
|
||
CASE cf.predicate WHEN 'credited_main_on' THEN 'main' ELSE 'featured' END AS role,
|
||
cf.fused_value AS confidence
|
||
FROM tracks t
|
||
JOIN claim_fusion cf
|
||
ON cf.subject_type = 'track' AND cf.subject_id = t.id
|
||
AND cf.predicate IN ('credited_main_on','featured_on')
|
||
AND cf.object_type = 'artist'
|
||
JOIN artists a ON a.id = cf.object_id;
|
||
|
||
-- Same shape for albums. Drops albums.artist_id reads over time.
|
||
CREATE OR REPLACE VIEW album_artists_v2 AS
|
||
SELECT al.id AS album_id,
|
||
a.id AS artist_id,
|
||
CASE cf.predicate WHEN 'credited_main_on_album' THEN 'main' ELSE 'featured' END AS role,
|
||
cf.fused_value AS confidence
|
||
FROM albums al
|
||
JOIN claim_fusion cf
|
||
ON cf.subject_type = 'album' AND cf.subject_id = al.id
|
||
AND cf.predicate IN ('credited_main_on_album','featured_on_album')
|
||
AND cf.object_type = 'artist'
|
||
JOIN artists a ON a.id = cf.object_id;
|
||
```
|
||
|
||
`artists.artist_similar` is retired in favour of the graph itself; a
|
||
compatibility view maps `same_scene_as` + `alias_of` fused edges to
|
||
the old `(artist_id, similar_artist_id, match)` shape for the lifetime
|
||
of any read that still wants it.
|
||
|
||
Genre hierarchy (`genre.parent_id`) is retired as a separate column.
|
||
`belongs_to_genre` claims carry a `confidence` weight; the hierarchy
|
||
becomes a `parent_of` claim series on genre entities, and the fusion
|
||
view's hierarchical rollup walks these claims instead of a column.
|
||
|
||
### A.5 Write paths
|
||
|
||
Five independent writers; all UPSERT into `claims`:
|
||
|
||
1. **MB spine writer** (in the worker enrichment pipeline): when
|
||
`lookupRecording` resolves a recording MBID, fetch the full
|
||
`artist-credit` (not just the first entry — extend the MB client).
|
||
Write one `credited_main_on` claim (artist = first credit) and one
|
||
`featured_on` claim per additional credit, with `source='mb'`,
|
||
`confidence=1.0`, `evidence_at=NOW()`. For release-group MBIDs on
|
||
albums, write `credited_main_on_album` / `featured_on_album`
|
||
analogously. For artist-relation ARs (`member of`, ` collaborations`,
|
||
`vocal`/`instrument`), write `member_of` / `featured_on` claims.
|
||
MBIDs anchor identity: if the credited artist resolves to a known
|
||
MBID, claims attach to that artist entity; otherwise a new artist
|
||
is created with the MBID set.
|
||
2. **Discogs writer**: `discogs_id` already set on artists today;
|
||
extend to write `credited_main_on_album` / `same_label_as` claims
|
||
from the release's label and artist credits, `source='discogs'`,
|
||
`confidence=0.7`.
|
||
3. **Last.fm writer**: existing `artist_similar` fetcher writes
|
||
`same_scene_as` claims (`source='lastfm'`, `confidence=match/100`)
|
||
instead of the `artist_similar` table. Last.fm tags write
|
||
`belongs_to_genre` claims with `confidence=tag.count/100`.
|
||
4. **Tag-derived writer** (scanner fallback): when no MBID, the
|
||
existing `parseArtistsFromMetadata` heuristic writes
|
||
`credited_main_on` / `featured_on` claims with `source='tag'`,
|
||
`confidence=1.0` (the trust weight, low at 0.30, is what dims it).
|
||
`resolveOrCreateArtist` writes the entity row; the claim carries
|
||
the entity, not the name string. Re-keying when MB later resolves
|
||
is an UPSERT, not an overwrite.
|
||
5. **Listener-behavior writer** (writes into the *user-keyed* region
|
||
of `claims`): on play sessions, derive weak edges — adjacent plays
|
||
within 30 min of two artists write `same_scene_as` (confidence
|
||
0.3); back-to-back play of two artists within a session writes
|
||
`alias_of` (confidence 0.2). These are *evidence*, not conclusions;
|
||
they fuse with the objective `alias_of` claim from MB (if any) at
|
||
read time. Behavioral claims decay by `last_reinforced_at`; the
|
||
belief-strengthening path in System B reinforces them on repeat.
|
||
|
||
All writes UPSERT on `(subject, predicate, object, source, user_id)`:
|
||
re-fetching a source refreshes `last_reinforced_at` and `evidence_at`
|
||
without duplicating rows.
|
||
|
||
### A.6 Replaces
|
||
|
||
| Old surface | Status |
|
||
|---|---|
|
||
| `artist_similar` table | Retired; replaced by `same_scene_as` / `alias_of` claims. Compat view for the transition. |
|
||
| `track_artists` (as truth) | Retired as truth; survives as a *view* `track_artists_v2` over `claim_fusion`. No insert path; reads only. |
|
||
| `albums.artist_id` (single-FK ownership) | Survives as a denormalised pointer (written by a trigger off `claim_fusion`'s main credit) for back-compat. Truth lives in `album_artists_v2` view. |
|
||
| `genre.parent_id` (column) | Retired; replaced by `parent_of` claims on genre entities. |
|
||
| Scanner `resolveAlbumArtist` `parts[0]` behaviour | The heuristic now writes *claims*, not truth. `parts[1:]` become `featured_on_album` claims instead of being dropped. |
|
||
| MB client `best['artist-credit']?.[0]` first-credit-only read | Extended to read the full `artist-credit` array. |
|
||
| Old Phase 2 (`album_artists` junction table) | Not built — subsumed by A. |
|
||
| Old Phase 3 (re-credit pass over whole library) | Not built as a destructive overwrite. MB claims UPSERT into the graph; no `track_artists.source` column, no destructive re-credit. |
|
||
| Old Phase 5 (`artist_groups`, `artist_group_members`) | Not built — `alias_of` / `member_of` claims are the grouping. |
|
||
|
||
### A.7 Acceptance
|
||
|
||
- A track tagged `artist="MF DOOM & Madlib"` with a resolved recording
|
||
MBID shows a `credited_main_on` claim from MB for "Madvillain" (if MB
|
||
credits Madvillain) AND a `credited_main_on` claim from the tag for
|
||
"MF DOOM". Both coexist. The fusion view, weighted by trust (MB 0.90
|
||
> tag 0.30), shows "Madvillain" as the higher-confidence main credit.
|
||
- An album tagged `albumartist="A & B"` has two `credited_main_on_album`
|
||
claims — A from tags, A and B from MB if MB credits both. The album
|
||
appears on both A's and B's artist pages via `album_artists_v2`.
|
||
- `SELECT * FROM claims WHERE subject_type='artist' AND subject_id=$doom
|
||
AND predicate='alias_of'` returns rows from MB (if it asserts an
|
||
alias) and from listener_behavior (if the user has played DOOM and
|
||
Madvillain back-to-back). Both decay; both reinforce.
|
||
- A never-played genre fished via the future `/vibe/from-genre` path
|
||
reaches tracks through the `belongs_to_genre` claims + hierarchical
|
||
rollup, not exact-id match.
|
||
|
||
### A.8 Risks
|
||
|
||
- **Fusion reasoning cost.** Every read now resolves a vote across
|
||
multiple claims. Mitigation: `claim_fusion` materialised view,
|
||
refreshed on `claims` insert/update; reads hit the view, not the raw
|
||
table. Estimate: a single `(subject, predicate, object)` fused value
|
||
is a point lookup on the materialised view.
|
||
- **MB rate limits during spine backfill.** Initial population walks the
|
||
whole library (~3.9k tracks) fetching full `artist-credit`. Expected
|
||
hours, not minutes, gated by MB's rate policy.
|
||
- **Display flips during transition.** When the fusion view's
|
||
high-confidence main credit is "Madvillain" but the file tag says
|
||
"MF DOOM & Madlib", the library view shows "Madvillain". This is
|
||
intended (MB is the structural spine) but may surprise the user at
|
||
first. Both claims remain auditable via `SELECT * FROM claims`.
|
||
- **Tag-only tracks** (no MBID) inherit the lowest-trust claims. This
|
||
is correct: the graph is honest about how much it knows.
|
||
|
||
---
|
||
|
||
## System B — Listener model
|
||
|
||
Maintains probabilistic beliefs about the listener. Keyed on
|
||
`user_id` from the start; behaviours and beliefs cannot be retrofitted
|
||
later without painful re-keying once belief data accumulates.
|
||
|
||
### B.1 Schema
|
||
|
||
```sql
|
||
-- Evidence: every observed interaction that should influence a belief.
|
||
-- Append-only. Never edited or deleted (purge policy separate).
|
||
CREATE TABLE evidence (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
user_id UUID NOT NULL,
|
||
entity_type TEXT NOT NULL, -- 'track'|'artist'|'genre'|'album'
|
||
entity_id UUID NOT NULL,
|
||
signal TEXT NOT NULL, -- see B.3
|
||
profile TEXT NOT NULL, -- which profile this evidence feeds; see B.2
|
||
weight REAL NOT NULL, -- signal strength, set by the signal rule
|
||
context JSONB, -- optional: {session_id, hour, weekday, activity, ...}
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||
);
|
||
CREATE INDEX evidence_user_entity_idx ON evidence (user_id, entity_type, entity_id, created_at DESC);
|
||
CREATE INDEX evidence_user_profile_idx ON evidence (user_id, profile, created_at DESC);
|
||
|
||
-- Listener beliefs: the derived state. Continuously decayed; reinforced by evidence.
|
||
CREATE TABLE listener_beliefs (
|
||
user_id UUID NOT NULL,
|
||
profile TEXT NOT NULL, -- see B.2
|
||
entity_type TEXT NOT NULL,
|
||
entity_id UUID NOT NULL,
|
||
dimension TEXT NOT NULL, -- 'affinity'|'fatigue'|'familiarity'|'novelty_tolerance'
|
||
value REAL NOT NULL CHECK (value >= -1.0 AND value <= 1.0),
|
||
confidence REAL NOT NULL CHECK (confidence >= 0 AND confidence <= 1.0),
|
||
evidence_count INTEGER NOT NULL DEFAULT 0,
|
||
last_reinforced_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
last_decayed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
PRIMARY KEY (user_id, profile, entity_type, entity_id, dimension)
|
||
);
|
||
CREATE INDEX listener_beliefs_user_profile_idx ON listener_beliefs (user_id, profile, entity_type, entity_id);
|
||
```
|
||
|
||
### B.2 Profiles
|
||
|
||
Enumerated. Adding a profile is a code change (an evaluator or
|
||
generator that reads it), not a schema migration.
|
||
|
||
| Profile | Decay half-life | Fed by signals | Read by |
|
||
|---|---|---|---|
|
||
| `longterm` | 365 days (slow) | replays, manual search, add-to-favorites, multiple sessions | comfort, deep-dive generators |
|
||
| `obsession` | 14 days (fast) | disproportionate play concentration | adjacent, novelty generators (to bound) |
|
||
| `discovery` | 30 days | play-of-never-before-seen-entity, accept-after-skip | discovery, experimental generators |
|
||
| `negative` | 180 days | skips, queue-removal, hide, manual-delete | all generators (exclusion) |
|
||
| `forgotten` | n/a (derived) | `longterm` high-affinity + 90d no plays | revival generator |
|
||
| `contextual` | 7 days | context-tagged play sessions | contextual generator |
|
||
|
||
`forgotten` is derived nightly from `longterm` beliefs that have not
|
||
been reinforced in 90 days; it is not written by signals directly.
|
||
|
||
### B.3 Signal → weight rules
|
||
|
||
A signal writes one `evidence` row with a `weight`. The weight feeds
|
||
into the belief update (B.4). Signal weights are constants, tunable:
|
||
|
||
| Signal | Profile | Weight | Direction |
|
||
|---|---|---|---|
|
||
| `playback_completed` | longterm | +0.10 | affinity up |
|
||
| `replay_within_24h` | longterm | +0.25 | affinity up |
|
||
| `replay_within_24h` | obsession | +0.40 | affinity up |
|
||
| `manual_search` | longterm | +0.50 | affinity up |
|
||
| `add_to_favorites` | longterm | +0.60 | affinity up |
|
||
| `shared` | longterm | +0.70 | affinity up |
|
||
| `play_of_never_seen` | discovery | +0.05 | discovery tolerance up |
|
||
| `accept_after_probe` | discovery | +0.30 | affinity up (mild) |
|
||
| `skip_quick` (≤ 30s) | negative | -0.20 | affinity down |
|
||
| `skip_repeated` | negative | -0.40 | affinity down |
|
||
| `queue_removed` | negative | -0.30 | affinity down |
|
||
| `hidden` | negative | -0.60 | affinity down |
|
||
| `manual_deleted` | negative | -0.90 | affinity down (strong) |
|
||
|
||
Neutral actions (seek, pause, volume) write no evidence. Lack of
|
||
interaction writes no evidence (a core principle: *absence of
|
||
interaction is not dislike*).
|
||
|
||
`weight` is per-signal; an event may write multiple `evidence` rows
|
||
across multiple profiles (a `replay_within_24h` writes both a
|
||
`longterm` +0.25 row and an `obsession` +0.40 row).
|
||
|
||
### B.4 Belief update + decay
|
||
|
||
On each new evidence row matching `(user, profile, entity, dimension)`:
|
||
|
||
```
|
||
belief.value = clamp(-1, 1, belief.value + Σ new_evidence.weight * (1 - belief.confidence))
|
||
belief.confidence = clamp(0, 1, belief.confidence + 0.05)
|
||
belief.evidence_count += count(new rows)
|
||
belief.last_reinforced_at = NOW()
|
||
```
|
||
|
||
Daily decay job (or on-read lazy decay):
|
||
|
||
```
|
||
h = (NOW() - belief.last_decayed_at) / profile.halflife_days
|
||
belief.value *= 0.5 ^ h
|
||
belief.confidence *= 0.5 ^ h -- confidence also decays
|
||
belief.last_decayed_at = NOW()
|
||
```
|
||
|
||
A belief not reinforced for 3 halflives approaches zero. Old evidence
|
||
becomes irrelevant; new evidence dominates. This prevents the v1
|
||
failure mode where heavily-played artists keep winning forever.
|
||
|
||
### B.5 Replaces
|
||
|
||
| Old surface | Status |
|
||
|---|---|
|
||
| `feedback(action='promoted')` | Not written. The implicit-promote intent from old §1.1 lives as a `playback_completed` evidence row → longterm affinity up. |
|
||
| `feedback(action='disliked'|'skipped'|'deleted_permanent')` | Not written. Same signals now write `negative`-profile evidence rows. The `feedback` table is retired; existing data is backfilled into `evidence` once and then the table is dropped. |
|
||
| `favorites` table | Survives as a UI collection (the Keep button is a UI concept, separate from engine affinity). Old §1.1 explicitly kept this split; we keep it. Keep writes a `favorites` row AND a `add_to_favorites` evidence row. |
|
||
| The clamped genre-affinity term in `local_pool` | Retired. Affinity is now per-(user, profile, entity) in `listener_beliefs`; generators read beliefs directly. |
|
||
| `play_history` reads by nothing in v1 | `play_history` survives (it is the audit log of plays) but the scorer doesn't read it; the evidence writer does, once per play, converting a play row into evidence signals. |
|
||
|
||
### B.6 Acceptance
|
||
|
||
- After one completed play of a never-played track, a `longterm`
|
||
affinity belief for that track exists at `value=+0.05`, `confidence=0.05`.
|
||
- After 5 completed replays within a week, that track's `longterm`
|
||
affinity is above +0.30; the artist's affinity (rolled up from track
|
||
beliefs) is above +0.20.
|
||
- A track skipped 3× in 30 days has a `negative`-profile affinity below
|
||
-0.50. The session director's exclusion filter consults this.
|
||
- A track not played for 365 days has decayed its `longterm` affinity
|
||
to ~50% of peak; it appears in the `forgotten` derived profile.
|
||
- `SELECT value FROM listener_beliefs WHERE user_id=$1 AND profile='obsession'
|
||
AND entity_type='artist' ORDER BY value DESC LIMIT 5` returns the current
|
||
obsessions, which the session director uses to bound overplay.
|
||
|
||
---
|
||
|
||
## System C — Candidate generators
|
||
|
||
Recommendations originate from independent generators. Each proposes
|
||
candidates without knowledge of final ranking — the session director
|
||
(D) does the ranking, mixing, and session composition. Each generator
|
||
returns candidates with a **graph-path explanation** (the chain of
|
||
claims that led to this candidate), so every recommendation is
|
||
auditable.
|
||
|
||
### C.1 Generator interface
|
||
|
||
```ts
|
||
interface Generator {
|
||
id: string; // 'comfort' | 'adjacent' | 'discovery' | ...
|
||
run(ctx: GeneratorContext): Promise<Candidate[]>;
|
||
}
|
||
|
||
interface GeneratorContext {
|
||
userId: string;
|
||
listenerState: ListenerState; // from D's state builder
|
||
beliefs: BeliefReader; // reads listener_beliefs
|
||
graph: GraphReader; // reads claim_fusion (A)
|
||
profile: ProfileName; // which profile this generator prefers
|
||
recentExclusions: Set<string>; // (entityType, entityId) already churned this session
|
||
}
|
||
|
||
interface Candidate {
|
||
trackId: string;
|
||
generatorId: string;
|
||
explanation: ClaimEdge[]; // the graph path that produced this candidate
|
||
// No score. Generators don't rank; the director does.
|
||
}
|
||
interface ClaimEdge {
|
||
subjectType: string; subjectId: string;
|
||
predicate: string;
|
||
objectType: string; objectId: string;
|
||
fusedValue: number;
|
||
}
|
||
```
|
||
|
||
### C.2 The generators
|
||
|
||
Each generator wraps a graph query. All read `claim_fusion` and
|
||
`listener_beliefs`; all return `Candidate[]` with explanations.
|
||
|
||
1. **Comfort** — reads `longterm` affinity beliefs, picks tracks by
|
||
artist with affinity > +0.5, fuses with `credited_main_on` /
|
||
`featured_on` claims to find tracks by those artists. Goal:
|
||
maintain satisfaction.
|
||
2. **Adjacent** — for each seed artist in current session state, walks
|
||
1–2 graph hops: `seed → credited_main_on → track → featured_on →
|
||
artist → member_of → group → member_of → artist`. Returns tracks
|
||
by reached artists, excluding those in the comfort pool.
|
||
3. **Discovery** — picks tracks whose artists have no `longterm` /
|
||
`obsession` belief (truly unfamiliar), filtered to those with at
|
||
least one graph edge to a trusted artist (`same_scene_as`,
|
||
`same_label_as`, `produced` by a producer who produced a favourite).
|
||
Reads the `discovery` profile's `novelty_tolerance` to set how many
|
||
to return.
|
||
4. **Deep-dive** — prioritises complete albums. Picks an album owned
|
||
(via `album_artists_v2`) by an artist with `obsession` affinity and
|
||
returns overlooked tracks (those with low `familiarity` belief) in
|
||
album order. Prefers tracks with no play history.
|
||
5. **Revival** — reads the `forgotten` derived profile, returns tracks
|
||
whose `longterm` affinity is high but `last_reinforced_at` is old
|
||
(> 90 days). Time window is adaptive: nostalgia horizon scales with
|
||
how established the longterm profile is.
|
||
6. **Novelty** — queries for tracks with `release_date` in the last 60
|
||
days whose artists share a `same_label_as` / `same_scene_as` edge
|
||
with a favourite, OR a `produced` edge from a known producer. Most
|
||
recent first, gated by `discovery` profile tolerance.
|
||
7. **Experimental** — deliberately challenges current assumptions.
|
||
Finds genres with very few `longterm` beliefs of any sign (i.e.
|
||
the system is uncertain), picks tracks from those genres with the
|
||
highest network-distance from favourites. Goal: learning, not
|
||
satisfaction. Run rate is low (one track per N, configurable).
|
||
8. **Contextual** — reads the `contextual` profile. If the listener
|
||
state has a context tag (coding / driving / sleeping), returns
|
||
tracks whose `listener_beliefs` context entries match that
|
||
context.
|
||
|
||
### C.3 Replaces
|
||
|
||
| Old surface | Status |
|
||
|---|---|
|
||
| `local_pool` CTE | Retired. Comfort and adjacent generators together cover what local_pool tried to be (genre-overlap + artist_sim + same-artist + audio + jitter). |
|
||
| `probation_pool` CTE (gated on `artist_sim > 0`) | Retired. Discovery + deep-dive generators cover the intended-but-unshipped behavior. |
|
||
| `getVibeChunkFromGenre` (stateless genre seed) | Survives briefly as a thin wrapper over the discovery generator seeded with a genre; cleaned up when D lands. |
|
||
|
||
### C.4 Acceptance
|
||
|
||
- Each candidate returned by any generator carries a non-empty
|
||
`explanation` array (graph path). A recommendation with no graph
|
||
path is invalid; generators refuse to return it.
|
||
- Seeding a DOOM track: adjacent generator returns tracks by artists
|
||
reached via `featured_on` from DOOM tracks (i.e. Madlib's other
|
||
projects) and via `member_of` from DOOM (i.e. Madvillain tracks) —
|
||
*all as candidates*, with explanations; the session director decides
|
||
whether to use them given the fatigue model.
|
||
- A genre with no library coverage (seeded via `/vibe/from-genre`)
|
||
returns zero comfort candidates and a non-empty discovery candidate
|
||
list — surfacing fresh material, not the empty result of v1's
|
||
`artist_sim > 0` gate.
|
||
|
||
---
|
||
|
||
## System D — Session director
|
||
|
||
The planner. Replaces `getNextVibeChunk` entirely. Where the old CTE
|
||
selected the highest-scoring 20 tracks in one query, D maintains a
|
||
rolling 20–50-track plan that is rewritten on every feedback event,
|
||
pursuing invisible long-term goals (finish an album over days,
|
||
introduce an artist gradually, balance decades) while optimising
|
||
multiple objectives simultaneously.
|
||
|
||
### D.1 Listener state
|
||
|
||
Built at the start of each session and updated on each play/skip:
|
||
|
||
```sql
|
||
-- Per-session state; persisted across heartbeats so resumes stay coherent.
|
||
CREATE TABLE session_state (
|
||
session_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
user_id UUID NOT NULL,
|
||
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
last_interaction TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
context TEXT, -- 'coding'|'driving'|'sleeping'|NULL (auto-detected or manual)
|
||
state_vector JSONB NOT NULL -- the computed state; see below
|
||
);
|
||
```
|
||
|
||
`state_vector` fields (computed on state build, refreshed on each event):
|
||
|
||
```jsonc
|
||
{
|
||
"energy": 0.62, // avg energy of last 5 plays
|
||
"focus": 0.40, // manual focus intent, 0..1
|
||
"novelty_hunger": 0.30, // from discovery profile's novelty_tolerance
|
||
"artist_fatigue": { "<artistId>": 0.71, ... }, // see D.2
|
||
"genre_fatigue": { "<genreId>": 0.55, ... },
|
||
"language_fatigue": { "ja": 0.83, "en": 0.10 },
|
||
"vocal_fatigue": 0.40, // 0 = vocals ok, 1 = want instrumental
|
||
"session_age_min": 73,
|
||
"current_mood": "energetic",
|
||
"target_entropy": 0.55 // see D.5
|
||
}
|
||
```
|
||
|
||
### D.2 Fatigue model
|
||
|
||
Everything gets fatigued. Everything recovers over time. Fatigues are
|
||
per-dimension cumulative decays over recent play history, NOT session
|
||
counters (v1's `artist_play_count` batch counter is retired).
|
||
|
||
For each dimension X (track / artist / genre / language / vocalist):
|
||
|
||
```
|
||
fatigue_X(entity, t) = Σ over plays p of X in last T_window
|
||
of exp( -(t - p.played_at) / decay_X )
|
||
|
||
T_window = 24h (artist, genre) | 7d (track) | 2h (language, vocalist)
|
||
decay_X = 8h (artist, genre) | 30d (track) | 1h (language, vocalist)
|
||
```
|
||
|
||
- `track` fatigue: a track played in the last hour `exp(-(0)/30d)=1`;
|
||
played yesterday `exp(-(1d)/30d)≈0.97` — *strong* recent-play penalty
|
||
on tracks; this is the v1 missing cross-session overplay fix, made
|
||
structural. Played a month ago: `exp(-(30d)/30d)≈0.37`.
|
||
- `artist` fatigue: rolled up from track fatigue over the artist's
|
||
tracks; collapses MF DOOM / Madvillain / Viktor Vaughn **iff** their
|
||
`alias_of` claims fuse them at read time (which depends on whether
|
||
MB or listener-behavior has asserted the alias). This is the honest
|
||
v1 Phase 5 fix — aliasing is a graph belief, not a flag.
|
||
- `genre` / `language` / `vocalist` fatigue: same formula, same
|
||
recency-aware decay.
|
||
|
||
A candidate's final rank incorporates `1 - fatigue_X(candidate, now)`
|
||
as multipliers per dimension; the v1 `GREATEST(0.15, ...)` floor
|
||
becomes a tunable per-dimension floor in `session_floor` config.
|
||
|
||
### D.3 Diversity budgets
|
||
|
||
Instead of hard caps ("max 1 per artist per chunk", "max 2 per genre"),
|
||
a budget the planner spends. Per-session, refreshed at session start:
|
||
|
||
```sql
|
||
INSERT INTO source_trust VALUES ('budget_default', 0.0, 'non-graph config sentinel') ON CONFLICT DO NOTHING;
|
||
|
||
CREATE TABLE diversity_budgets (
|
||
user_id UUID NOT NULL,
|
||
dimension TEXT NOT NULL, -- 'artist'|'genre'|'language'|'instrumental'|'new_artist'|'favorite'
|
||
budget_share REAL NOT NULL, -- fraction of session, e.g. 0.20
|
||
horizon_min INTEGER NOT NULL, -- budget window, e.g. 30 (min)
|
||
PRIMARY KEY (user_id, dimension, horizon_min)
|
||
);
|
||
```
|
||
|
||
Default budgets (seeded on first session per user):
|
||
|
||
```jsonc
|
||
{
|
||
"artist": { "share": 0.20, "horizon": 30 },
|
||
"genre": { "share": 0.40, "horizon": 30 },
|
||
"language": { "share": 0.60, "horizon": 30 },
|
||
"instrumental": { "share": 0.10, "horizon": 30 },
|
||
"new_artist": { "share": 0.15, "horizon": 60 },
|
||
"favorite": { "share": 0.25, "horizon": 60 }
|
||
}
|
||
```
|
||
|
||
"Not more than 2 songs of the same artist in the last 30 min" becomes
|
||
a 20%-of-30min budget. The planner spends, replenishes at window edge.
|
||
If recent listening has blown a budget, the planner refuses further
|
||
spends in that dimension — the structural replacement for v1's
|
||
diversity cap.
|
||
|
||
### D.4 Arcs
|
||
|
||
The planner doesn't pick songs; it picks *arcs* and slots songs into
|
||
them. Templates:
|
||
|
||
- **Comfort arc**: known → known → adjacent → favorite.
|
||
- **Discovery arc**: favorite → similar → new → favorite.
|
||
- **Energetic arc**: medium → high → peak → cooldown.
|
||
- **Late-night arc**: soft → ambient → acoustic → slow electronic.
|
||
|
||
At session start, given the state vector and the long-term schedule
|
||
(D.7), the planner picks an arc template and fills it. The plan is a
|
||
list of "slots" (`{arc_position, role}`, e.g. `{1, "peak"}`); the
|
||
planner queries the matching generator for each slot. On replan,
|
||
remaining slots can shift arc.
|
||
|
||
### D.5 Surprise, callbacks, entropy target, anti-loop
|
||
|
||
- **Surprise budget**: ~1 per hour (configurable). Reserved slot in
|
||
the arc for a forgotten favorite, live version, cover, acoustic
|
||
version, producer side project, or old obsession. Surfaced via the
|
||
revival generator with a `surprise=true` flag.
|
||
- **Callbacks**: every N tracks the planner intentionally re-introduces
|
||
an artist / theme / energy level from earlier in the session (or
|
||
earlier session that day). Makes the session feel intentional.
|
||
- **Entropy target**: the state vector carries `target_entropy`. If
|
||
the autocorrelation of last-20 chosen-track features is too high
|
||
(predictable), entropy goes up (planner prefers experimental /
|
||
discovery candidates). If too low (chaotic), planner injects
|
||
comfort. Target is **controlled unpredictability**, not randomness.
|
||
- **Anti-loop detector**: continuously monitor the last-50 plan
|
||
choices for collapsing into a narrow graph region (same artist /
|
||
label / producer / genre / decade / BPM / mood / language). If
|
||
collapse detected, the planner forcibly expands: zero-out the
|
||
dominant dimension's budget for the next window and surge a
|
||
non-dominant generator.
|
||
|
||
### D.6 Repetition rules
|
||
|
||
Adaptive minimum-distance, not absolute "don't repeat":
|
||
|
||
```sql
|
||
CREATE TABLE repetition_rules (
|
||
user_id UUID NOT NULL,
|
||
dimension TEXT NOT NULL, -- 'track'|'artist'|'album'|'genre'|'energy'
|
||
min_distance INTEGER NOT NULL, -- adaptive; minutes
|
||
PRIMARY KEY (user_id, dimension)
|
||
);
|
||
```
|
||
|
||
Defaults: `track=2h, artist=20min, album=no immediate; spread across
|
||
hours, genre=don't-dominate, energy=smooth transitions`. All adapt:
|
||
if a listener shows high `focus` (deep work signal), distances relax
|
||
(loop tolerance up); if skipping-after-replay pattern appears,
|
||
distances tighten.
|
||
|
||
### D.7 Long-term scheduling + invisible goals
|
||
|
||
The planner also has week-scale objectives, tracked in
|
||
`session_state.state_vector.goals`:
|
||
|
||
- Finish an album over several days (track which album is "in
|
||
progress"; the deep-dive generator keeps returning its overlooked
|
||
tracks until the album is fully played).
|
||
- Introduce a new artist gradually (e.g. one track per session for a
|
||
week, escalating if survival rate is high).
|
||
- Revisit old favorites monthly.
|
||
- Balance decades, languages, producers (the budgets cover most of
|
||
this; the planner periodically nudges an under-represented decade to
|
||
surge).
|
||
- Complete discovery probation (see System E).
|
||
- Guarantee at least one surprise per hour.
|
||
|
||
The listener should never notice these goals directly.
|
||
|
||
### D.8 Don't maximise enjoyment
|
||
|
||
The planner optimises multiple objectives simultaneously:
|
||
|
||
```
|
||
maximise:
|
||
enjoyment (predicted from beliefs × relevance)
|
||
discovery (fraction of unfamiliar entities in the plan)
|
||
diversity (1 - Herfindahl index across artists in horizon)
|
||
learning (information gain on uncertain beliefs)
|
||
session coherence (arc-template adherence)
|
||
long-term freshness (entropy target met)
|
||
|
||
minimise:
|
||
fatigue (cumulative per-dimension fatigue)
|
||
repetition (autocorrelation of recent choices)
|
||
predictability (1 - entropy)
|
||
wasted discoveries (candidates surfaced then immediately skipped)
|
||
```
|
||
|
||
This is why single-objective score maximisation (the v1 approach)
|
||
collapses to "ADO, Yoasobi, ADO, Zutomayo, ADO" — those are the
|
||
"optimal" tracks by predicted enjoyment alone.
|
||
|
||
### D.9 Plan + replan loop
|
||
|
||
```
|
||
session start
|
||
↓
|
||
build state_vector (D.1)
|
||
↓
|
||
pick arc template (D.4) + target entropy (D.5)
|
||
↓
|
||
for each slot in arc:
|
||
query matching generator (C) → candidates
|
||
rank candidates across the D.8 objectives
|
||
pick winner, respecting budgets (D.3) + repetition rules (D.6)
|
||
↓
|
||
20–50 track plan
|
||
↓
|
||
playback
|
||
↓
|
||
on play / skip / manual action:
|
||
write evidence (B.3)
|
||
refresh state_vector fatigue (D.2)
|
||
if plan slot < 10 remaining OR anti-loop fires OR entropy drift > 0.2:
|
||
replan from current state
|
||
↓
|
||
loop
|
||
```
|
||
|
||
### D.10 Replaces
|
||
|
||
| Old surface | Status |
|
||
|---|---|
|
||
| `getNextVibeChunk` CTE (~220 lines in `db.service.ts`) | Retired entirely on D ship. |
|
||
| `recommendation_batch_track` exclusion set | Survives as the recent-exclusions `Set` passed to generators; no longer the source of artist-play-count. |
|
||
| `recommendation_batch` row + `seed_track_id` center-walk in `recordPlay` | Retired. The center-walk was a hack for "engine can't escape the seed neighbourhood"; D's arc + fatigue together replace it. |
|
||
| Old §1.2 (recency term added to local_pool) | Not a scored term anymore; recency is a fatigue-dimension multiplier in D.2. |
|
||
| Old §1.3 (track-level overplay penalty) | D.2's `track` fatigue, structural. |
|
||
| Old §1.4 (artist-level overplay penalty, identity-best-effort) | D.2's `artist` fatigue rolled up via alias fusion in A. Identity collapse happens *iff the claims graph says so*, not as a separate code path. |
|
||
| Old §1.5 (lower W_SAMEART) | No `W_SAMEART` to tune; the comfort generator alone handles "more of this artist" and is naturally bounded by D.3's artist budget. |
|
||
| Old §1.7 (cap on `track_artists.artist_id` not name string) | Subsumed by D.3's budgets (artist dimension). |
|
||
|
||
### D.11 Acceptance
|
||
|
||
- After 6 hours of listening, the listener is still engaged, has
|
||
discovered at least one unfamiliar but tolerable track, has not
|
||
become fatigued by any single artist / genre / language, and the
|
||
next session would still feel fresh.
|
||
- Seeding a DOOM track does not collapse the next chunks into DOOM
|
||
pseudonyms even though alias fusion may treat them as one artist —
|
||
because D.2's artist fatigue rises fast in the session, D.3's
|
||
artist budget blocks further spends, and D.4's arc pulls toward
|
||
adjacent generators (Madlib's other projects reached via graph
|
||
hops, not DOOM).
|
||
- Within a session, tracks played in the last hour do not re-appear
|
||
(track fatigue multiplier ≈ 0 after recent plays).
|
||
- Across sessions, the same top-20 does not return: track fatigue
|
||
half-life of 30 days means yesterday's plays still dampen today's
|
||
rank.
|
||
- Forgotten favorites resurface naturally (revival generator + monthly
|
||
long-term goal).
|
||
- Anti-loop detector fires when the dominant dimension's share exceeds
|
||
budget × 1.5, forcibly diversifying the next window.
|
||
|
||
---
|
||
|
||
## System E — Acquisition pipeline
|
||
|
||
The library is not the universe. E continuously searches beyond the
|
||
current collection, identifies music worth evaluating, acquires it
|
||
(via the unbuilt yt-dlp worker, `progress.md:29`), validates it, and
|
||
either permanently integrates it into the graph (A) or discards it.
|
||
Discovery is independent of playback; it writes into A.
|
||
|
||
### E.1 Discovery sources
|
||
|
||
Six independent strategies run continuously as low-priority worker
|
||
jobs:
|
||
|
||
1. **Graph exploration** — walk the graph beyond the library. For each
|
||
favourite artist (per `longterm` beliefs), follow `featured_on` /
|
||
`member_of` / `produced` / `same_label_as` / `same_scene_as` edges
|
||
to artists not in the library. Each traversal is a discovery path
|
||
candidate.
|
||
2. **Release monitoring** — monitor favourite artists, related artists
|
||
(graph adjacents), labels, and producers for new releases. New
|
||
releases become discovery candidates at high priority.
|
||
3. **Scene exploration** — discover music through communities rather
|
||
than artists: city scenes, internet communities, niche genres,
|
||
underground movements, independent labels. Avoids recommendation
|
||
loops around the same popular artists.
|
||
4. **Temporal exploration** — search different musical eras for
|
||
forgotten classics, overlooked releases, albums that became
|
||
influential years later.
|
||
5. **Relationship expansion** — instead of "people also listen to",
|
||
prefer structural relationships: same producer, same composer,
|
||
live band members, guest vocalists, touring partners, soundtrack
|
||
contributors.
|
||
6. **Curiosity exploration** — dedicated exploration budget for
|
||
unfamiliar genres, different languages, experimental music,
|
||
geographically distant scenes. Success is measured by learning,
|
||
not immediate satisfaction.
|
||
|
||
### E.2 Candidate universe
|
||
|
||
Before downloading, discoveries live as `claims` rows with
|
||
`subject_type='track'` and a special marker — they are *candidate*
|
||
tracks, not library tracks. The candidate carries the discovery
|
||
source, the relationship path that led to it, an estimated relevance,
|
||
and an explanation.
|
||
|
||
This reuses `claims` rather than a dedicated table, with a dedicated
|
||
predicate:
|
||
|
||
```sql
|
||
-- A discovery candidate is a claim: subject=track (candidate), predicate='discovery_candidate', object=source artist / scene / label.
|
||
-- The 'confidence' field is the estimated relevance; 'raw' holds the full path + explanation.
|
||
INSERT INTO claims (subject_type, subject_id, predicate, object_type, object_id, source, confidence, raw)
|
||
VALUES ('track', $candidateId, 'discovery_candidate', 'artist', $relatedArtistId, 'graph_exploration', $relevance, $pathJson)
|
||
ON CONFLICT (subject_type, subject_id, predicate, object_type, object_id, source, user_id) DO NOTHING;
|
||
```
|
||
|
||
Candidate tracks themselves are stored as stub rows in a
|
||
`discovery_candidates` table — thin rows holding the external identity
|
||
only, no library path / audio / metadata yet:
|
||
|
||
```sql
|
||
CREATE TABLE discovery_candidates (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
source TEXT NOT NULL, -- 'mb'|'discogs'|'lastfm'|'spotify'|...
|
||
external_id TEXT NOT NULL, -- MBID / discogs_id / etc.
|
||
title TEXT,
|
||
artist_credit JSONB, -- the full artist-credit array from the source
|
||
notes JSONB, -- discovery path, source-only fields
|
||
first_seen_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
last_eval_at TIMESTAMPTZ,
|
||
status TEXT NOT NULL DEFAULT 'candidate', -- 'candidate'|'acquiring'|'probation'|'retained'|'retired'
|
||
UNIQUE (source, external_id)
|
||
);
|
||
|
||
-- Keeping the graph reference:
|
||
-- discovery_candidates.id is referenced by claims rows with subject_type='track' (UUID aligns).
|
||
```
|
||
|
||
### E.3 Acquisition policy + queue
|
||
|
||
Not every candidate downloads. Decision factors:
|
||
|
||
- expected usefulness (relevance confidence from the discovery claim)
|
||
- novelty (does the listener's `discovery` profile tolerate this
|
||
territory?)
|
||
- storage budget
|
||
- artist diversity (don't acquire 10 tracks from one new artist in a
|
||
day)
|
||
- existing backlog depth
|
||
- current listener fatigue (don't acquire more of a fatigued artist)
|
||
- current exploration budget share
|
||
|
||
The decision maximises **expected information gain**, not download
|
||
count.
|
||
|
||
Priority queue:
|
||
|
||
| Priority | Source |
|
||
|---|---|
|
||
| 1 (highest) | favourite artists' new releases |
|
||
| 1 | active obsession new releases |
|
||
| 2 | adjacent artists, collaborations, graph discoveries |
|
||
| 3 (lowest) | experimental discoveries, curiosity experiments |
|
||
|
||
Downloads happen invisibly (yt-dlp worker, throttled, low bandwidth).
|
||
On successful download, the audio is scanned (existing scanner), which
|
||
writes `credited_main_on` / `featured_on` claims and a `tracks` row.
|
||
The discovery_candidate row transitions to `probation`.
|
||
|
||
### E.4 Probation
|
||
|
||
Downloaded music is never trusted immediately. Every acquisition
|
||
enters probation. During probation, the session director (D)
|
||
occasionally injects probation tracks into normal sessions — the
|
||
listener should not feel they are being tested.
|
||
|
||
Probation is a `tracks.probation_status` column (new), one short
|
||
migration:
|
||
|
||
```sql
|
||
ALTER TABLE tracks ADD COLUMN IF NOT EXISTS probation_status TEXT
|
||
DEFAULT 'retained' CHECK (probation_status IN ('probation','retained','retired'));
|
||
ALTER TABLE tracks ADD COLUMN IF NOT EXISTS probation_entered_at TIMESTAMPTZ;
|
||
CREATE INDEX tracks_probation_idx ON tracks (probation_status) WHERE probation_status = 'probation';
|
||
```
|
||
|
||
Existing library tracks default to `retained`. Newly acquired tracks
|
||
are `probation` with `probation_entered_at = NOW()`.
|
||
|
||
Each probation track accumulates evidence (B.3) over multiple
|
||
sessions — single interactions rarely provide enough. Possible
|
||
outcomes:
|
||
|
||
- **retain** — survival threshold met; probation_status → `retained`.
|
||
Associated artist's `longterm` affinity gets a small bump, the
|
||
discovery path that produced this candidate gets reinforced (a meta
|
||
signal for E.5).
|
||
- **archive** — kept on disk but hidden from normal sessions; exempt
|
||
from the planner.
|
||
- **delete** — file removed, `tracks` row marked `retired`. Library
|
||
is not an ever-growing archive.
|
||
- **ignore temporarily** — back to candidate state for re-evaluation
|
||
later; rarer path.
|
||
|
||
Probation duration adapts to confidence: a candidate discovered via a
|
||
trusted path (favourite producer's new signing) gets a longer
|
||
probation than a curiosity-experiment candidate.
|
||
|
||
### E.5 Meta-learning
|
||
|
||
The discovery system continuously evaluates itself. A periodic job
|
||
writes claims back into the graph about which *strategies* and *graph
|
||
paths* have produced long-term retainers, vs which consistently retire:
|
||
`source_trust` already tunes per-source; meta-learning additionally
|
||
tunes per-predicate-path:
|
||
|
||
- which discovery sources (graph_exploration, release_monitoring,
|
||
scene_exploration, ...) produce long-term favourites?
|
||
- which graph paths consistently fail? (e.g. `same_label_as` may be a
|
||
weak edge; down-weight it.)
|
||
- which labels repeatedly introduce successful artists?
|
||
- which exploration depth performs best?
|
||
- which experiments produce the highest information gain?
|
||
|
||
This meta-learning itself writes back as `source_trust` adjustments
|
||
and as tunable per-path-weight constants. Discovery learns how to
|
||
discover better, not just what to recommend.
|
||
|
||
### E.6 Acceptance
|
||
|
||
- A discovery candidate surfaced via "producer A produced favourite B
|
||
AND new artist C" is auditable: `SELECT raw FROM claims WHERE
|
||
subject_id=$candidateId AND predicate='discovery_candidate'` shows
|
||
the full path.
|
||
- A downloaded probation track on which the listener completed 3 plays
|
||
in its first 2 sessions transitions to `retained` automatically.
|
||
- A downloaded probation track skipped on every injection retires to
|
||
`retired` after its probation window; file is removed; library does
|
||
not accumulate indefinitely.
|
||
- The meta-learning job, run weekly, down-weights a discovery strategy
|
||
(e.g. `scene_exploration`) whose recent candidates have a <20%
|
||
retention rate, observable in `source_trust` deltas or per-path
|
||
weight constants.
|
||
|
||
### E.7 Replaces
|
||
|
||
Nothing — E is net new. It consumes A (graph) and writes back into A.
|
||
Built on top of the yet-unbuilt yt-dlp worker (`progress.md:29`),
|
||
independent of B/C/D.
|
||
|
||
---
|
||
|
||
## §F — Phase 4: Image quality (preserved from old v2)
|
||
|
||
**This section is preserved verbatim from the previous v2 doc.** It is
|
||
orthogonal to recommendation; the bad-image problem is a provenance
|
||
problem, unrelated to the engine. Ships any time, independent of A–E.
|
||
|
||
### F.1 Schema
|
||
|
||
```sql
|
||
CREATE TABLE image_candidates (
|
||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||
entity_type TEXT NOT NULL CHECK (entity_type IN ('artist','album')),
|
||
entity_id UUID NOT NULL,
|
||
source TEXT NOT NULL,
|
||
url TEXT,
|
||
width INTEGER,
|
||
verified BOOLEAN DEFAULT FALSE,
|
||
fetched_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||
UNIQUE (entity_type, entity_id, source)
|
||
);
|
||
CREATE INDEX image_candidates_entity_idx ON image_candidates (entity_type, entity_id);
|
||
```
|
||
|
||
- `artists.image_path` and `albums.artwork_id` remain as the
|
||
"currently preferred" denormalised pointer, written by the selection
|
||
step. Existing queries keep working.
|
||
|
||
### F.2 Enrichment write
|
||
|
||
Each image-fetch step (Wikidata, TheAudioDB, Fanart, iTunes, Deezer,
|
||
Discogs, Last.fm, Cover Art Archive) writes a `image_candidates` row
|
||
even on failure to find one — a "negative" row with `url=NULL` so we
|
||
don't re-fetch that source for that entity until the row is aged out.
|
||
|
||
### F.3 Selection
|
||
|
||
A selection step (worker job or enrichment sub-step) picks the
|
||
preferred URL by tier:
|
||
|
||
1. Wikidata via MBID (verified, broad coverage) — highest tier.
|
||
2. TheAudioDB via MBID.
|
||
3. Fanart via MBID.
|
||
4. Cover Art Archive (albums) / Deezer (albums) — high-res.
|
||
5. iTunes upscaled to 600 — broad coverage fallback.
|
||
6. Last.fm — last resort.
|
||
7. Wikimedia via name match — excluded (historically wrong; migrations
|
||
cleared these twice).
|
||
|
||
Tiers are a config table or constants, not magic strings in code.
|
||
Selection writes the winner into `artists.image_path` /
|
||
`albums.artwork_id`.
|
||
|
||
### F.4 Re-evaluation
|
||
|
||
- `image_candidates` rows older than `N` days (config, default 90) are
|
||
eligible for re-fetch. A periodic job re-runs enrichment for stale
|
||
candidates, replacing rows.
|
||
- The selector re-runs whenever candidates change. So if a low-tier
|
||
winner was selected and a higher-tier candidate lands later, the
|
||
preferred pointer is upgraded in place.
|
||
|
||
### F.5 Acceptance
|
||
|
||
- After re-enrichment, an artist that previously showed a 100×100
|
||
Last.fm thumbnail shows the Wikidata/TheAudioDB image instead.
|
||
- Re-running enrichment does not re-fetch sources that already returned
|
||
(negative cache).
|
||
- Selection is auditable: `SELECT * FROM image_candidates WHERE
|
||
entity_id = X` shows every candidate considered.
|
||
|
||
---
|
||
|
||
## Rollout
|
||
|
||
Recommended order, each system independently shippable; the old v1
|
||
CTE stays until D lands:
|
||
|
||
1. **A** (knowledge graph + claim_fusion view). Existing queries move
|
||
to the compat views (`track_artists_v2`, `album_artists_v2`); v1
|
||
engine keeps running against the views. MB spine backfill runs in
|
||
the background.
|
||
2. **B and E in parallel** (B listener model + evidence writer; E
|
||
acquisition pipeline + yt-dlp worker + probation). Both need only
|
||
A. The evidence writer starts converting play_history into
|
||
evidence; the existing CTE ignores evidence for now.
|
||
3. **C** (generators). Built and run in shadow mode alongside the v1
|
||
CTE — both produce chunks; the UI shows v1 chunks, but C's outputs
|
||
are logged for comparison. Generators don't replace v1 reads until
|
||
D ships.
|
||
4. **D** (session director). D switches over and the v1 CTE is
|
||
deleted in the same release. Acceptance is the session-feel test
|
||
(D.11).
|
||
5. **Phase 4 (image candidates)** at any point. Independent.
|
||
|
||
No phase is blocked except B/E on A, C on A+B, D on C+B. Phase 4 is
|
||
fully independent.
|
||
|
||
## Retiring v1 — explicit deletion list
|
||
|
||
On System D ship, this code goes:
|
||
|
||
- `db.service.ts` `getNextVibeChunk` (~220 lines).
|
||
- `recordPlay`'s center-walk (`UPDATE recommendation_batch SET
|
||
seed_track_id = $2 ...`, line ~580–590).
|
||
- The `artist_play_count` decay term (line ~829).
|
||
- The `W_SAMEART`, `W_ARTSIM`, `W_FEEDBCK`, `W_AUDIO`, `W_RANDOM`
|
||
constants + `local_pool` / `probation_pool` CTEs (lines 717–866).
|
||
- The "max 1 per artist per chunk" cap, replaced by D.3's budgets.
|
||
- `getVibeChunkFromGenre` (it becomes a thin wrapper over the discovery
|
||
generator; then collapses into D's session-by-genre entry).
|
||
- The `feedback` table write paths (`recordSkip`, `recordFeedback`,
|
||
hardDelete insert). The `feedback` table itself is dropped after
|
||
backfill into `evidence`.
|
||
- `recommendation_batch_track` exclusion set (replaced by D's
|
||
recent-exclusions set, in-memory per session).
|
||
- `albums.artist_id` single-FK pointer (kept as a denormalised
|
||
trigger-maintained column off the fusion view; removed as a *read*
|
||
source).
|
||
- `artist_similar` table (compat view retained briefly, then dropped).
|
||
- `genre.parent_id` column (replaced by `parent_of` claims; column
|
||
dropped after a backfill claims-migration).
|
||
|
||
## Out of scope
|
||
|
||
- **Filesystem reorganisation.** Confirmed out of scope. The bind is
|
||
read-only; the DB is the index; reorganising the FS inverts the
|
||
dependency in the wrong direction.
|
||
- **Auth itself.** Schemas are `user_id`-keyed from the start so no
|
||
retrofit is needed later, but building auth is a separate project
|
||
(progress.md #30).
|
||
- **A user-facing override UI for MB credits.** Competing claims
|
||
coexist in the graph; resolution at read time is weighted. A future
|
||
UI can show "per MB" vs "per tag" and let the user assert a
|
||
`curated` claim (trust 1.0) that overrides. Out of scope here.
|
||
- **Manual artist-group / alias curation UI.** `alias_of` is a graph
|
||
belief; `curated` claims (trust 1.0) override the learned belief
|
||
when human assertion is needed — but the UI to do that is separate. |