kami
58 lines
2.9 KiB
Markdown
58 lines
2.9 KiB
Markdown
# Worker & Job Specification
|
|
|
|
This document defines the background processing architecture using **BullMQ**. Workers are responsible for CPU-intensive tasks and time-sensitive cleanup.
|
|
|
|
## 1. Job Architecture
|
|
|
|
All jobs are dispatched via the Backend API and processed by specialized Worker containers.
|
|
|
|
| Job Name | Priority | Responsibility | Trigger |
|
|
| :--- | :--- | :--- | :--- |
|
|
| `metadata_refresh` | Medium | Re-scanning files for tag/metadata changes. | Manual (`POST /api/library/reindex`) |
|
|
| `artwork_download` | Low | Fetching covers from Cover Art Archive/Discogs. | On metadata enrichment or new file. |
|
|
| `lyrics_download` | Low | Fetching synced lyrics (LRCLib). | On metadata enrichment. |
|
|
| `recommendation_gen` | Low | Calculating new "Vibe" seeds and batching. | On session start or after playback completion. |
|
|
| `audio_analysis` | High | Running `essentia` for BPM, Key, Energy. | New file/re-index. |
|
|
| `filesystem_rescan` | High | Reconciling DB with actual disk state. | Scheduled (Daily/Weekly). |
|
|
| `cleanup_sweep` | Medium | Managing the Dislike Lifecycle/Deletion. | Scheduled (Hourly). |
|
|
|
|
## 2. Detailed Job Workflows
|
|
|
|
### **A. Audio Analysis Pipeline (`audio_analysis`)**
|
|
This is the most resource-intensive job.
|
|
1. **Input:** `track_id`.
|
|
2. **Process:**
|
|
* Spin up `essentia` subprocess.
|
|
* Extract BPM, Key, Energy, and Melodic features.
|
|
3. **Output:** Update `track_audio_features` table and mark `audio_features_ready = true`.
|
|
|
|
### **B. Metadata Enrichment Pipeline (`metadata_refresh` / `artwork_download`)**
|
|
Triggered when a new file is detected or a re-index occurs.
|
|
1. **Input:** `track_id`.
|
|
2. **Process:**
|
|
* Lookup `mbid` via MusicBrainz.
|
|
* Fetch lyrics via LRCLib.
|
|
* Fetch artwork via Cover Art Archive.
|
|
3. **Output:** Update `tracks`, `artists`, and `albums` tables.
|
|
|
|
### **C. The "Consistency" Worker (`filesystem_rescan`)**
|
|
Ensures the database is an accurate reflection of the disk.
|
|
1. **Process:**
|
|
* Walk through `/mnt/hdd1/media/Music`.
|
|
* Compare `mtime` and `size` against DB.
|
|
* **Action:** If a file is missing, set `track.state = 'MISSING'`. If a new file is found, trigger `metadata_refresh`.
|
|
|
|
### **D. The "Cleanup" Worker (`cleanup_sweep`)**
|
|
Handles the temporal logic of the dislike lifecycle.
|
|
1. **Process:**
|
|
* Check `dislikes` where `state = 'warned'` and `warned_at < now - 24h`.
|
|
* Trigger physical file deletion and DB row removal.
|
|
* Check `dislikes` where `state = 'hidden'` and `disliked_at < now - 48h`.
|
|
* Trigger `ntfy` notification.
|
|
|
|
## 3. Error Handling & Retries
|
|
|
|
- **Exponential Backoff:** All external API jobs (MusicBrainz, etc.) must use exponential backoff to respect rate limits.
|
|
- **Dead Letter Queue (DLQ):** Jobs that fail after 5 retries are moved to a DLQ for manual inspection via the Admin Dashboard.
|
|
- **Idempotency:** All jobs must be idempotent. Running `audio_analysis` twice on the same `track_id` must not create duplicate data or errors.
|