fix: serialize the per-host throttle so MusicBrainz rate limiting holds

throttle() read lastRequestAt, awaited delay(), then wrote it back — a
TOCTOU race with no mutex or queue, under concurrency: 10. Ten jobs read the
same timestamp, slept the same duration and fired in the same tick, giving
up to ~10 req/s against MusicBrainz's 1 req/s policy and risking an IP
block.

Replaced the lastRequestAt map with a per-host { lastRequestAt, tail }
limiter; each call links onto that host's promise chain, so the
read-sleep-write critical section is serialized and N concurrent callers
space out by minIntervalMs. Chain rejections are swallowed so one failure
cannot poison the queue. Per-host rather than global, so other integrations
are not starved by MusicBrainz.

Measured: 5 concurrent same-host calls at 200ms -> 802ms (previously all in
one tick); 3 distinct hosts at 1000ms -> 0ms, confirming no cross-host
starvation.

musicbrainz.client caught HttpError and returned null at all 7 catch sites,
making a rate-limited MusicBrainz indistinguishable from "no data for your
library" while every job reported success. A shared logMbFailure() now logs
429 (and 503 whose body mentions a rate limit) at error, stating results are
INCOMPLETE. The error model is otherwise unchanged.

REVIEW-2026-07-30.md finding 8.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

workers/src/integrations/http.ts

@@ -134,9 +134,24 @@ export interface RequestJsonOptions {
   timeoutMs?: number;
 }
 
-// Per-host timestamp of the last request start, used to enforce the rate limit.
-// Module-scoped so all clients sharing a host coordinate automatically.
-const lastRequestAt = new Map<string, number>();
+// Per-host rate-limit state. Module-scoped so all clients sharing a host
+// coordinate automatically.
+//
+// `tail` is the promise chain for a host: every throttle() call links onto the
+// host's current tail, so slot acquisition is strictly serialized. Without this
+// chain the old implementation was a TOCTOU race — under the worker's
+// `concurrency: 10`, ten jobs read the same `lastRequestAt`, slept the same
+// duration and fired in the same tick, i.e. ~10 req/s against MusicBrainz's
+// 1 req/s policy. Serialization is deliberately PER HOST (not global) so a slow
+// MusicBrainz queue cannot starve Last.fm, Discogs, cover art, etc.
+interface HostLimiter {
+  /** Timestamp of the last granted request slot. */
+  lastRequestAt: number;
+  /** Tail of the serialization chain; resolves when the previous waiter is done. */
+  tail: Promise<void>;
+}
+
+const hostLimiters = new Map<string, HostLimiter>();
 
 const delay = (ms: number): Promise<void> =>
   new Promise((resolve) => setTimeout(resolve, ms));
@@ -149,17 +164,43 @@ function hostOf(url: string): string {
   }
 }
 
-/** Block until at least `minIntervalMs` has elapsed since this host's last request. */
-async function throttle(host: string, minIntervalMs: number): Promise<void> {
-  const now = Date.now();
-  const last = lastRequestAt.get(host);
-  if (last !== undefined) {
-    const wait = minIntervalMs - (now - last);
-    if (wait > 0) await delay(wait);
+/**
+ * Acquire this host's next request slot, blocking until at least
+ * `minIntervalMs` has elapsed since the previously granted slot.
+ *
+ * Concurrent callers for the same host queue strictly in arrival order: each
+ * one appends to `limiter.tail` and only computes its wait once every earlier
+ * waiter has already claimed its timestamp, so N concurrent callers are spaced
+ * `minIntervalMs` apart rather than all firing at once.
+ */
+function throttle(host: string, minIntervalMs: number): Promise<void> {
+  let limiter = hostLimiters.get(host);
+  if (!limiter) {
+    limiter = { lastRequestAt: 0, tail: Promise.resolve() };
+    hostLimiters.set(host, limiter);
   }
-  lastRequestAt.set(host, Date.now());
+  const lim = limiter;
+
+  // Link onto the tail. The critical section (read lastRequestAt → sleep →
+  // write lastRequestAt) runs only after the previous waiter finished it.
+  const slot = lim.tail.then(async () => {
+    const wait = minIntervalMs - (Date.now() - lim.lastRequestAt);
+    if (wait > 0) await delay(wait);
+    lim.lastRequestAt = Date.now();
+  });
+
+  // The next caller waits for this slot. Swallow rejections on the chain itself
+  // so one failure can never poison the queue for subsequent requests.
+  lim.tail = slot.catch(() => {});
+  return slot;
 }
 
+/**
+ * Test-only handle on the per-host throttle, so its serialization can be
+ * asserted without issuing real network requests. Not used in production code.
+ */
+export const __throttleForTest = throttle;
+
 /** Parse a Retry-After header (delta-seconds or HTTP date) into ms, or null. */
 function parseRetryAfter(value: string | null): number | null {
   if (!value) return null;