Normalize every intake path into one event envelope (#283) #78

2026-08-01T04:05:26+02:00

claude commented

2026-08-01 04:05:26 +02:00

What changed

internal/event — the unified intake envelope from 20-07-2026-BACKLOG.md item 1:

type Event struct { Source, Kind string; EntityIDs []string; Title, Body, Priority string; OccurredAt time.Time; Payload json.RawMessage }

plus Bus, a bounded in-memory ring with subscribers. Both pure: Publish and Normalize take now as a parameter, so no clock read sits on a path a replay would drive (#284 depends on that).

Why it did not require rewriting eight callers

Every intake path in the repo already converges on three ipc.CoreAPI methods:

method	who reaches it
`WriteFact`	`POST /api/ambient`, mavcaldav, mavpoll (zenmoney + wg), `/api/signal` presence probes, RSS/crawl watermarks
`WriteNote`	the RSS poller, the page crawler, meeting transcripts, image descriptions
`CaptureTask`	the voice path, the web form, the mail reader

cmd/mavend/intake.go decorates that one interface. mavweb, mavcaldav, mavpoll, mavmaild and the in-core feed/crawl/capture/vision workers are unchanged — they call the interface they always called, and it now also narrates. The only explicit publish is in cmd/mavend/mail.go, which captures through the store directly and so is invisible to the decorator.

What it deliberately does not do

Nothing dispatches on an event. An event is a report that something arrived, never an instruction to speak — a feed item becoming a notification is the nag this repo refuses. The journal is memory-only and is never read by the tick loop, the router or delivery.

Read surface

ipc.MethodRecentEvents — AuthRead, daemon-cached exactly like TickTrace/DayPlan (the store adapter errors; a bare store cannot serve a ring).
/events in mavweb, read-only, sidebar "Intake" under Automation.

Config

intake_journal (int, internal/config): absent ⇒ 512 entries; negative ⇒ off, and then there is no decorator on the intake path at all. Not gated behind an "off unless configured" block: that rule is for capabilities that reach out, and this is a bounded in-memory log of writes core already performed.

Files

internal/event/{event,bus,event_test}.go, cmd/mavend/{intake,intake_test}.go, cmd/mavend/{main,tick,mail}.go, cmd/mavweb/{main.go,events.html,events_test.go}, internal/ipc/{wire,api,client,server,unimplemented}.go, internal/auth/policy.go, internal/config/config.go.

Verified

make build and make test (go test -race) both clean. New tests cover the envelope and ring (internal/event at 95.7% statements), the decorator's invariants — a failed write publishes nothing, a deduped capture publishes nothing, OccurredAt carries the fact's Ts rather than notice time, a nil bus means no wrapper — and the /events page, including escaping of feed-supplied titles.

Vikunja #283

## What changed `internal/event` — the unified intake envelope from 20-07-2026-BACKLOG.md item 1: ```go type Event struct { Source, Kind string; EntityIDs []string; Title, Body, Priority string; OccurredAt time.Time; Payload json.RawMessage } ``` plus `Bus`, a bounded in-memory ring with subscribers. Both pure: `Publish` and `Normalize` take `now` as a parameter, so no clock read sits on a path a replay would drive (#284 depends on that). ## Why it did not require rewriting eight callers Every intake path in the repo already converges on three `ipc.CoreAPI` methods: | method | who reaches it | |---|---| | `WriteFact` | `POST /api/ambient`, mavcaldav, mavpoll (zenmoney + wg), `/api/signal` presence probes, RSS/crawl watermarks | | `WriteNote` | the RSS poller, the page crawler, meeting transcripts, image descriptions | | `CaptureTask` | the voice path, the web form, the mail reader | `cmd/mavend/intake.go` decorates that one interface. mavweb, mavcaldav, mavpoll, mavmaild and the in-core feed/crawl/capture/vision workers are unchanged — they call the interface they always called, and it now also narrates. The only explicit publish is in `cmd/mavend/mail.go`, which captures through the store directly and so is invisible to the decorator. ## What it deliberately does not do Nothing dispatches on an event. An event is a report that something arrived, never an instruction to speak — a feed item becoming a notification is the nag this repo refuses. The journal is memory-only and is never read by the tick loop, the router or delivery. ## Read surface - `ipc.MethodRecentEvents` — AuthRead, daemon-cached exactly like `TickTrace`/`DayPlan` (the store adapter errors; a bare store cannot serve a ring). - `/events` in mavweb, read-only, sidebar "Intake" under Automation. ## Config `intake_journal` (int, `internal/config`): absent ⇒ 512 entries; negative ⇒ off, and then there is no decorator on the intake path at all. Not gated behind an "off unless configured" block: that rule is for capabilities that reach out, and this is a bounded in-memory log of writes core already performed. ## Files `internal/event/{event,bus,event_test}.go`, `cmd/mavend/{intake,intake_test}.go`, `cmd/mavend/{main,tick,mail}.go`, `cmd/mavweb/{main.go,events.html,events_test.go}`, `internal/ipc/{wire,api,client,server,unimplemented}.go`, `internal/auth/policy.go`, `internal/config/config.go`. ## Verified `make build` and `make test` (go test -race) both clean. New tests cover the envelope and ring (internal/event at 95.7% statements), the decorator's invariants — a failed write publishes nothing, a deduped capture publishes nothing, `OccurredAt` carries the fact's `Ts` rather than notice time, a nil bus means no wrapper — and the `/events` page, including escaping of feed-supplied titles. Vikunja #283

claude added 1 commit 2026-08-01 04:05:26 +02:00

Normalize every intake path into one event envelope (#283 ) 45b5e16eff

Things arrive at Maven from eight directions — a relayed Android
notification on POST /api/ambient, mail candidates from mavmaild, RSS
items, changed pages from the crawler, zenmoney and wg reads from
mavpoll, CalDAV events, presence probes, meeting transcripts and image
descriptions. Each grew its own shape and its own log line, and nothing
could answer "what came in today, from where".

internal/event is that answer: a flat source-agnostic envelope (Source,
Kind, EntityIDs, Title, Body, Priority, OccurredAt, Payload) plus a
bounded in-memory journal. Both are pure — Publish and Normalize take
`now` as a parameter, so no clock read sits on a path a replay would
drive.

Adopting it did not touch eight callers, because every intake path
already converges on three ipc.CoreAPI methods: WriteFact, WriteNote and
CaptureTask. cmd/mavend/intake.go decorates that ONE interface, so
mavweb, mavcaldav, mavpoll, mavmaild and the in-core feed/crawl/capture/
vision workers publish envelopes without knowing events exist. The lone
exception is cmd/mavend/mail.go, which captures through the store
directly and now publishes explicitly.

Nothing dispatches on an event. It is a report that something arrived,
never an instruction to speak — "a feed item appeared" becoming a
notification is the nag this repo refuses. Digestion may read the
journal later; it will still go through internal/loop's rules and the
severity/presence routing table.

Read surface: ipc.MethodRecentEvents (AuthRead, daemon-cached like
TickTrace — a bare store cannot serve a ring) and a read-only /events
page in mavweb.

Production is unchanged when nobody is watching: a nil *event.Bus makes
Publish a no-op and newIntakeAPI returns the wrapped API untouched, so
config.intake_journal < 0 leaves no decorator on the call path at all.
The default is 512 entries; the "off unless configured" rule is for
capabilities that reach out, and a bounded in-memory log of writes core
already performed reaches nowhere.

Verified: make build, make test (go test -race) both clean. New tests
cover the envelope and ring (internal/event, 95.7%), the decorator's
invariants — a failed write publishes nothing, a deduped capture
publishes nothing, OccurredAt is the fact's Ts and not notice time — and
the /events page including escaping of feed-supplied titles.

claude reviewed 2026-08-01 11:31:14 +02:00

claude left a comment

Decorating one interface instead of patching eight callers is the right seam. ipc.CoreAPI really is where every intake path converges. The nil-bus-is-a-no-op rule means a daemon with the journal off carries no decorator at all. WriteFact publishing req.Ts rather than now is the correct call and the comment explains why. CaptureTask publishing only on resp.Created is the detail that keeps a mailbox re-read from refilling the ring.

1. The ring is insertion-ordered, but the page sorts nothing and calls itself newest first.

Bus.Recent walks the ring backwards from next, so it returns publish order. events.html says "Everything that arrived, newest first" and renders OccurredAt in the when column. Those are two different orderings. The envelope's own doc comment is what makes them diverge: "OccurredAt is when the thing happened, NOT when Maven noticed it."

Walk a cold start. rss.Poller.write passes it.Published as the note ts, and a feed's first read returns twenty items spread over a week. All twenty go through intakeAPI.WriteNote in feed order. /events then shows, top to bottom, the poll:rss mark fact stamped now. Below it sit notes stamped six days ago, two days ago, and yesterday, in that order. The column walks forwards and backwards. cmd/mavweb/ambient.go:106 is the other half. It writes Ts: ev.Start. A 09:00 relay of an 18:00 meeting then sits above rows that arrived after it, carrying a future timestamp.

TestBusRecentIsNewestFirst does not catch this because it publishes with a monotonically increasing OccurredAt. Nothing in the suite publishes out of order. Either sort Recent by OccurredAt, or add a NoticedAt and order and label the page by that. The second is more honest, because the whole point of keeping OccurredAt truthful is that it is not arrival time.

2. factPriority inverts attention, and confidence is not recoverable anywhere else.

Confidence > 0 && Confidence < 1.0 means low. Two writers in the tree pass anything below 1.0. One is the ambient relay, at calendar.AmbientConfidence of 0.6. The other is mavcaldav, when it chooses to. So a relayed phone notification about a meeting lands as PriorityLow. Meanwhile feeds.go:165 writes rss:latest:<feed> with Confidence: 1.0 and lands as PriorityNormal. The bookkeeping watermark outranks the meeting.

Confidence is also gone after that mapping. Payload is nil at every one of the four publish sites, so nothing downstream can recover the 0.6. A reader cannot tell an inference from a credentialled read. That is the distinction internal/calendar went out of its way to preserve in the fact row. Carry Confidence in Payload, or add the field. The three-value priority is a display hint and should not be the only surviving trace.

Related and smaller: Confidence: 0 (unset) and Confidence: 1.0 both yield PriorityNormal. The > 0 guard makes "nobody said" and "certain" identical.

3. Bookkeeping self-writes are journalled as things that arrived.

The decorator sees every WriteFact, including the ones Maven makes about her own state:

feeds.go:165, source poll:rss, key rss:latest:<feed>, once per poll that found anything.
crawls.go:165, source poll:crawl, key crawl:hash:<name>, once per detected change. A changed page so produces two rows, the note and the hash.
ecosystem_acts.go:204, source praxis:trace, key praxis:<operation>. That is the audit trail of an act she performed, not an arrival.
quiet_toggle.go:33, source tap:voice, key quiet_hours. He toggled a setting.

The page header claims "Everything that arrived". Four of the writers on it did not arrive from anywhere. With a 512 ring and a handful of feeds, a cold start can evict real intake behind marks. Filter on Kind == "config" and Kind == "system" in the decorator, or make the page say what it lists.

Note the second-order effect on the PR 53 finding. Quiet toggles from the text path are stored with Source: "tap:voice". That mislabel used to live in a facts row nobody reads by source. It now has a page that prints the source in its own column.

4. VoidsID is dropped, so a retraction reads as an observation.

intakeAPI.WriteFact copies Source, Key, Value and Ts and ignores VoidsID. A POST /api/revert that voids the latest value for a key publishes an envelope indistinguishable from a fresh reading of that key. On a page whose whole job is "what came in", a correction rendered as an arrival is the wrong shape. Set PriorityLow and prefix the title, or put the voided id in Payload.

Smaller notes:

req.Kind (env, self, config, system) is discarded. The envelope's Kind is a different taxonomy, which is fine. But the fact's own kind is the field that would have made finding 3 a one-line filter. Payload is the natural home and is unused.
SourceKind maps an email: prefix to KindTask unconditionally. Any future WriteFact with an email:<mailbox> source is journalled as a task. The mail path today captures tasks, so this is latent, not live.
eventsTmpl was inserted between morningTmpl's doc comment and morningTmpl in cmd/mavweb/main.go. The morning comment now documents the events template.
newEventBus logs "intake journal: off" before checking anything else when cfg == nil. Harmless in the daemon, noisy in a test that passes nil.

Decorating one interface instead of patching eight callers is the right seam. `ipc.CoreAPI` really is where every intake path converges. The nil-bus-is-a-no-op rule means a daemon with the journal off carries no decorator at all. `WriteFact` publishing `req.Ts` rather than `now` is the correct call and the comment explains why. `CaptureTask` publishing only on `resp.Created` is the detail that keeps a mailbox re-read from refilling the ring. **1. The ring is insertion-ordered, but the page sorts nothing and calls itself newest first.** `Bus.Recent` walks the ring backwards from `next`, so it returns publish order. `events.html` says "Everything that arrived, newest first" and renders `OccurredAt` in the `when` column. Those are two different orderings. The envelope's own doc comment is what makes them diverge: "OccurredAt is when the thing happened, NOT when Maven noticed it." Walk a cold start. `rss.Poller.write` passes `it.Published` as the note ts, and a feed's first read returns twenty items spread over a week. All twenty go through `intakeAPI.WriteNote` in feed order. `/events` then shows, top to bottom, the `poll:rss` mark fact stamped now. Below it sit notes stamped six days ago, two days ago, and yesterday, in that order. The column walks forwards and backwards. `cmd/mavweb/ambient.go:106` is the other half. It writes `Ts: ev.Start`. A 09:00 relay of an 18:00 meeting then sits above rows that arrived after it, carrying a future timestamp. `TestBusRecentIsNewestFirst` does not catch this because it publishes with a monotonically increasing `OccurredAt`. Nothing in the suite publishes out of order. Either sort `Recent` by `OccurredAt`, or add a `NoticedAt` and order and label the page by that. The second is more honest, because the whole point of keeping `OccurredAt` truthful is that it is not arrival time. **2. `factPriority` inverts attention, and confidence is not recoverable anywhere else.** `Confidence > 0 && Confidence < 1.0` means low. Two writers in the tree pass anything below 1.0. One is the ambient relay, at `calendar.AmbientConfidence` of 0.6. The other is mavcaldav, when it chooses to. So a relayed phone notification about a meeting lands as `PriorityLow`. Meanwhile `feeds.go:165` writes `rss:latest:<feed>` with `Confidence: 1.0` and lands as `PriorityNormal`. The bookkeeping watermark outranks the meeting. Confidence is also gone after that mapping. `Payload` is `nil` at every one of the four publish sites, so nothing downstream can recover the 0.6. A reader cannot tell an inference from a credentialled read. That is the distinction `internal/calendar` went out of its way to preserve in the fact row. Carry `Confidence` in `Payload`, or add the field. The three-value priority is a display hint and should not be the only surviving trace. Related and smaller: `Confidence: 0` (unset) and `Confidence: 1.0` both yield `PriorityNormal`. The `> 0` guard makes "nobody said" and "certain" identical. **3. Bookkeeping self-writes are journalled as things that arrived.** The decorator sees every `WriteFact`, including the ones Maven makes about her own state: - `feeds.go:165`, source `poll:rss`, key `rss:latest:<feed>`, once per poll that found anything. - `crawls.go:165`, source `poll:crawl`, key `crawl:hash:<name>`, once per detected change. A changed page so produces two rows, the note and the hash. - `ecosystem_acts.go:204`, source `praxis:trace`, key `praxis:<operation>`. That is the audit trail of an act she performed, not an arrival. - `quiet_toggle.go:33`, source `tap:voice`, key `quiet_hours`. He toggled a setting. The page header claims "Everything that arrived". Four of the writers on it did not arrive from anywhere. With a 512 ring and a handful of feeds, a cold start can evict real intake behind marks. Filter on `Kind == "config"` and `Kind == "system"` in the decorator, or make the page say what it lists. Note the second-order effect on the PR 53 finding. Quiet toggles from the text path are stored with `Source: "tap:voice"`. That mislabel used to live in a facts row nobody reads by source. It now has a page that prints the source in its own column. **4. `VoidsID` is dropped, so a retraction reads as an observation.** `intakeAPI.WriteFact` copies `Source`, `Key`, `Value` and `Ts` and ignores `VoidsID`. A `POST /api/revert` that voids the latest value for a key publishes an envelope indistinguishable from a fresh reading of that key. On a page whose whole job is "what came in", a correction rendered as an arrival is the wrong shape. Set `PriorityLow` and prefix the title, or put the voided id in `Payload`. Smaller notes: - `req.Kind` (`env`, `self`, `config`, `system`) is discarded. The envelope's `Kind` is a different taxonomy, which is fine. But the fact's own kind is the field that would have made finding 3 a one-line filter. `Payload` is the natural home and is unused. - `SourceKind` maps an `email:` prefix to `KindTask` unconditionally. Any future `WriteFact` with an `email:<mailbox>` source is journalled as a task. The mail path today captures tasks, so this is latent, not live. - `eventsTmpl` was inserted between `morningTmpl`'s doc comment and `morningTmpl` in `cmd/mavweb/main.go`. The morning comment now documents the events template. - `newEventBus` logs "intake journal: off" before checking anything else when `cfg == nil`. Harmless in the daemon, noisy in a test that passes nil.

kami referenced this issue from a commit

2026-08-01 14:43:51 +02:00

event: order the journal by notice time, and keep it to what arrived

kami closed this pull request

2026-08-01 14:52:04 +02:00

kami commented

2026-08-01 14:52:58 +02:00

Landed on master. The stack was one linear chain, so #84 carried every commit from #50 up, and master now contains this branch in full. Merging this PR on its own is an empty diff, so it is closed rather than merged. The review findings for it were fixed in the 2026-08-01 pass and are on master as commits on the stack tip, not on this branch.

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: kami/Maven#78