A non-2xx reply was read in full first and only then rejected. Two costs
followed. A 500 with a large error page pulled up to MaxBytes off the wire for
nothing. An error page over the cap returned ErrTooLarge, which names the size
and hides the status the server actually sent.
The status is a typed error now. webfetch.StatusError carries the code and
unwraps to ErrStatus, so errors.Is keeps working and errors.As reads the number.
crawl.StatusError is the same shape on the other side of the seam, and
cmd/mavend/crawls.go carries the code across.
That removes the string grep in crawl.isServerError, which decided whether a
failed robots.txt blocks a crawl by looking for " 50" in an error message it did
not own. A reworded error would have turned a 503 robots.txt into permission to
crawl. It reads the code now.
Two comments corrected. webfetch.HostMatches said the crawler calls it and
nothing outside the package does. rss.PlainText said the crawler's extractor
goes through it and crawl/extract.go has its own pass.
The rss poller parses the feed straight off the byte slice instead of copying a
document that can run to a megabyte through a string.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
docs/plans/06-mcp-support.md asks for the host direction — Maven connects OUT
to MCP servers and consumes what they offer. This is the client half: the
protocol, the transports, the connection manager, the config block. Nothing is
wired into a turn yet, and nothing here exposes Maven's own capabilities to an
outside caller.
internal/mcp:
- hand-rolled JSON-RPC 2.0 (the wire format is four fields, and the repo
vendors its deps, so a library would cost more than it saves);
- two transports: a stdio subprocess on this box, and streamable HTTP, which
accepts a plain JSON reply or an SSE frame because servers disagree about
which they send;
- Client: initialize handshake, tools/list, tools/call, resources/list,
resources/read. Text content only — everything downstream is a sentence;
- Manager: lazy dial, per-server failure that never blocks boot or the other
servers, backoff reconnect, Status for a web surface, graceful Close;
- the allowlist encoding: a discovered tool becomes the store row
"vikunja_list_tasks" with cmd ["mcp","vikunja","list_tasks"], scope
"mcp:vikunja". No new column, no migration, and ProposeTool, EnableTool,
the act matcher and the confirm turn all keep working untouched.
Constraints held, in code rather than in prose:
- OFF unless configured, and a server is dark until "enabled": true.
- A url server goes through internal/webfetch, so the SSRF guard, the size
cap, the redirect cap and the per-host rate limit apply. Reaching loopback
needs allow_private on THAT server, and each server gets its own fetcher so
one loopback exemption cannot become a hole for a public endpoint.
- readOnlyHint decides destructive: no hint means "assume it mutates", which
will route the call through the existing confirm turn. Guessing wrong in
that direction only costs a question.
- The catalogue stays small on purpose — allow_tools, and max_tools=12 per
server. The resident model is a 1.7B with a 4096-token context; a tool name
it half-remembers is a wrong act.
- Only the tool name and the router's arguments are sent. There is no API
here through which a note, a fact or the persona block could travel.
webfetch grows Post (JSON-RPC cannot be a GET) and surfaces response headers
for Mcp-Session-Id. It shares Get's guards exactly: a body buys a caller
nothing, a POST to the LAN is refused for the same reason a GET is.
Verified against the real Vikunja MCP server on homesrv
(http://localhost:9100/mcp): handshake, three discovered tools with update_task
correctly NOT read-only, a live list_projects call, a tool excluded by
allow_tools refused, and the same server refused outright once allow_private
was dropped. Tests cover both transports (the stdio one against a real
subprocess), SSE and JSON framing, session echo, reconnect, and the config
validation.
internal/rss parses RSS 2.0 and Atom, and polls each configured feed on its own
interval; internal/webfetch is the one door either of them uses to touch the
network. The poller writes items as notes with source "rss:<feed>" and nothing
else: the answer path reads them back when he asks "что нового в лентах?", and
nothing is announced on arrival. A feed that dispatched would be a nag, which is
why the plan's breaking-news rule was left out rather than built.
webfetch is where the limits live, as code rather than a paragraph: http(s)
only, an allowlist (the configured feeds' hosts) and a denylist, a 2 MiB body
cap, a 3-redirect cap, one request per host per second, and a refusal to connect
to any private address — checked in the dialer's Control hook so it holds for
every resolved address and every redirect hop, not just for a literal IP.
Off unless configured: no "feeds" block, no poller, no outbound request. How far
a feed was read is a config fact (rss:latest:<name>), so a restart does not
re-note yesterday's headlines.
claude
kami