Searches the self-hosted SearXNG JSON API: local-first, no API keys, no third-party
telemetry. Hits only the configured endpoint, so unlike web_fetch there is no
arbitrary-host egress to police — T1. Returns the top-N results as a readable list
(title / url / snippet) with the result URLs in metadata so the fetch stage can act on
them and the operator can approve the source list rather than each URL.
Tested with Ktor MockEngine (query encoding, result parsing/cap, empty + error paths).
Completes the §3 "two tools" pair; with slice-1 extraction this is the deterministic
+ network-tool foundation of the research workflow.
The first tool that punches through the sandbox to the network. T2, declares
NETWORK_ACCESS with url marked NETWORK_TARGET, so egress is enforced by the harness
(NetworkHostRule, plane 2) — not tool courtesy. Header-first guards (§3): content-type
checked before the body is read (binary/media rejected at the header via the extractor's
shared content-type policy); declared Content-Length capped; the read itself is bounded
so a server lying about length can't blow the cap (default 10 MB).
On success returns the extracted markdown plus metadata for citation lineage + replay:
url, content_sha256 (over the raw bytes), fetched_bytes, extractor_version, quality.
Consumes slice-1's HtmlMarkdownExtractor. Tested with Ktor MockEngine.
NOT yet registered in the live tool registry: with an empty networkAllowedHosts the
host rule allows everything, so exposing a fetch tool before the egress allowlist
(SearXNG endpoint + approved-source hosts, per-session) is wired would be an open-egress
hole. Registration + egress config land with the research workflow slice.
First slice of the research workflow: the pure, zero-inference extraction pipeline
that the WebFetchTool will consume. New :infrastructure:research module (jsoup).
HtmlMarkdownExtractor: content-type dispatch (HTML extracted; JSON/text pass
through; binary/media/pdf rejected at the header), then structural strip
(MainContentSelector: drop script/style/nav/footer/aside/form + class/id blocklist)
+ density extraction (text − link-text score, semantic <article>/<main> preferred)
+ DOM→markdown (MarkdownRenderer/Table/Inline: headings, lists incl. nested, tables,
fenced code, blockquotes, links, emphasis). Output below the min-length threshold is
flagged LOW_QUALITY so the workflow can route around dead sources (SPAs, paywalls).
extractorVersion ("html-md-1") is pinned on every result so replay reads recorded
artifacts and never re-extracts — the embedding-hash discipline (§5, ADR-0000 §10).
Next slices: WebFetchTool + egress policy + CAS wiring (emits the fetch +
LowQualityExtraction events), then WebSearchTool/SearXNG, workflow graph, synthesis.
kami