First slice of the research workflow: the pure, zero-inference extraction pipeline
that the WebFetchTool will consume. New :infrastructure:research module (jsoup).
HtmlMarkdownExtractor: content-type dispatch (HTML extracted; JSON/text pass
through; binary/media/pdf rejected at the header), then structural strip
(MainContentSelector: drop script/style/nav/footer/aside/form + class/id blocklist)
+ density extraction (text − link-text score, semantic <article>/<main> preferred)
+ DOM→markdown (MarkdownRenderer/Table/Inline: headings, lists incl. nested, tables,
fenced code, blockquotes, links, emphasis). Output below the min-length threshold is
flagged LOW_QUALITY so the workflow can route around dead sources (SPAs, paywalls).
extractorVersion ("html-md-1") is pinned on every result so replay reads recorded
artifacts and never re-extracts — the embedding-hash discipline (§5, ADR-0000 §10).
Next slices: WebFetchTool + egress policy + CAS wiring (emits the fetch +
LowQualityExtraction events), then WebSearchTool/SearXNG, workflow graph, synthesis.
kami
fb1d97058a