feat(research): deterministic HTML→markdown extraction (research-workflow §5)
First slice of the research workflow: the pure, zero-inference extraction pipeline
that the WebFetchTool will consume. New :infrastructure:research module (jsoup).
HtmlMarkdownExtractor: content-type dispatch (HTML extracted; JSON/text pass
through; binary/media/pdf rejected at the header), then structural strip
(MainContentSelector: drop script/style/nav/footer/aside/form + class/id blocklist)
+ density extraction (text − link-text score, semantic <article>/<main> preferred)
+ DOM→markdown (MarkdownRenderer/Table/Inline: headings, lists incl. nested, tables,
fenced code, blockquotes, links, emphasis). Output below the min-length threshold is
flagged LOW_QUALITY so the workflow can route around dead sources (SPAs, paywalls).
extractorVersion ("html-md-1") is pinned on every result so replay reads recorded
artifacts and never re-extracts — the embedding-hash discipline (§5, ADR-0000 §10).
Next slices: WebFetchTool + egress policy + CAS wiring (emits the fetch +
LowQualityExtraction events), then WebSearchTool/SearXNG, workflow graph, synthesis.
This commit is contained in:
@@ -0,0 +1,14 @@
|
||||
plugins {
|
||||
id 'java-library'
|
||||
id 'org.jetbrains.kotlin.jvm'
|
||||
id 'org.jetbrains.kotlin.plugin.serialization'
|
||||
}
|
||||
|
||||
dependencies {
|
||||
implementation project(":core:events")
|
||||
implementation 'org.jsoup:jsoup:1.20.1'
|
||||
}
|
||||
|
||||
// Pure deterministic extraction utility; coverage gate is enforced via its own thorough
|
||||
// unit tests, not the global line-count bound (same stance as :infrastructure:tools).
|
||||
tasks.named("koverVerify").configure { enabled = false }
|
||||
Reference in New Issue
Block a user