kami
fb1d97058a
First slice of the research workflow: the pure, zero-inference extraction pipeline
that the WebFetchTool will consume. New :infrastructure:research module (jsoup).
HtmlMarkdownExtractor: content-type dispatch (HTML extracted; JSON/text pass
through; binary/media/pdf rejected at the header), then structural strip
(MainContentSelector: drop script/style/nav/footer/aside/form + class/id blocklist)
+ density extraction (text − link-text score, semantic <article>/<main> preferred)
+ DOM→markdown (MarkdownRenderer/Table/Inline: headings, lists incl. nested, tables,
fenced code, blockquotes, links, emphasis). Output below the min-length threshold is
flagged LOW_QUALITY so the workflow can route around dead sources (SPAs, paywalls).
extractorVersion ("html-md-1") is pinned on every result so replay reads recorded
artifacts and never re-extracts — the embedding-hash discipline (§5, ADR-0000 §10).
Next slices: WebFetchTool + egress policy + CAS wiring (emits the fetch +
LowQualityExtraction events), then WebSearchTool/SearXNG, workflow graph, synthesis.
15 lines
458 B
Groovy
15 lines
458 B
Groovy
plugins {
|
|
id 'java-library'
|
|
id 'org.jetbrains.kotlin.jvm'
|
|
id 'org.jetbrains.kotlin.plugin.serialization'
|
|
}
|
|
|
|
dependencies {
|
|
implementation project(":core:events")
|
|
implementation 'org.jsoup:jsoup:1.20.1'
|
|
}
|
|
|
|
// Pure deterministic extraction utility; coverage gate is enforced via its own thorough
|
|
// unit tests, not the global line-count bound (same stance as :infrastructure:tools).
|
|
tasks.named("koverVerify").configure { enabled = false }
|