The PR 113 review found four defects in one line file. Swept the other four
families and the Go side for the same four.
The JSON was clean: no undeclared placeholder, no abbreviation spoken, no
single-variant entry left unfixed. One register leak — page_blocked read
"robots.txt" out loud, which is a filename, not a reason he can act on.
The count rule was not clean. Four more copies of the three-way agreement
existed and two of them were wrong: ruPlural produced «1 минут назад» and
«5 часа назад» because formatTime spelled the noun out. pluralTasksRU was a
fifth copy. All of them now call say.CountWord. The pending-notification
summary picks the whole phrase, because the adjective declines with the noun.
Netscan and the crawler both declined their own turn when the wiring was
nil, and the question fell through to the search leg. "какие устройства в
сети?" came back as a paragraph about routers in general, and a question
about his own LAN went to an upstream engine — the personal boundary
exists to stop exactly that. A URL he named came back answered as though
he had not named it.
Both now claim the turn once their own recogniser has matched, and say
which capability is missing: net_off and page_off in the query family.
TestQueryWebPassesWhenNotConfigured encoded the old decision, that
announcing a configuration status is only for a capability that exists and
failed. It is rewritten, not deleted: the gap is the answer now.
The three daemon tests that pinned a wording ask the entry instead. The eval
scores every query variant on the persona checks, minus hisgender: it reads her
own feminine verb next to "у тебя" as addressing him as a woman.
The on-demand crawler was built over allow_hosts plus every watched host.
webfetch reads a non-empty allow list as these and nothing else, so a config
with one watch and no allow_hosts at all silently narrowed on-demand reading
to the watched site. Every other url he pasted came back as a flat refusal
with nothing in the log to explain it. The two crawlers now take two host
lists from one crawlHosts helper.
Crawl-delay was parsed into Rules and never read. The only pacing was the
fetcher's flat one request per host per second, which cannot express what a
site asked for, and deploy/README claimed the field was honoured. Page now
waits it out between the robots fetch and the page fetch, and a delay longer
than the turn fails the read instead of hanging it.
A robots.txt that failed was treated as no rules, so a site whose server was
having a bad minute became a site with no restrictions. A 5xx now refuses the
crawl. A 404 still means unrestricted, which is what the standard says.
The refusal check matched substrings of webfetch's message text from a package
that cannot import webfetch, so a reworded error would have silently turned
into a robots verdict. internal/crawl now exports ErrFetchRefused and
ErrFetchStatus and the adapter in cmd/mavend maps the webfetch sentinels onto
them. Robots group selection picks the longest matching agent prefix instead
of the first one in file order.
queryWeb passed a claim it could not serve when no crawler was configured, so
an unconfigured deployment answered a web question with an apology instead of
falling through to the model.
Found in review of #67.
The network fallback behind the local sources, off unless configured.
internal/crawl is pure: a stdlib robots.txt parser (group specificity,
wildcards, Crawl-delay, cached per host), HTML-to-plaintext extraction, and a
watcher that notes a watched page only when its text changed. It has no store
access and no net/http; cmd/mavend/crawls.go is the impure half.
Every limit is code and tested: the guarded fetcher from #258 enforces the host
allowlist/denylist, refuses private addresses in the dialer Control hook (so DNS
rebinding and each redirect hop are covered), caps size and redirects, times out,
and spaces requests per host. A robots.txt Disallow is refused with no override.
On demand, reading is a query source placed last in the chain, after his memory,
his notes, and the local Kiwix ZIMs once those are wired: no URL in the
utterance means no fetch, and only the URL ever leaves the box. Scheduled
watches write notes and announce nothing.
The vendored tree has no x/net/html, goquery or temoto/robotstxt, so the parsers
are stdlib. No new dependency.
claude
kami