store: keep what she read out of what he said
Nothing at read time told a feed item or a crawled page apart from his own notes. QueryNotes ranked every note by cosine and the notes answer handed the nearest five to the phraser, so "что я говорил про переезд" could be answered out of a stranger's web page, prefixed with "вот что я нашла: ". Recall now excludes the read sources, rss: and crawl:, and the list is one place. The feed answer needed a different read as a result, and it needed one anyway: it scanned the last 200 notes of any source, so a busy day of voice notes pushed the newest headline out of the window and she said "в лентах пока ничего нового" while the poller was working fine. RecentNotesFromSource asks for feed notes by source, so the window holds 200 of them. Found in review of #66 and #67.
This commit is contained in:
+11
-2
@@ -65,7 +65,13 @@ What it does and does not do:
|
||||
response at 2 MiB and redirects at 3, and makes at most one request per host
|
||||
per second. See `internal/webfetch`;
|
||||
- how far each feed was read is stored as a config fact `rss:latest:<name>`, so
|
||||
a restart does not re-note yesterday's headlines.
|
||||
a restart does not re-note yesterday's headlines. A feed whose items carry no
|
||||
dates gets the same mark, and the first poll after a restart takes those items
|
||||
as already read rather than writing them all again;
|
||||
- `max_items` paces, it does not drop: a burst larger than the cap arrives over
|
||||
the following polls, oldest first;
|
||||
- feed notes are **not** part of recall. "что я говорил про X" searches what he
|
||||
said; headlines are read back only by asking about the feeds.
|
||||
|
||||
### Reading a page (`crawl`, also off by default)
|
||||
|
||||
@@ -95,7 +101,10 @@ switched:
|
||||
refusal she says out loud. `Crawl-delay` is honoured;
|
||||
- same guarded fetcher as the feeds: allowlist/denylist, no private addresses,
|
||||
size cap, redirect cap, timeout, one request per host per second;
|
||||
- dedup state is the config fact `crawl:hash:<name>`.
|
||||
- dedup state is the config fact `crawl:hash:<name>`;
|
||||
- like feed notes, watch notes are kept out of recall (`store.ReadSourcePrefixes`).
|
||||
Text from someone else's page is not something he said, so it must not come
|
||||
back as an answer to a question about him. Watch notes are visible on `/dash`.
|
||||
|
||||
## Not yet verified / host-dependent
|
||||
|
||||
|
||||
Reference in New Issue
Block a user