fix(inference): tokenize endpoint sent wrong field — every token estimate was 0

LlamaCppTokenizer posted {"tokens":[text]} but llama.cpp's /tokenize expects
{"content":text}; the server silently answered {"tokens":[]}, so countTokens
returned 0 for every string. Every context entry carried tokenEstimate=0 and
the whole budget/trim pipeline was blind — live QA saw a 34k-token prompt
sail through a 16384 stage budget (three raw file_read results of plan docs),
crater t/s, degrade output, and fail the stage on validation 3x.

Also guard estimateTokens: a zero count for non-blank content falls back to
the chars/4 heuristic so a lying tokenizer can never blind budgeting again.
This commit is contained in:
2026-06-11 22:50:53 +04:00
parent e503a28db2
commit 35e921c6d1
4 changed files with 39 additions and 3 deletions
@@ -31,6 +31,36 @@ class DefaultContextPackBuilderTest {
tokenEstimate = tokens
)
private fun typedEntry(id: String, layer: ContextLayer, tokens: Int, sourceType: String) = ContextEntry(
id = ContextEntryId(id),
layer = layer,
content = "content-$id",
sourceType = sourceType,
sourceId = id,
tokenEstimate = tokens
)
@Test
fun `oversized tool results from a stage tool loop are trimmed to budget`() {
// Live repro (2026-06-11): analyst stage with three file_read results of ~5.6k/11.9k/10.7k
// tokens sailed through a 16384 budget untrimmed and produced a 34k-token prompt.
val entries = listOf(
typedEntry("sys", ContextLayer.L0, 3500, "systemPrompt"),
typedEntry("task", ContextLayer.L1, 150, "agentPrompt"),
typedEntry("call1", ContextLayer.L2, 40, "assistantToolCall"),
typedEntry("res1", ContextLayer.L2, 5600, "toolResult"),
typedEntry("call2", ContextLayer.L2, 40, "assistantToolCall"),
typedEntry("res2", ContextLayer.L2, 11900, "toolResult"),
typedEntry("call3", ContextLayer.L2, 40, "assistantToolCall"),
typedEntry("res3", ContextLayer.L2, 10700, "toolResult"),
)
val pack = builder.build(packId, sessionId, stageId, entries, TokenBudget(limit = 16384))
assertTrue(
pack.budgetUsed <= 16384,
"budgetUsed ${pack.budgetUsed} must not exceed the 16384 stage budget",
)
}
@Test
fun `build groups entries by layer`() {
val entries = listOf(