fix(inference): tokenize endpoint sent wrong field — every token estimate was 0
LlamaCppTokenizer posted {"tokens":[text]} but llama.cpp's /tokenize expects
{"content":text}; the server silently answered {"tokens":[]}, so countTokens
returned 0 for every string. Every context entry carried tokenEstimate=0 and
the whole budget/trim pipeline was blind — live QA saw a 34k-token prompt
sail through a 16384 stage budget (three raw file_read results of plan docs),
crater t/s, degrade output, and fail the stage on validation 3x.
Also guard estimateTokens: a zero count for non-blank content falls back to
the chars/4 heuristic so a lying tokenizer can never blind budgeting again.
This commit is contained in:
+7
-1
@@ -1542,7 +1542,13 @@ abstract class SessionOrchestrator(
|
||||
protected open suspend fun estimateTokens(content: String): Int {
|
||||
val t = tokenizer
|
||||
if (t != null) {
|
||||
return runCatching { t.countTokens(content) }.getOrElse { fallbackTokenEstimate(content) }
|
||||
// A zero count for non-blank content means the tokenizer endpoint is lying
|
||||
// (e.g. a malformed request silently answered with an empty token list) —
|
||||
// trust the heuristic instead, or every budget check goes blind.
|
||||
return runCatching { t.countTokens(content) }
|
||||
.getOrElse { fallbackTokenEstimate(content) }
|
||||
.takeIf { it > 0 || content.isBlank() }
|
||||
?: fallbackTokenEstimate(content)
|
||||
}
|
||||
return fallbackTokenEstimate(content)
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user