feat(inference): operator-tunable sampling knobs (top_k/min_p/repeat_penalty) for stage requests
GenerationConfig only carried temperature/top_p/max_tokens/stop/seed. Added nullable topK/minP/repeatPenalty, serialized to the llama.cpp and OpenAI-compat request bodies via @EncodeDefault(NEVER) so an unset knob is omitted (the model keeps its own default) and behavior is unchanged unless the operator opts in. Surfaced as a new [sampling] config section feeding the default stage GenerationConfig (the main agentic loop) through TomlWorkflowLoader + ExecutionPlanCompiler; the former hardcoded temperature=0.7/topP=1.0 stage defaults now come from config. Talkie chat/narration keep their own generation settings. Vikunja #46 (task 76) — sampling half.
This commit is contained in:
@@ -385,10 +385,23 @@ fun main() {
|
||||
// Plan-compile gate: wrap the ExecutionPlanCompiler as a post-stage check so a compile-invalid
|
||||
// architect plan is handed back through the retry-feedback loop instead of parking the session in
|
||||
// ACTIVE (post-planning compile dead-end). Returns null on success, the compiler message on failure.
|
||||
// Operator sampling defaults ([sampling] config) sent on every stage inference request. maxTokens=0
|
||||
// is a placeholder; the loader/compiler pin it per-stage to the token budget via copy().
|
||||
val stageSamplingDefaults = with(correxConfig.sampling) {
|
||||
GenerationConfig(
|
||||
temperature = temperature,
|
||||
topP = topP,
|
||||
maxTokens = 0,
|
||||
topK = topK,
|
||||
minP = minP,
|
||||
repeatPenalty = repeatPenalty,
|
||||
)
|
||||
}
|
||||
val planCompiler = ExecutionPlanCompiler(
|
||||
artifactKindRegistry,
|
||||
toolRegistry.all().map { it.name }.toSet(),
|
||||
injectRecovery = true,
|
||||
samplingDefaults = stageSamplingDefaults,
|
||||
)
|
||||
// Startup-load orchestration tuning from the [orchestration] config section into the kernel's
|
||||
// OrchestrationTuning. ponytail: read once at boot — a config edit needs a restart to take effect
|
||||
@@ -449,7 +462,7 @@ fun main() {
|
||||
tuning = orchestrationTuning,
|
||||
)
|
||||
val workflowRegistry = FileSystemWorkflowRegistry(
|
||||
InfrastructureModule.createWorkflowLoader(configArtifactKindsEarly),
|
||||
InfrastructureModule.createWorkflowLoader(configArtifactKindsEarly, stageSamplingDefaults),
|
||||
)
|
||||
// Builds the router facade from a config snapshot, mapping the [router] block onto the domain
|
||||
// TalkieConfig. Reused by ConfigService's rebuild hook so router knob edits apply live (the
|
||||
|
||||
Reference in New Issue
Block a user