Swap the workstation model to gemma-4-E4B (V-486)
Owner's call. E4B is 4.2GB against 6.7GB plus a 0.86GB draft, so with CW2 resident the card holds 5.8GB of 16GB instead of 9.2GB. Measured against a same-session 12B control on the 96-case fixture: 83.3% full against 84.4%, 89.6% intent-only against 91.7%, destination 19/33 against 23/33, p50 294ms against 344ms. Destination is the column that moved. E4B names nothing where the 12B names recall or calendar, which walks the whole chain rather than answering wrong. MTP is gone with the 12B and cannot come back. It is a separate gguf of architecture gemma4-assistant with nextn_predict_layers=4, and the only one on disk is trained against the 12B's hidden states. Neither target gguf carries nextn tensors, so neither self-speculates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
This commit is contained in:
+8
-5
@@ -2,9 +2,14 @@
|
||||
"listen": ":8080",
|
||||
"llama_addr": "127.0.0.1:10000",
|
||||
"llama_bin": "llama-server",
|
||||
"//llama_args": [
|
||||
"E4B carries no MTP tensors, so the speculative flags are gone with the 12B.",
|
||||
"MTP on this box is a separate gguf of architecture gemma4-assistant with",
|
||||
"nextn_predict_layers=4, and mtp-gemma-4-12B-it-BF16 is the only one there is.",
|
||||
"Its head is trained against the 12B's hidden states, so it cannot drive E4B."
|
||||
],
|
||||
"llama_args": [
|
||||
"-m", "/mnt/D/AI/gemma4/gemma-4-12B-it-qat-UD-Q4_K_XL.gguf",
|
||||
"-md", "/mnt/D/AI/gemma4/mtp-gemma-4-12B-it-BF16.gguf",
|
||||
"-m", "/mnt/D/AI/gemma4/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf",
|
||||
"-ngl", "99",
|
||||
"-fa", "on",
|
||||
"-np", "1",
|
||||
@@ -15,9 +20,7 @@
|
||||
"--batch-size", "2048",
|
||||
"--ubatch-size", "512",
|
||||
"--jinja",
|
||||
"--chat-template-kwargs", "{\"enable_thinking\":false}",
|
||||
"--spec-type", "draft-mtp",
|
||||
"--spec-draft-n-max", "2"
|
||||
"--chat-template-kwargs", "{\"enable_thinking\":false}"
|
||||
],
|
||||
|
||||
"//stt": [
|
||||
|
||||
Reference in New Issue
Block a user