claude
a1a2fa3704
Owner's call. E4B is 4.2GB against 6.7GB plus a 0.86GB draft, so with CW2 resident the card holds 5.8GB of 16GB instead of 9.2GB. Measured against a same-session 12B control on the 96-case fixture: 83.3% full against 84.4%, 89.6% intent-only against 91.7%, destination 19/33 against 23/33, p50 294ms against 344ms. Destination is the column that moved. E4B names nothing where the 12B names recall or calendar, which walks the whole chain rather than answering wrong. MTP is gone with the 12B and cannot come back. It is a separate gguf of architecture gemma4-assistant with nextn_predict_layers=4, and the only one on disk is trained against the 12B's hidden states. Neither target gguf carries nextn tensors, so neither self-speculates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ptwopxyo3Z2kwFckHkLvN
49 lines
1.6 KiB
JSON
49 lines
1.6 KiB
JSON
{
|
|
"listen": ":8080",
|
|
"llama_addr": "127.0.0.1:10000",
|
|
"llama_bin": "llama-server",
|
|
"//llama_args": [
|
|
"E4B carries no MTP tensors, so the speculative flags are gone with the 12B.",
|
|
"MTP on this box is a separate gguf of architecture gemma4-assistant with",
|
|
"nextn_predict_layers=4, and mtp-gemma-4-12B-it-BF16 is the only one there is.",
|
|
"Its head is trained against the 12B's hidden states, so it cannot drive E4B."
|
|
],
|
|
"llama_args": [
|
|
"-m", "/mnt/D/AI/gemma4/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf",
|
|
"-ngl", "99",
|
|
"-fa", "on",
|
|
"-np", "1",
|
|
"--host", "127.0.0.1",
|
|
"--port", "10000",
|
|
"--ctx-size", "32768",
|
|
"--threads", "6",
|
|
"--batch-size", "2048",
|
|
"--ubatch-size", "512",
|
|
"--jinja",
|
|
"--chat-template-kwargs", "{\"enable_thinking\":false}"
|
|
],
|
|
|
|
"//stt": [
|
|
"CrisperWhisper 2.0 turbo, which Maven reaches directly on port 8081.",
|
|
"mavgpud runs it because it is a ROCm process on this card: under its own",
|
|
"systemd unit it registered on the KFD and the supervisor evicted",
|
|
"llama-server every few seconds. CW2_TOKEN comes from the unit's",
|
|
"EnvironmentFile and is never a flag value."
|
|
],
|
|
"stt": {
|
|
"addr": "127.0.0.1:8081",
|
|
"bin": "/home/kami/Programs/cw2-eval/.venv/bin/python",
|
|
"args": ["/home/kami/Programs/cw2-service/serve.py"]
|
|
},
|
|
|
|
"kfd_root": "/sys/class/kfd/kfd/proc",
|
|
"drm_device": "/sys/class/drm/card1/device",
|
|
|
|
"poll": "1s",
|
|
"idle_timeout": "15m",
|
|
"stop_grace": "20s",
|
|
"min_free_vram_bytes": 10737418240,
|
|
"evict_after_polls": 2,
|
|
"start_after_polls": 5
|
|
}
|