claude
95e7427153
mavgpud reverse-proxied every path to llama-server with no authentication on a LAN port. Any client on the network could submit model work, hold the model resident by touching the idle clock, and read /slots, which returns the prompts of whoever else was using the card. It now reads a bearer token from token_file and requires it on every request, /health included: /health reports whether the card is loaded and free, which is what someone deciding to take it would ask. A listen address reachable from the network with no token is a startup failure rather than a downgrade to loopback. homesrv is the client and it is on the LAN, so a loopback default would look safe and take the model arm down. Beyond the token: an allowlist of the five paths Maven calls, so a leaked token buys the model API and not llama-server's admin surface; a body cap and an in-flight cap on the proxy; and header and idle timeouts on the server. No read or write timeout — a completion on this card legitimately takes minutes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ESv8hqNPseYt1CnotZpqDz
58 lines
2.0 KiB
JSON
58 lines
2.0 KiB
JSON
{
|
|
"listen": ":8080",
|
|
"//token_file": [
|
|
"The bearer token every request must carry. homesrv is the client and it",
|
|
"is on the LAN, so this port cannot be loopback and the token is what",
|
|
"stops anything else on the network spending the card or reading /slots.",
|
|
"A path, never the token: mavgpud refuses to start when listen is",
|
|
"reachable from the network and this is unset.",
|
|
"Same value as MAVEN_GPU_TOKEN in homesrv's deploy/telegram.env."
|
|
],
|
|
"token_file": "/home/kami/.config/mavgpud.token",
|
|
"llama_addr": "127.0.0.1:10000",
|
|
"llama_bin": "llama-server",
|
|
"//llama_args": [
|
|
"E4B carries no MTP tensors, so the speculative flags are gone with the 12B.",
|
|
"MTP on this box is a separate gguf of architecture gemma4-assistant with",
|
|
"nextn_predict_layers=4, and mtp-gemma-4-12B-it-BF16 is the only one there is.",
|
|
"Its head is trained against the 12B's hidden states, so it cannot drive E4B."
|
|
],
|
|
"llama_args": [
|
|
"-m", "/mnt/D/AI/gemma4/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf",
|
|
"-ngl", "99",
|
|
"-fa", "on",
|
|
"-np", "1",
|
|
"--host", "127.0.0.1",
|
|
"--port", "10000",
|
|
"--ctx-size", "32768",
|
|
"--threads", "6",
|
|
"--batch-size", "2048",
|
|
"--ubatch-size", "512",
|
|
"--jinja",
|
|
"--chat-template-kwargs", "{\"enable_thinking\":false}"
|
|
],
|
|
|
|
"//stt": [
|
|
"CrisperWhisper 2.0 turbo, which Maven reaches directly on port 8081.",
|
|
"mavgpud runs it because it is a ROCm process on this card: under its own",
|
|
"systemd unit it registered on the KFD and the supervisor evicted",
|
|
"llama-server every few seconds. CW2_TOKEN comes from the unit's",
|
|
"EnvironmentFile and is never a flag value."
|
|
],
|
|
"stt": {
|
|
"addr": "127.0.0.1:8081",
|
|
"bin": "/home/kami/Programs/cw2-eval/.venv/bin/python",
|
|
"args": ["/home/kami/Programs/cw2-service/serve.py"]
|
|
},
|
|
|
|
"kfd_root": "/sys/class/kfd/kfd/proc",
|
|
"drm_device": "/sys/class/drm/card1/device",
|
|
|
|
"poll": "1s",
|
|
"idle_timeout": "15m",
|
|
"stop_grace": "20s",
|
|
"min_free_vram_bytes": 10737418240,
|
|
"evict_after_polls": 2,
|
|
"start_after_polls": 5
|
|
}
|