85 lines
3.6 KiB
Markdown
85 lines
3.6 KiB
Markdown
# Where the resident model's 7.9GB of RSS goes (2026-08-03, homesrv)
|
|
|
|
Measured for Vikunja #499. The deployed llama-server held 7.9GB RSS for a 1.1GB
|
|
model file. Half a gigabyte of it was in swap, on a box that also runs
|
|
whisper.cpp, piper and the embedder.
|
|
|
|
## Method
|
|
|
|
`maven-mavend-1` was stopped for the measurement, with the owner's approval.
|
|
Its own binary then ran on the host with the exact deployed command line. That
|
|
binary is `/opt/maven/bin/llama-server`, version `1 (4c65955)`, a Vulkan build.
|
|
|
|
```sh
|
|
llama-server -m /mnt/hdd1/llms/qwen3/Qwen3-1.7B-UD-Q4_K_XL.gguf \
|
|
--host 127.0.0.1 --port 18099 -c 4096 -ngl 99 --no-webui
|
|
```
|
|
|
|
RSS was read from `/proc/<pid>/status` after load and after each of 8 distinct
|
|
1521-token prompts. `smaps` of the deployed process was read first, from inside
|
|
the container, since the host user cannot read another user's maps.
|
|
|
|
## The cause: the prompt cache, not the weights and not the offload
|
|
|
|
The startup log says it outright:
|
|
|
|
```text
|
|
srv load_model: prompt cache is enabled, size limit: 8192 MiB
|
|
srv llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
|
|
```
|
|
|
|
The server saves the full KV state of every idle slot it evicts. It keeps up to
|
|
8GiB of those states in host RAM (llama.cpp PR 16391). One saved prompt of 1521
|
|
tokens costs 166.377 MiB. That is 112 kiB per token, exactly Qwen3-1.7B's KV
|
|
footprint (28 layers x 2 x 1024 dims x 2 bytes).
|
|
|
|
RSS at rest, and per distinct prompt:
|
|
|
|
| Prompts served | RSS, default | RSS, `--cache-ram 512` |
|
|
|---|---|---|
|
|
| 0 (just loaded) | 443 MB | 411 MB |
|
|
| 1 | 445 MB | 411 MB |
|
|
| 4 | 958 MB | 929 MB |
|
|
| 8 | 1641 MB | 932 MB |
|
|
|
|
Uncapped, RSS climbs about 170MB per distinct prompt and does not stop until
|
|
the 8GiB limit. Capped at 512 MiB it plateaus at 932MB from the fourth prompt
|
|
on, with the cache holding steady at `3 prompts, 499.132 MiB` and evicting.
|
|
|
|
The 7.9GB on the running daemon was that climb, weeks of it. Its `smaps` showed
|
|
one 6.03GB anonymous mapping at 5.32GB resident plus a 1.45GB mapping at 1.27GB
|
|
resident, and only 30MB of file-backed RSS.
|
|
|
|
## The task's leading guess was wrong
|
|
|
|
`-ngl 99` on the Vega iGPU costs almost no process RSS. A freshly loaded server
|
|
has 95MB of anonymous RSS in total. RADV allocates device memory through the
|
|
kernel, outside the process, and the log sees 8202 MiB free on `Vulkan0`. The
|
|
weights are mmapped and file-backed, so they are evictable and do not pin RSS. The logit buffer is not visible in the numbers above at all.
|
|
|
|
## Decision
|
|
|
|
`--cache-ram 512` is now the default, wired as `phraser.cache_ram_mib` and set
|
|
in `deploy/mavend.json`. 512 MiB caps total RSS near 1GB, an eighth of what the
|
|
box carried. It still holds three of the 1521-token probes above. Maven's real
|
|
routing and phrasing prompts are much shorter, so it holds more of those than
|
|
the table suggests. `-c 4096` is untouched, as #499
|
|
required. A negative `cache_ram_mib` passes no flag, for a llama-server too old
|
|
to know it.
|
|
|
|
Not changed: `n_parallel = 4`. With `kv_unified = true` the four slots share one
|
|
4096-token KV cache, so they do not multiply it.
|
|
|
|
The other half of #499 was that none of these lines were reachable. mavend
|
|
scraped llama-server's stderr for the listen line and discarded it, and never
|
|
piped stdout at all. Both streams now go to mavend's log with a `llama:` prefix.
|
|
The last 12 startup lines go into the error when the server dies before it
|
|
listens.
|
|
|
|
## Deployed
|
|
|
|
The `mavenai:latest` image was rebuilt and `maven-mavend-1` recreated the same
|
|
day. The daemon's own log now carries the child's startup, it reads
|
|
`prompt cache is enabled, size limit: 512 MiB`, and the resident server sat at
|
|
439MB RSS after load and 613MB after one served turn.
|