To handle simultaneous requests in llama.cpp, start with llama-server and size its context and batching settings for the model, memory and workload your machine can sustain. Its parallel decoding accepts multiple users; the separate llama-parallel example simulates clients, while llama-batched-bench helps test prompt delay and generation speed.
1. Set context and batching capacity for the server
llama-server lists parallel decoding, multi-user support and continuous batching. That makes it the starting point for a local AI service receiving requests from several clients. It does not mean that every additional request will finish at the same speed: requests still compete for context capacity and processing time.
The server options give you the main controls:
-cor--ctx-sizesets the prompt-context size. Its documented default,0, uses the value from the model. Set it deliberately if the model’s value is larger than the context your memory budget can support.-bor--batch-sizesets the logical maximum batch size; its documented default is2048. This limits how much work can be grouped for processing, rather than specifying a number of clients.--threadssets CPU threads for generation.--threads-batchsets them for batch or prompt processing and defaults to the generation-thread setting. Separate them if prompt processing and generation call for different CPU allocations on your machine.--predictsets the number of tokens to generate. Its documented default is-1, meaning no fixed token limit. A finite limit makes a test workload easier to compare with the requests you expect to serve.
Do not copy a --parallel command from an example and assume its syntax or effect matches your installed server. Check llama-server --help for that build’s concurrency option, then size context and batching for the requests you intend to accept.
2. Memory depends on the model and the active context
A model’s weights are only part of its serving memory. The key-value (KV) cache holds information from active token sequences, so longer contexts and concurrent requests can increase memory use even when the model itself stays the same. The following maintainer guide figures show model data, compute buffers and context-dependent KV-cache costs for two gpt-oss models. They are illustrations of those costs, not a fixed memory allowance per user.
| Model | Model data (GB) | Compute buffers (GB) | KV cache per 8 192 tokens (GB) | Total @ 8 192 tokens (GB) | Total @ 32 768 tokens (GB) | Total @ 131 072 tokens (GB) |
|---|---|---|---|---|---|---|
| gpt‑oss 20B | 12.0 | 2.7 | 0.2 | 14.9 | 15.5 | 17.9 |
| gpt‑oss 120B | 61.0 | 2.7 | 0.3 | 64.0 | 64.9 | 68.5 |
Maintainer guide’s memory requirements for two gpt-oss models — github.com
The same guide explains that attention tensors and the KV cache can remain in video memory (VRAM) while the rest of the model stays in CPU memory, with decent performance still possible. You need not assume the entire model must fit in VRAM, but you do need room for the active context.
For a rough starting point, daily.dev’s hardware guide assigns 16 GB of memory to 7B–8B models. That is a single-model guide, not a promise that several long conversations will fit. Its estimate of around 20 GB to download a quantised 30B model concerns storage, which is a different budget from the memory needed to serve it.
3. -np 8 simulates eight clients, not eight server settings
llama-parallel is an example application that simulates concurrent clients; it is not the llama-server HTTP service. In its documented example, -np 8 simulates eight clients and -ns 128 generates 128 client requests. Neither flag in that example configures eight settings on a running server.
The example can also use --junk 10 to put up to ten junk questions before each request’s actual question. That changes the test workload, so its results are useful only if the workload resembles what you want to investigate. The example recommends base models because instruction-tuned models may not follow its custom chat template and can give unexpected results. Treat that caveat as part of choosing a test model, not as a rule against instruction-tuned models on the server.
4. Shared prompts change the KV-cache calculation
Two groups of concurrent requests can have the same number of clients and generated tokens yet need different KV-cache capacity. The difference is whether each request carries its own prompt or the requests share one.
- Separate prompts: Each batch has its own prompt. For
Bbatches,PPprompt tokens andTGgenerated tokens per batch, llama.cpp’s batching benchmark gives the required KV-cache size in tokens asB*(PP + TG). - Shared prompt: The prompt is counted once, giving
PP + B*TGunder the same benchmark calculation. The benchmark uses-ppsto select this mode; in the parallel example,-ppsshares the system prompt so it is computed once at the start.
That saving matters most when a substantial prompt genuinely is common to the requests. It should not be budgeted for independent conversations with different histories: their prompt tokens are separate work.
5. Benchmark prompt delay and generation speed separately
A throughput figure alone can hide an uncomfortable wait before the first token. The batching benchmark reports prompt-processing time as time to first token and text-generation speed separately. It accepts lists of prompt lengths with -npp, generated-token lengths with -ntg, and parallel batch counts with -npl, so you can vary all three within a run.
The sample separate-prompt results below show the measurements the benchmark produces. Read them as a comparison of the workloads shown, not a speed guarantee for your model, hardware or HTTP server.
| PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s |
|---|---|---|---|---|---|---|---|---|---|
| 128 | 128 | 1 | 256 | 0.108 | 1186.64 | 3.079 | 41.57 | 3.187 | 80.32 |
| 128 | 128 | 2 | 512 | 0.198 | 1295.19 | 5.029 | 50.90 | 5.227 | 97.95 |
| 128 | 128 | 4 | 1024 | 0.373 | 1373.96 | 6.878 | 74.44 | 7.251 | 141.23 |
| 128 | 128 | 8 | 2048 | 0.751 | 1363.27 | 7.344 | 139.43 | 8.095 | 252.99 |
| 128 | 128 | 16 | 4096 | 1.570 | 1304.68 | 8.455 | 242.23 | 10.024 | 408.60 |
| 128 | 128 | 32 | 8192 | 3.408 | 1201.73 | 8.801 | 465.40 | 12.209 | 670.96 |
| 128 | 256 | 1 | 384 | 0.107 | 1196.70 | 6.329 | 40.45 | 6.436 | 59.67 |
| 128 | 256 | 2 | 768 | 0.194 | 1317.45 | 10.239 | 50.00 | 10.433 | 73.61 |
| 128 | 256 | 4 | 1536 | 0.366 | 1399.03 | 13.960 | 73.35 | 14.326 | 107.22 |
| 128 | 256 | 8 | 3072 | 0.751 | 1363.92 | 15.110 | 135.54 | 15.861 | 193.69 |
| 128 | 256 | 16 | 6144 | 1.569 | 1304.93 | 18.073 | 226.64 | 19.642 | 312.80 |
| 128 | 256 | 32 | 12288 | 3.409 | 1201.35 | 19.223 | 426.15 | 22.633 | 542.93 |
llama.cpp sample batching results with separate prompts — github.com
For your own test, keep the model and machine fixed while varying batch count and token lengths that resemble real requests. Watch both the delay to the first token and generation speed as concurrency rises. The benchmark also supports --output-format jsonl if you want JSONL output rather than Markdown for comparing runs.
6. Choose a model, confirm it runs offline, and set quality expectations
Start with a model suited to the task and a quantisation that fits the machine before increasing concurrent load. llama-server supports F16 and quantised inference on CPU and GPU. Quantisation choices include Q4_K_M and Q8_0, while model choices differ for coding, vision, reasoning and general use, as the local-AI guide explains.
A Qwen3-8B GGUF is one concrete starting model to assess against the 7B–8B memory tier, with several quantisations available. Choose among them by checking fit and testing your actual tasks; the model card does not designate one quantisation as best for parallel serving. Once the runtime and model file are available locally, CPU or GPU inference need not send prompts to an online model service. Confirm that the complete setup works without a connection before relying on offline use.
Local AI can be adequate for routine tasks you test successfully, but it is not a blanket substitute for ChatGPT. The local-AI guide identifies inconsistent tool use, complicated vision pipelines and hallucinations as limitations, and favours cloud models for complex tasks. Quality testing belongs alongside speed testing if other people will depend on the answers.
7. Use the OpenAI-compatible API routes for client connections
llama-server provides REST APIs and a web interface. For client applications, it includes OpenAI-compatible chat-completions, responses and embeddings routes, as well as a separate Anthropic Messages API-compatible chat-completions route.
Point each client at the route matching the type of request it sends, then test those clients together rather than assuming API compatibility settles capacity. A route lets a request reach the server; the context and processing budget determine how well the server handles the requests arriving through it.
Accepting requests is not the same as sustaining them
Parallel decoding lets llama-server accept concurrent work, but the useful limit is set by active context, how much prompt content can be shared and measured generation speed on the chosen hardware. For professional-scale use, we would favour a bring-your-own-compute hosting provider: Infersec is an affiliate sovereign AI hosting provider whose platform lets users and companies bring their own hardware or compute for production-ready, authenticated and routed inference endpoints compatible with OpenAI and Anthropic endpoint standards.