Parallel request handling in vLLM depends on how many requests the scheduler can run, how many tokens it can process per iteration and how much GPU memory is available for the key-value (KV) cache. Those are different limits from the number of requests still waiting. Extra GPUs and hosted routing may help with other parts of a local AI setup, but neither removes a bottleneck inside the serving instance.
1. Running requests, queued requests and tokens are separate limits
vLLM can accept work that it cannot run immediately. In its scheduler configuration, max_num_active_seqs limits requests admitted to the RUNNING state, while max_num_queued_reqs limits requests in flight across both waiting and running states. A request can therefore be accepted and still spend time in a queue.
The same configuration defines max_num_batched_tokens as the token budget for one scheduler iteration. Raising the running-request limit does not give that iteration a larger token budget. For prompts that need prefill work, enable_chunked_prefill permits that work to be split into chunks rather than handled as one indivisible prefill.
If you are tuning the CLI flag --max-num-seqs, do not assume from its name that it caps every incoming request, or equate it with a configuration field without checking your installed version’s flag mapping. The practical questions are how many requests may be running, how many may remain in flight and how much token work the scheduler can take on at once.
2. Give the KV cache room before adding GPU instances
Sending requests asynchronously prevents the client from waiting for each response before submitting the next. It does not create GPU memory. vLLM warns that a conservative --kv-cache-memory allocation limits batch concurrency and throughput; an allocation that is too large fails when vLLM tries to allocate it. If responses slow as concurrency grows, the cache allocation deserves attention alongside the scheduler limits.
The current cache configuration gives gpu_memory_utilization a default of 0.92, with an allowed value greater than zero and no higher than one. It is a per-instance setting: two vLLM instances on the same GPU do not automatically divide that allowance between them. Adding an instance to a busy GPU is therefore not a substitute for checking what each instance can allocate.
A manually reused --kv-cache-memory value is tied to the same GPU and the same initial free memory. If a hardware or co-tenant change makes the server fail at boot with an out-of-memory error, vLLM advises removing the flag so it can profile memory again. Placing a model across GPUs is a separate deployment decision; it does not change why an undersized cache constrains concurrent work.
3. Faster startup is not the same as faster decoding
Some options save time before the server accepts requests. Reusing a measured --kv-cache-memory value skips memory profiling and CUDA-graph memory estimation on a subsequent boot. By contrast, --enforce-eager skips compilation and CUDA-graph capture for a faster start, but vLLM documents a cost to steady-state decoding performance. We would not use a faster boot as evidence that responses under load will be faster.
The optimisation levels make that trade-off more explicit. In vLLM’s guide, -O0 prioritises startup without optimisations; -O1 adds simple compilation, fast fusions and piecewise CUDA graphs; -O2 is the default and adds further compilation, fusions and CUDA graphs. -O3 currently equals -O2.
For deployments where compilation time matters, vLLM stores torch.compile artefacts under VLLM_CACHE_ROOT, which defaults to ~/.cache/vllm; that cache can be copied to another machine or put in a container image. Setting VLLM_FORCE_AOT_LOAD=1 makes a cache miss fail instead of quietly recompiling. These choices make startup more predictable. Running-request capacity, token budget and cache space still govern the load the server can carry.
4. Different base models need routing; LoRA adapters share a base
Serving two unrelated base models and serving variations of one base model are different jobs:
| What clients need | How to handle it |
|---|---|
| Different base models | Run the required model deployments and route each request to the appropriate one. Infersec describes public or private OpenAI-compatible endpoints with composable routing rules for this layer. |
| Adaptations of one base model | Start vllm serve with --enable-lora and --lora-modules {name}={path} to make named low-rank adaptation (LoRA) adapters available on that base, as shown in the LoRA documentation. |
The LoRA server applies settings including max_loras, max_lora_rank and max_cpu_loras to forthcoming requests. An adapter is not a way to load an unrelated base model into the same instance. If clients need distinct base models, routing between deployments answers the model-selection question; each deployment still has its own serving capacity to tune.
5. Measure concurrent serving by latency as well as throughput
The test for an asynchronous client or a new vLLM configuration is a benchmark against a running server using a workload like the one you intend to serve. The documented procedure starts vllm serve, then runs vllm bench serve against it. Its serving example uses --backend vllm, --endpoint /v1/completions, a ShareGPT dataset path and --num-prompts 10; your prompt lengths and concurrency should reflect your own clients rather than that example.
Compare request, output-token and total-token throughput with latency. The benchmark reports time to first token (TTFT), time per output token after the first (TPOT) and inter-token latency (ITL), including mean, median and P99 values. A throughput gain is less useful if the slowest responses now miss the latency your users can tolerate.
Keep prefix-cache reuse consistent between comparisons. vLLM cautions that repeating a benchmark against the same server can reuse cached prompts and inflate throughput. Vary --seed, reset or restart the server, or use vllm bench sweep serve to reset server caches between runs when reuse is not part of the intended workload. Shared prefixes can also produce cache hits within a single run, so interpret a fast result in light of the prompts that produced it.
6. Keep local compute and expose routed production endpoints
Your own hardware can back an OpenAI-compatible API. For professional or at-scale endpoint management, a bring-your-own-compute hosting provider such as Infersec is one option: it keeps the serving compute under your control while providing a separate layer for exposing and routing endpoints. Infersec says its self-hostable Conduit agent connects vLLM, llama.cpp and other inference-engine servers to its platform. It describes registering a Mac or Linux machine as a compute source, with live engine, model and health telemetry, and says its endpoints are compatible with OpenAI and Anthropic.
Its cloud option is described as a managed multi-instance platform with pay-as-you-go credits, an auto-scaling API and a console that can connect to customers’ hardware. Its self-hosted option puts the console and API in one Docker service on the customer’s infrastructure, under a licence rather than platform billing or metering. The choice changes how you operate the endpoint layer; it does not allocate more KV cache to the vLLM process underneath.
Endpoint reach does not raise the serving ceiling
A routed endpoint can make locally supplied compute accessible to multiple clients, but the vLLM scheduler, KV-cache allocation and measured latency still determine how their requests perform. Infersec is a bring-your-own-compute hosting option for managing that professional endpoint layer, not a performance setting for the engine. The serving benchmark shows whether the instance meets the clients’ targets.