NVIDIA Flare is an open-source research prototype that serves local AI models on supported Blackwell systems, with long conversations and coding agents as its focus. Independent tests show unusually large cache budgets, but not a consistent speed advantage: the result changes with the model, hardware and workload.
Two NVIDIA projects share the Flare name
Flare for local inference is separate from NVIDIA FLARE, the federated-learning platform whose Python APIs organise training jobs. Their similar names can be confusing, as the two projects have very different documentation and deployment paths.
The hardware in question is DGX Spark, NVIDIA’s compact local-AI system previously called Project DIGITS. Flare is the serving software; Spark is one kind of computer on which a suitable Flare model recipe can run. Neither name, on its own, tells you how a chosen model will perform.
Flare is built around persistent conversation state
An AI inference server accepts requests, runs an already-trained model and returns its output. Flare’s stated targets are SM120 and SM121 GPUs, long conversations, coding agents and repeated tool use. NVIDIA makes it available as a research prototype for technical collaboration and early feedback, aiming to use the memory, bandwidth and compute of compact systems and workstations efficiently without making generation sluggish.
Its more distinctive choice concerns what happens between requests. Flare preserves conversation state as a whole, including recurrent state as well as key-value (KV) caches, when a session is reused, branched or resumed. That suits an agent that repeatedly calls tools, returns to earlier context or tries a different branch: retaining tokens alone may not retain everything its model needs to continue from the same state.
To host your own AI this way is to run the model and serve requests on hardware you control. It is a deployment choice, not a privacy guarantee; the endpoint, clients and routes through which requests pass still matter.
Independent speed results differ by model and workload
Flare was not the fastest engine in a four-DGX-Spark comparison. A separate test on one DGX Station GB300 found Flare faster on some tasks and slower on others. We would use those results to choose a benchmark for a particular deployment, not to award an overall speed ranking.
The four-Spark test ran DeepSeek-V4.1-Flash-1017 sequentially across the same four nodes for each engine, with the same switch, clocks, host settings and harness. The model had 552 billion parameters, 8 billion active per token and a stated one-million-token context. Flare’s measured KV cache was 37.8 million tokens, the largest of the four, but it had the slowest native decode and prefill. The tester described the build as a preview that was only a day old, which matters if you are judging how settled that result is.
The comparison’s vLLM lane was Christopher Owen’s spark-ds41f fork, not stock vLLM. It reached 287 tokens per second across six streams and read a 300,000-token document in 55 seconds on that setup. Those figures show why you should test the actual engine build you intend to deploy, rather than treating a project name as a fixed performance profile.
On the GB300, the tester instead ran GLM-5.3-Flash against its SGLang setup. In a single stream, Flare decoded code at 313 tokens per second versus 272 for SGLang. For prose with thinking enabled and a 512-token output, the figures were 261 and 235. At 16 concurrent requests the pattern split: Flare versus SGLang delivered 1,027 versus 784 tokens per second for prose, but 880 versus 907 for code and 987 versus 1,046 for JSON.
Cache capacity again favoured Flare in that setup: the tester reported roughly 10.5 million KV-cache tokens against 857,000 for SGLang. Both engines answered 194 of 200 GSM8K problems correctly in the initial comparison. A later within-Flare tuning run raised speculative-token capacity from 16 to 32 at 16 concurrent requests; per-stream decode rose from 76 to 95 tokens per second for prose and from 77 to 97 for code. That last improvement is a Flare configuration result, not another Flare-versus-SGLang result.
GPU compatibility still depends on the model recipe
NVIDIA lists the GB10 in DGX Spark at CUDA compute capability 12.1, corresponding to SM121. The RTX PRO 6000 Blackwell Workstation and Server Editions are listed at 12.0, corresponding to SM120. Those match Flare’s stated GPU targets, but a matching compute capability does not make every model fit.
Spark uses the GB10 Grace Blackwell Superchip with CPU–GPU-coherent memory connected through NVLink-C2C. NVIDIA specifies 128 GB of unified memory, 273 GB per second of memory bandwidth and up to 1 petaflop of FP4 AI performance; it also says the AI software stack comes preinstalled. NVIDIA’s claim that Spark can run local inference on models of up to 200 billion parameters is a capacity statement, not a promised serving speed. The RTX PRO 6000 Blackwell Workstation Edition has 96 GB of GDDR7, with 1.8 TB per second of listed memory bandwidth and a 600 W power design. Memory type, capacity and bandwidth are not interchangeable specifications.
For Flare, the decisive check is the recipe for your checkpoint. Recipes specify pinned checkpoint revisions, supported tensor-parallel GPU counts, concurrency ranges and a memory-cost model; Flare estimates whether the selected GPUs can hold the model before loading weights. The GB300 tester, for example, could not run its intended DeepSeek-V4.1-Flash comparison on one Station because that recipe required four GPUs. The same tester did serve GLM-5.3-Flash on SM103 with the project’s tested build, despite its stated SM120 and SM121 targets. Check the recipe and the build you will use, then obtain a current quote for the complete hardware configuration rather than budgeting from a peak-performance specification.
Ignition prepares packages before Helios serves requests
A compatible source checkpoint is not yet an immediately runnable Flare model. Ignition uses PyTorch, Transformers and FlashInfer/b12x to load it, tune kernels, compile the forward pass and capture CUDA graphs into a runtime package. Helios, the Rust runtime, then serves that prepared package without Python; compilation and graph capture have already happened.
The recipe owns model computation, while the shared runtime handles scheduling, cache storage and requests. The command line selects the model, GPUs, concurrency, storage and listeners; Flare derives settings such as context length, graph capacities and cache size from those choices and the recipe. Advanced preparation settings can override its prefill-chunk and speculation defaults. Ignition writes a metadata-only primary package file with separate, hash-named payloads for resources including weights, graphs and kernels, so changing a supporting resource need not rewrite unchanged weight shards.
Memory allocation follows the same prepared configuration. After accounting for weights, graphs, scratch space, live state, a concurrency-sized prefix cache and engine headroom, Flare assigns remaining available GPU memory to KV pages. Active requests and cached prefixes share that pool, so spare capacity can retain conversation history; memory limits and an optional KV-cache limit can bound it. Client documentation covers Chat Completions, raw Completions, Anthropic Messages and stateless Responses, along with authentication configuration. An API shape being documented is still different from having your own client’s tool use, session handling and failure behaviour tested.
Production use needs separate deployment checks
The prototype label has operational consequences. The GB300 tester reported two packaging patches before first token, roughly 60 minutes for its first boot and six minutes for a cached relaunch. No production client used that instance; the tester reported text-only operation, sampling constrained to the model card and no tool-call parity history on that machine. These are observations from one installation, but they make a first-token run and an actual client test sensible prerequisites for deployment.
Package handling needs attention too. Final packages use a signed format-4 envelope with encrypted cubins and graph records. Without publishing settings, Ignition creates a per-user development signing key; direct serving requires the corresponding publisher profile and first-use acceptance. Unsigned rank-staging files cannot be loaded in Helios. Before treating a prepared checkpoint as deployable, verify that its signing and acceptance path works on the serving machine.
Flare is not the only way to run local AI. LocalAI describes CPU-only as well as GPU operation and an OpenAI-compatible API over backends including llama.cpp, vLLM, SGLang and MLX; it also documents API keys, quotas and roles for multiple users. Those are reasons to compare deployment requirements if you do not specifically need Flare’s recipes and state handling, not evidence that LocalAI will be faster for your model.
For a company endpoint, authentication and a local GPU are only parts of the data-path decision. Map where the client sends requests, where the endpoint runs, who can reach it and what is retained along the way. If you need managed endpoint infrastructure around compute you supply, Infersec is an affiliate sovereign AI hosting provider whose platform lets individuals and companies bring their own hardware or compute for production-ready, authenticated, routed inference endpoints compatible with OpenAI and Anthropic endpoint standards. That is a different deployment proposition from assuming Flare alone supplies those production controls.
Flare’s fit is decided by the recipe and the workload
Flare is a local-inference research prototype built for persistent agent sessions on supported GPUs, and its independent tests show substantial cache capacity without a general speed lead. Whether it belongs on your system is settled first by the chosen model recipe’s GPU and memory requirements, then by tests of your actual concurrency and requests. For company data, the equally concrete boundary is the full request path and its access controls, not simply who owns the hardware.