A local Qwen3.8-27B chat and API endpoint can run on your own computer with Ollama, provided the build and context length fit its memory. In Hardware Corner’s test, a quantised 27B build used about 22 GB of video memory at 64K context, making that a practical long-context point on a 24 GB GPU under its test conditions. Qwen3.8-Flash-Next is downloadable too, but its memory requirements put it in a different class.
Before you start: your aim and starting skills
This setup is for inference on your own computer: you download a model, then use your machine’s resources to produce responses. It assumes you can open a terminal and enter a command, but not that you know how language models work internally.
If your aim is simply to access Qwen through a hosted service, you do not need a local model download.
What to obtain for the local setup
You need a Windows, macOS or Linux computer whose available memory you can check, an internet connection for installation and the model download, and. Check your GPU’s video memory as well as system memory; the build you choose must fit while it runs, not merely leave enough disk space for its download.
Choose the downloadable Qwen model
For local AI, the defining test is where inference happens. A downloaded model runs on your device if it fits the available resources, as LM Studio explains. Ollama can run models locally or through its cloud: local models run on your computer, while cloud models run on Ollama’s servers without a model download. In its command-line naming, :cloud marks a cloud model.
| Model | What you get | Fit for this local setup |
|---|---|---|
| Qwen3.8-27B | A downloadable, 27-billion-parameter dense model with native image and video understanding, according to Qwen’s model card. | The starting choice here: quantised community builds give you memory and context options. |
| Qwen3.8-Flash-Next | Downloadable open weights for a multimodal mixture-of-experts model. Its Qwen repository specifies a 125-billion-parameter main model, another 51 billion n-gram embedding parameters and 6 billion parameters activated per token. | Local, but substantially more demanding to hold in memory. |
| Qwen3.8-Flash | Qwen’s hosted version based on Flash-Next, with a one-million-token context by default and official built-in tools. Its Qwen Cloud ID is qwen3.8-flash. |
A cloud service, not the model download used below. |
The smaller number of parameters activated per token in Flash-Next does not mean only that portion needs to be stored. Its n-gram embedding table can be offloaded to host memory, but that is a way of placing part of a large model, not turning it into a 27B-sized download.
Check memory against your intended context length
Context length is how much material the model can take into account in a request. Longer prompts also consume running memory: the key-value cache keeps attention calculations for reuse and, in the default dynamic form described by Hugging Face, grows as tokens are generated. A model file’s size or advertised context limit therefore cannot tell you, by itself, whether your GPU will hold a session.
The measurements below are for Hardware Corner’s 16.68 GiB Qwen3.8-27B Q4_K Small build, with full GPU offload. Its GPU tests used Ubuntu 24.04, NVIDIA driver 595.58.03, CUDA 12.8 and a specified llama.cpp build, with Flash Attention enabled and MTP disabled. Treat the table as a sizing guide for that tested build and configuration, not a promise that a different Ollama build will use identical memory.
| Context Length | Measured VRAM Usage |
|---|---|
| 4k | 18 GB |
| 8k | 18 GB |
| 16k | 19 GB |
| 32k | 20 GB |
| 64k | 22 GB |
| 128k | 26 GB |
| 256k | 34 GB |
Measured VRAM use by context length for Hardware Corner’s Qwen3.8-27B test build — hardware-corner.net
For that build, roughly 22 GB at 64K context made a fully GPU-resident run practical on one 24 GB card. At 128K it used about 26 GB, beyond that card’s capacity, while a 32 GB card could handle it; at 256K, about 34 GB exceeded even the 32 GB card. If your intended prompts push a build past available video memory, choose a shorter context or a build with different requirements rather than treating the tag’s maximum context as a target.
Flash-Next calls for a separate budget. Unsloth puts its smallest quantisation at a minimum of 75 GB of combined RAM and video memory, or unified memory, and recommends a 96 GB device for headroom. Its listed 75 GB one-bit and 355 GB BF16 figures are build sizes, not measurements of memory used while running. Ollama’s Flash-Next tags likewise list a 120 GB 125b-a6b-q4_K_M build and a separately MLX-labelled 105 GB 125b-a6b-nvfp4 build, each with a 256K context window and text and image input. Those download sizes do not make a 256K session fit automatically. For a different local route, Unsloth documents Flash-Next GGUFs for llama.cpp or Unsloth Desktop and memory-mapping its n-gram layer to an SSD to reduce CPU and GPU memory use.
Select a Qwen3.8-27B build for your computer
Community Ollama builds package the same model at different quantisations and context settings. The community-build listing behind the comparison below states memory targets alongside each tag. Use those targets to narrow your choice, then check the selected build’s own requirements and instructions: a target is not a measured guarantee for your computer.
| Tag | Quant | Size | Context | Vision | MTP | VRAM target | Use case |
|---|---|---|---|---|---|---|---|
| Q2_K_XL | 2-bit | 10.7 GB | 256K | ✅ | ✅ | 24 GB | 2-bit XL, full native context |
| Q2_K_XL-12gb | 2-bit | 10.7 GB | 8K | — | — | 12 GB | 2-bit XL, optimized for 12 GB VRAM |
| Q2_K_XL-16gb | 2-bit | 10.7 GB | 128K | — | — | 16 GB | 2-bit XL, optimized for 16 GB VRAM |
| Q3_K_M-128k | 3-bit | 13.8 GB | 128K | ✅ | ✅ | 24 GB | Balanced size / quality, long context |
| Q3_K_M-192k | 3-bit | 13.8 GB | 192K | — | ✅ | 24 GB | Balanced, longer context |
| Q3_K_M | 3-bit | 13.8 GB | 24K | — | ✅ | 16 GB | 3-bit KM, shares blob with 192k |
| Q3_K_XL | 3-bit | 13.4 GB | 219K | ✅ | ✅ | 24 GB | 3-bit XL, high quality |
| Q4_K_XL | 4-bit | 17.9 GB | 120K | ✅ | ✅ | 24 GB | Recommended default, 100% GPU |
| Q5_K_XL | 5-bit | 20.2 GB | 60K | ✅ | ✅ | 24 GB | 5-bit XL, max 100% GPU fit |
| Q5_K_XL-96k | 5-bit | 20.2 GB | 96K | ✅ | ✅ | 32 GB | 5-bit XL, more context |
| latest | 4-bit | 17.9 GB | 144K | — | ✅ | 24 GB | Default, MTP, no vision |
| IQ2_XXS | 2-bit | 9.0 GB | 32K | — | — | 12 GB | Smallest, most compatible |
| IQ2_XXS-12gb | 2-bit | 9.0 GB | 48K | — | — | 12 GB | 2-bit XXS, optimized for 12 GB VRAM |
| IQ2_M-12gb | 2-bit | 10.3 GB | 16K | — | — | 12 GB | 2-bit KM, optimized for 12 GB VRAM |
Qwen3.8-27B community Ollama builds and their stated memory targets — ollama.com
The commands that follow use the separately documented dna5rm/qwen3.8:27b-128k build. It is Q4_K_M, defaults to 131,072 tokens of context, is tuned for a 32 GB RTX 5090 and specifies Ollama 0.32.12 or newer. It is not the 24 GB, 64K build from Hardware Corner’s test: that test’s Q4_K Small build used about 26 GB at 128K. If your memory budget points to another tag in the comparison, substitute that tag throughout the remaining commands and follow its listing.
Install Ollama on your operating system
Get Ollama from its official download page, which provides Windows, macOS and Linux options. If you prefer the documented terminal installers, use Windows PowerShell for:
irm https://ollama.com/install.ps1 | iex
On macOS or Linux, the documented command is:
curl -fsSL https://ollama.com/install.sh | sh
Run an installer command only after checking that its address is the official one. Installation is separate from choosing and downloading a model; once installed, open the app or a terminal and follow Ollama’s setup prompts.
Download the selected local model
For the worked example, the community listing gives this download command:
ollama pull dna5rm/qwen3.8:27b-128k
Confirm you have Ollama 0.32.12 or newer for this build. A local model needs this download before it can run; an Ollama cloud model runs on Ollama’s servers instead and does not need one. If you selected another build to fit your hardware, pull its exact tag rather than assuming this 128K example will fit.
Start a local chat
Run the downloaded tag using the listing’s command:
ollama run dna5rm/qwen3.8:27b-128k
Enter a short prompt and check that it responds. Local Ollama models do not need an API key; cloud access requires sign-in or an API key for the relevant route.
If the response feels slow, first consider whether the build fits your GPU and whether the prompt is putting pressure on context memory. Also bear in mind that Qwen3.8-27B has thinking mode enabled by default; it can be disabled per request, and reasoning_effort can adjust reasoning depth. A slower answer is not, on its own, proof that the model failed to load.
Test the local chat API
Ollama’s local API base is http://localhost:11434/api. With Ollama running, this macOS or Linux terminal request sends a short message to its /api/chat endpoint. The model name is exactly the tag pulled above; no authorisation header is needed for local inference.
curl http://localhost:11434/api/chat
-H "Content-Type: application/json"
-d '{
"model": "dna5rm/qwen3.8:27b-128k",
"messages": [{"role": "user", "content": "Say hello in one sentence."}],
"stream": false
}'
On Windows PowerShell, the equivalent avoids shell-specific JSON quoting:
$body = @{
model = "dna5rm/qwen3.8:27b-128k"
messages = @(@{ role = "user"; content = "Say hello in one sentence." })
stream = $false
} | ConvertTo-Json -Depth 4
Invoke-RestMethod -Uri "http://localhost:11434/api/chat" `
-Method Post -ContentType "application/json" -Body $body
A returned assistant message confirms that the local server accepts the request, recognises the downloaded model tag and responds through this route. Do that check before adding another app: it leaves fewer possible causes if the app connection fails.
Point a compatible app at the local endpoint
An app that accepts a custom OpenAI-compatible endpoint can use Ollama’s local base URL, http://localhost:11434/v1, with dna5rm/qwen3.8:27b-128k as the model name. This is a different interface from the direct /api/chat request you just tested, so configure the URL the app asks for rather than appending /api/chat to its OpenAI-compatible setting. The community listing also has Python and JavaScript Ollama client examples if you are connecting your own code.
An app’s name alone does not tell you which connection each mode uses. Ollama’s desktop quickstart says ChatGPT Desktop uses Ollama models in Codex mode, while its regular Chat and voice modes keep their usual ChatGPT connection. Check the mode and endpoint before assuming a conversation is local.
When the local setup does not behave as expected
If the pull or run command fails, check the installed Ollama version against the worked build’s 0.32.12 minimum, then check that the tag in your command matches the build you chose. A successful download followed by a run that exhausts GPU memory points instead to model fit: shorten the intended context or select a build whose stated memory target suits your machine. Long prompts can increase cache memory use even when a short chat works.
If terminal chat responds but /api/chat does not, check that Ollama is running, the request goes to localhost:11434, and its model field matches the downloaded tag. If that direct request works but another app does not, inspect the app’s endpoint setting and mode. ChatGPT Desktop’s regular Chat and voice modes continuing to use ChatGPT is expected behaviour, not evidence that the local model has stopped.
Qwen3.8-27B can run as a local chat and API on suitable hardware
A downloaded Qwen3.8-27B build can respond on your own computer through Ollama, both in terminal chat and at a local API endpoint. The deciding constraint is the particular build at the context you intend to use: the tested 24 GB GPU reached 64K with one quantisation, while the worked 128K Ollama tag is tuned for 32 GB. Flash-Next remains a downloadable but much larger alternative; Qwen3.8-Flash is hosted, not the local model in this setup.