Run Qwen3.8-Flash-Next Locally with Strata: 64 GB Quickstart

Qwen3.8-Flash-Next (QFN) has 125B total parameters, 6B active per token, and a native 256K context window. It has thinking enabled by default and records the highest agentic-coding benchmarks in its local-model class.

This article describes running a 3-bit quantization of QFN on a 64 GB RAM desktop with an RTX 5090 using Strata, and documents why sub-Q4 quantizations do not produce reliable tool calls for agentic workloads.

The install commands, the RAM each quantization pins, and the tool-call failures on a 64 GB desktop are below. Speed and failure figures mix runs on that machine — an RTX 5090, 64 GB of RAM, Ubuntu 26.04 — with reports on the Strata tracker and the 0.2.11 release notes.

Commands and engine defaults were checked on 06/11/2026 against Strata 0.2.11, when the repository showed 40,836 stars and the engine packages were 0.1.57. Where this runtime sits next to Ollama, llama.cpp, and vLLM is the job of the LLM hosting guide.

What Strata Is

The engine in this quickstart is Niko1221/Strata, MIT-licensed. A different project, the OSDI 2026 paper on hierarchical KV caching for SGLang, uses the same name. Strata builds on llama.cpp and ggml, and the repository includes a write-up at docs/paper/Strata-Paper.pdf. It runs Qwen3.8-Flash-Next across one consumer GPU, system RAM, and an SSD. The same engine also serves Qwen3.6-35B and DeepSeek V4.1-Flash.

Qqwen 3.8 Flash Next has 24,576 routed experts across 48 layers; each token activates 10 experts (6B active params in total). Strata divides the model across three memory tiers instead of requiring it to fit in one.

The 51B n-gram parameter block serves as a 20M-entry lookup table indexed by bigrams and trigrams. Because this is a lookup rather than a matrix multiplication, it can reside on an SSD with minimal runtime cost.

VRAM capacity determines the scaling axis. Every additional gigabyte of VRAM holds approximately 700 experts, and each expert cached on the GPU reduces CPU compute. A slow card with more VRAM outperforms a fast card with less.

RAM capacity is a hard constraint because all experts must be pinned for simultaneous CPU and GPU computation. Strata does not support lazy memory mapping to disk. Download size and RAM usage are nearly identical for QFN.

How the decode speed is built

Three mechanisms supply the decode rate.

1. Multi-token prediction. The checkpoint includes one MTP layer trained for multi-step drafts. Strata drafts up to 3 tokens and verifies them in a single pass over all 48 layers — the draft-verify cycle in the speculative decoding guide. The project measures average acceptance at 2.4–3.2 tokens per pass, and reports that as 1.6–1.8× decode speed with the same output.

2. Prompt lookup. The server pulls up to 5 draft tokens from earlier copies of the context when the model is repeating it, as in a code edit or a quotation. The project reports code edits 6–11% faster, and other text unchanged.

3. Chunked prefill. Prompts arrive in blocks of up to 8,192 tokens, and the next layer’s experts move over PCIe while the current layer’s attention runs. The project puts long-document and codebase ingestion above 1,000 tokens/s, and a follow-up starts within seconds because the conversation state stays resident.

The server ships a local chat UI, as a desktop app or in the terminal, plus OpenAI- and Anthropic-compatible endpoints on localhost. Claude Code, Codex CLI, OpenCode, Hermes Agent, and OpenClaw attach through strata connect. Hardware the project lists: NVIDIA RTX 20/30/40/50 series (CUDA 13 engine packages need driver 580 or newer), AMD Radeon on HIP, Intel Arc on Linux, Apple Silicon, and AMD Strix Halo. Release 0.2.11 is Strata-0.2.11-linux-x86_64.AppImage for the desktop app; the terminal path below is the one this quickstart walks through.

QFN Benchmarks Against Local Alternatives

Alibaba’s model card compares Flash-Next with Qwen3.8-27B, DeepSeek-V4-Flash (the 0731 checkpoint), and Claude Opus 4.6 (Max).

Agentic-coding rows on that card used temperature 1.0, top_p 0.95, and a 256K window. DeepSWE reports the higher of the Claude Code and mini-SWE-agent harnesses, and the card says Flash-Next’s higher score is on mini-SWE-agent. SWE-bench Pro re-ran the baselines on a corrected task set; the Claude Opus 4.6 cell is the vendor’s published score, and the other cells are Claude Code.

Two rows on the same card go the other way. NL2Repo-Bench is 48.1 for Flash-Next and 54.2 for DeepSeek-V4-Flash. HLE, judged by GPT-4o, is 35.9 against Claude Opus 4.6 (Max) at 40.0.

For agent work the card’s defaults are preserve_thinking on, so the reasoning trace stays across turns, and reasoning_effort at xhigh. The other levels are medium and low. The Qwen 4 family page tracks what Alibaba has confirmed about this line, including the Qwen Community License 1.0 on these weights. The table above is the part that can be measured today.

Quantization Builds and Sizes

ISTA-DASLab GSQ-RCO

The ISTA-DASLab GSQ-RCO GGUFs use non-uniform quantization. GSQ is low-bit scalar quantization per tensor, and RCO assigns a type per tensor inside a size budget. The method spends almost all of the compression on the routed experts, which are 95% of the searchable weights. Each build is two shards. The second is the n-gram table at about 4.5 bits per weight, and it can be mapped from disk.

Those GPQA figures are ISTA’s harness, not the 91.7 on Alibaba’s card. A Coder variant keeps 256 of the 512 routed experts per layer, at 58.4 GB and 1.89 effective bits per weight. A 0.91 GB BF16 vision projector is the multimodal add-on. The ISTA line stops at IQ3_S; it has no Q4 build.

Unsloth packs in Strata 0.2.11

On 04/11/2026 Unsloth re-uploaded its Qwen3.8-Flash-Next GGUFs as Dynamic v3.1. The 0.2.11 release notes say Strata can download, prepare, and run seven of them from the app, from strata launch --family unsloth, or from a folder you already have. Each file is checked by size and SHA-256 against Unsloth revision d9620119. The last file of every set is the 28.8 GB per-token table. Sets that share a table share the file: UD-IQ1_M with UD-IQ2_XS, and UD-IQ3_XXS with UD-IQ4_XS-v31.

UD-IQ4_XS-v31 is a different file from the UD-IQ4_XS Strata already offered: other tensor types, four files instead of three. UD-Q4_K_XL, UD-Q5_K_XL, UD-Q6_K_XL, and Q8_0 stay on the 30/09/2026 files. The 04/11 UD-Q4_K_XL is the same weights in a four-file split, and Strata accepts that split if you already have it. UD-IQ1_S does not load — one layer mixes two expert formats — so UD-IQ1_M is the 1-bit choice.

UD-IQ4_XS-v31 uses the Q5_0 expert format in 33 layers. The same notes say Strata’s Q5_0 math lands 1.2–1.3% from the exact result, against 2.6–3.0% for llama.cpp’s formula on those tensors. On a Mac, the app rewrites the experts into one GPU file before the first start, so the disk budget is the download plus that file plus a few gigabytes (about 115 GB for UD-IQ1_M).

If the 28.8 GB table stays mapped, the rest of UD-IQ4_XS-v31 is about 64.9 GB before KV cache, which is already past a 64 GB machine under Strata’s pinning rule. UD-IQ1_M’s remainder is about 39.2 GB. That file can fit where IQ3_S cannot, and it is a 1-bit pack, further from the tool-call behaviour measured below at 3 bits.

Quickstart: Install, Launch, and First-Run Setup

A Linux desktop needs an RTX 20/30/40/50 series card or another GPU the project lists, enough RAM for the chosen pack plus KV-cache headroom, and SSD space for the download. CUDA 13 engine packages need driver 580 or newer.

Terminal install:

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh && sh install.sh
strata launch

strata launch asks for the model, the quantization, the context, the storage folder, and the access settings, downloads the engine for the card, starts the server, and shows a monitor with speed, memory, and request data. The default endpoint is http://127.0.0.1:8080.

To see what fits before a 76 GB download, or to start without the prompts:

strata launch --family qwen --model q2_0 --plan      # dry run: what's missing, what fits
strata launch --family qwen --model q2_0 --yes --no-tui   # start in background
strata launch --status
strata connect    # prints the config for Claude Code / OpenCode / Hermes / OpenClaw / ...
strata launch --stop

Those --family qwen --model q2_0 flags are the dry run used for the ISTA Q2_0 build on Ubuntu 26.04. Unsloth packs use --family unsloth and the name from the table above. strata connect prints a provider block per tool. Both API shapes use an OpenAI-compatible base URL (http://127.0.0.1:8080/v1) and the model name the launch prints. Any client that accepts a custom OpenAI-compatible endpoint, including Cursor, uses that pair.

Two settings to verify on first run

1. Sampling. In 0.2.11 a pack config with no sampling block follows the model card. Thinking requests get temperature 1.0, top_p 0.95, and top_k 20. Non-thinking requests get 0.7, 0.8, and 20, with presence_penalty 1.5. Precedence is the request, then the web UI’s Chat settings, then the config block, then the card. Temperature 0 applies only when the request or the block sets it, and those greedy runs are the ones that fell into repetition loops on this checkpoint. If an older pack config still contains "sampling": {"temperature": 0}, replace it with the card’s thinking preset.

2. tool_call_recovery. The key lives in the model run config (strata-*.json). It has been on by default since 0.2.8, and off in 0.2.7 and earlier. Strata’s engine notes, citing 1,462 agent turns, report 98.3% of calls read as intended with the switch and 92.9% without. It accepts four shapes of a call whose name was in the request: a <parameter=NAME> tag, JSON inside <tool_call>, a <function=NAME> block with no <tool_call> wrapper, and a second call inside one <tool_call>. An undeclared name or broken JSON stays text. On 0.2.7 and earlier, set "tool_call_recovery": true. Leave the 0.2.11 default on for agent work.

Keep the server on localhost

The server binds to localhost by default, and the access settings are part of strata launch. The endpoints expose the model and the conversation on that machine, and they were not built for an open network. When another machine needs them, put an authenticated reverse proxy in front of the port.

ss -ltnp | grep :8080    # confirm the listener is 127.0.0.1, not 0.0.0.0
curl -s http://127.0.0.1:8080/v1/models   # the server answers on localhost

Cards with 13 GB of VRAM or less

STRATA_ROUTE_PRIOR=1.5 tells the router to prefer experts already on the card. Strata’s notes for an RTX 3060 12 GB running IQ3_XXS report +15% on story decode and +37% on code decode against the prior left off, and fewer agent tasks passed: 4.5 of 10 on average over 4 runs, against 5.6 over 7 runs with the prior off. Version 0.2.7 turned 1.5 on by itself for discrete NVIDIA cards of 13 GiB or less. Version 0.2.8 turned that back off on 29/10/2026, after repetition and weaker long agent sessions (issue #4402). With the variable unset, 0.2.11 routes as 0.2.6 did. The project points the prior at chat and short answers on a 12 GB card, and away from long agent runs.

Testbed: A 64 GB Desktop Running an Agent

The measured box:

With ISTA IQ3_XXS the resident shard is 47 GB. Because the experts are pinned, about 9 GB remains for everything else. The Hermes VM — the agent stack in the Hermes production setup — plus Cursor leaves about 5 GB, which is not enough page cache or KV cache for a long agent session.

ISTA IQ3_S does not fit this machine as a daily desktop. The 54.8 GB resident shard plus the 28.8 GB mapped n-gram table use the 64 GB. With Cursor running, 3 GB stay free, and the Hermes VM has nowhere to go.

IQ3_S decoded at 100–130 tokens/s, the same speed class as a 27B build on llama.cpp with MTP 3 in the Qwen 3.6 MTP comparison. The pack that fits the RAM budget (IQ3_XXS) is the one that misses the tool-call format. The pack ISTA recommends (IQ3_S) leaves no room for the rest of the desktop.

On IQ3_XXS the model emitted malformed tool-call arguments, invented tool names that were not in the schema, and re-issued the same command three to five times after a short thinking block. The harness treats each miss as a retry, so the session fills with the same error. On this hardware IQ3_XXS is not an agent model.

What the Community Reports

Speed reports for Qwen3.8-Flash-Next on Strata, from issue threads and the 0.2.11 notes:

Three failure modes are written up for low-bit Qwen3.8-Flash-Next in agent workloads.

1. Mangled tool-call emission that tracks the schema’s size. Issue #1431 reports that with 39 function schemas (about 12K tokens) and a 6K system prompt, IQ3_XXS on Strata 0.1.40 produced a usable turn in about 30–50% of 20-plus controlled replays. The model either emitted a tool-call shape the parser could not map — the text landed in content with finish_reason: stop — or closed the thinking block and emitted end-of-sequence with no call and no answer. The same weights on llama.cpp parsed tool calls in all 8-plus consecutive runs. llama.cpp constrains the emission with a PEG grammar and lifts mangled content back into tool_calls. A wrapper that repaired the text afterwards did not push the useful-turn rate above that 30–50% band, because the repair runs after the tokens are already chosen.

2. Config errors and a non-deterministic expert path. Issue #2685 reports an RTX 5070, IQ3_S, and 60 GB of RAM, with tool calls failing through Strata: malformed arguments, hallucinated tools, and retry loops, including after a sampling block and tool_call_recovery were turned on. The same quant through llama serve succeeded on 5 of 5 runs of the same harness. The reporter found that identical seeds diverged in the MMQ expert path during prefill, and that disabling that path made the runs bit-identical. Strata 0.1.44 shipped determinism fixes. Later reports still do not show Strata matching llama.cpp on the same quant.

3. Tool calls left inside the thinking block. Issue #804 reports that a complete tool call inside an unclosed think block comes back as reasoning_content, with empty content and no tool_calls, and the agent loop stops. One user counted about 1 in 100 responses over roughly 1,300 real agent replies, using Hermes with IQ3_XXS, IQ2_XS, and the Coder builds.

Why Sub-Q4 Breaks Agentic Tool Calls

A tool call has to match a wire format. At 3 bits this 125B model misses that format on some turns, and the misses grow with the tool-schema context. A larger schema asks more of the working memory that formats the call, and quantization noise lands on that formatting. Chat turns hide it. Agent turns, which are mostly tool calls, do not.

The same compression hurts more when few parameters are active. The local-LLM comparison for OpenCode measured a dense 27B at IQ3_XXS with about a 5% error rate on a structured agent task. Low-active MoE checkpoints at similar bit widths failed harder: the 35B-A3B had 64.2% errors on IQ3_S and 98.8% on IQ4_XS, and the 122B-A10B had 80% errors on IQ3_S and 90% on IQ3_XXS. Qwen3.8-Flash-Next activates 6B parameters per token, so a 3-bit pack behaves like a small model on a heavy quant. Those OpenCode runs show IQ3 and IQ4 missing the structured task for the 35B-A3B and the 122B-A10B; they do not include a Q5 run of Flash-Next. Strata’s UD-IQ4_XS-v31 has been downloadable since 0.2.11, has no agent-task comparison in the sources checked here, and does not fit the 64 GB box.

The same GSQ-RCO weights fail when they are loaded in llama.cpp and in Ollama. llama.cpp still constrains the emission with a grammar and can lift a mangled span back into tool_calls, so the client sees a call instead of a broken string. Strata’s JSON mode, in the engine notes, is schema prompting plus a check after generation, not grammar-constrained decoding: a bad object returns HTTP 502 with structured_output_failed. tool_call_recovery and the later releases repair the four shapes listed above. They do not change the distribution the model samples from, which is why issue #1431’s wrapper could not climb past the emission rate.

Strata spends RAM to keep the CPU and the GPU in step. On 64 GB, the largest ISTA pack that still leaves room for a desktop and an agent VM is IQ3_XXS, and that pack is below the emission threshold measured here. IQ3_S already decodes in the same band as a 27B model with MTP 3, so the limit on that pack is RAM. A llama.cpp run with a lazy map would commit less RAM and give up the decode rate that makes Strata the faster of the two on this model.

Verdict

Qwen 2.8 Flash Next outperforms Qwen3.8-27B on agentic benchmarks: DeepSWE 58.7 vs 42.2, JobBench 55.7 vs 33.4, and Toolathlon 73.5 vs 67.1. Multi-step work is noticeably more competent. Per-token compute is closer to a small model than a 125B model.

Quantizations below Q4 do not provide reliable tool-call emission for agentic work, failing at rates that produce retry loops on both Strata and llama.cpp. Q4 performance is not yet proven through controlled comparison on agent benchmarks. Q5-class quants are the demonstrated floor for reliable agentic work; the Unsloth UD-Q5_K_XL exists but lacks published benchmarks.

A Mac Studio with an M5 Max and 128 GB unified memory can hold the 105 GB MLX build or a Q4 pack with sufficient headroom for the agent stack. The QFN architecture was designed for this offloading model.

I will continue using in my current setup uses an NVFP4 27B model — 17 GB on the RTX 5090 — with MTP 3, It is achieving 100+ tokens/s with reliable tool calls and no RAM competition. I will return to QFN when a Q4-class quant is benchmarked on agent tasks, when Strata’s emission recovery matures, or when hardware is upgraded.

References

• Niko1221/Strata — the inference engine in this article. The OSDI 2026 hierarchical-KV paper for SGLang is a different project with the same name.
• Strata 0.2.11 release notes — Unsloth Dynamic v3.1 packs, Q5_0 math, and the short-prompt speeds cited above.
• Engine notes (docs/DETAILS.md) — sampling precedence, tool_call_recovery, and the route-prior measurements.
• Qwen/Qwen3.8-Flash-Next — architecture, sampling presets, and the benchmark table.
• ISTA-DASLab Qwen3.8-Flash-Next GSQ-RCO GGUFs — the Q2 and IQ3 builds in the testbed.
• Issue #1431, issue #2685, issue #804, and issue #4402 — the failure reports in the community and small-card sections.

Useful links

• Choosing the Best LLM for Ollama on a 16 GB VRAM GPU — dense-model alternatives when the card is the limit.
• Ollama cheatsheet — the other local server the same GGUFs can be imported into.