A 4B Model Matching GPT-5-mini running Locally

The Cloud API Your Sensitive Data Is Quietly Funding

Your legal team has a question about the merger documentation. Your compliance officer needs to cross-reference three years of internal policy against a new regulatory filing. Your support engineers are trying to pull accurate answers from 40,000 pages of product documentation. The RAG system you built can handle all of this. There is one problem: every query, every retrieved document fragment, every generated answer is traveling through an external API to a model running on someone else's infrastructure. For most organisations, nobody noticed this was a trade-off. It was just the default.

Researchers at Siberian Neuronets LLC published a study in June 2026 that forces a rethink of that default. They evaluated 17 small language models as the generation stage of a RAG system, running entirely on CPU hardware with no GPU acceleration, within a 16 GB RAM ceiling. The headline result is harder to dismiss than the category label suggests. Qwen3-4B-Instruct-2507-Q5KM, a 4-billion-parameter quantised open-source model, achieved an Answer Relevance score of 0.89 against GPT-5-mini's 0.88 in the same RAG setup. A model that fits on a laptop, costs nothing per query, and never sends your data anywhere matched a frontier proprietary model on the metric that captures whether answers actually address what was asked.

The Correctness gap between the best SLM (Qwen3-8B, scoring 0.72) and GPT-5-mini (0.73) was 0.01 absolute. The Faithfulness gap was more meaningful: GPT-5-mini scored 0.89 against the best SLM's 0.83, a 0.06 difference that matters in applications where grounding every claim to retrieved context is non-negotiable. These are the numbers as they stand, on a Russian-language benchmark, with a uniform prompt applied across all models. They are not a clean sweep for local deployment. They are a more honest picture.

The Infrastructure Assumption That Has Been Running Unchallenged

The conventional architecture for enterprise RAG looks like this: a lightweight embedding model retrieves relevant chunks from a local knowledge base, and then a large language model, accessed via cloud API, does the generation work. The retrieval half runs locally. The generation half does not. This split emerged from a real constraint: embedding models are small and efficient enough to run anywhere, while models capable of sophisticated generation were, until recently, too large to run without serious hardware.

What changed is the combination of aggressive quantisation, architectural improvements in smaller model families, and the shift from treating "capable of generation" as synonymous with "tens of billions of parameters." The 4-bit and 5-bit quantisation formats used in this study (Q4KM and Q5KM) compress model weights enough to fit meaningful models into standard RAM without requiring dedicated GPU memory. The question the paper actually answers is whether this compression cost, in terms of generation quality, is acceptable for production RAG workloads. On most metrics, the answer is closer to yes than the industry has been acting.

The practical consequence matters for specific categories of organisation:

• Healthcare systems handling patient records that cannot legally traverse external APIs
• Law firms and in-house legal teams where client confidentiality prohibits cloud processing of case documents
• Government and defence agencies with data sovereignty requirements that rule out foreign-hosted inference
• Enterprises in cost-constrained environments where GPU infrastructure procurement is not budgeted and cloud API costs at scale are prohibitive

For these organisations, the choice has not been between good AI and great AI. It has been between compliant AI and non-compliant AI. This paper suggests the quality of the compliant option is now much closer to the non-compliant one than it was 18 months ago.

What the Benchmark Actually Measured, and Where to Read It Carefully

The study constructed a 500-sample Russian-language RAG benchmark drawing from five datasets, including DaNetQA, SberQuAD, a proprietary dataset built from conference lecture presentations, and two others. Each dataset contributed 100 samples, with question types classified across factoid, reasoning, evidence-based, comparison, experience-based, and instruction categories. Pairwise cosine similarities between datasets ranged from 0.06 to 0.12, confirming genuine diversity rather than dataset overlap. The mean question complexity was rated 4.94 out of 10 by an independent classifier, placing these in the moderately challenging range rather than the PhD-tier benchmark territory.

The evaluation framework deserves attention because it is doing a lot of work here. Traditional lexical metrics like BLEU and ROUGE measure word overlap, not meaning. They systematically fail to detect correct paraphrases and reward surface-level similarity over actual correctness. The paper instead uses an LLM-as-Judge ensemble: three judge models (GPT-5-mini, Qwen3-8B, and GLM-4.7) independently score each generated answer across four metrics, Correctness, Answer Relevance, Context Relevance, and Faithfulness, on a 0 to 1 continuous scale. The inter-rater reliability of the selected judges was measured using the Intraclass Correlation Coefficient, reaching 0.96, which is a strong signal that the judges are agreeing on what good looks like rather than producing random noise.

The judge selection process itself was validated by constructing intentionally broken examples: mismatched questions, answers, and contexts from different datasets, designed to be obviously wrong. The Average Bad Score metric measured how often each judge gave high marks to these broken examples. The worst performer was Gemma-3-4B-it, which scored broken answers generously (ABS = 0.22 and a Faithfulness correlation of just 0.37) and was excluded. The selected judges all had ABS at or below 0.03, meaning they rarely rewarded the wrong answer.

Two important caveats belong alongside these numbers. First, no statistical significance testing is reported. There are no confidence intervals, no standard deviations, no p-values. A 0.01 difference between a top SLM and GPT-5-mini on any single metric should be read as approximate parity, not a definitive ranking. Second, all models received the same uniform prompt. The authors acknowledge this themselves: prompt customisation for individual model architectures could change the results, likely upward for models that are currently underperforming their actual capability.

The Model Families That Separated From the Pack, and the Ones That Collapsed

The performance spread across 17 SLMs was wider than the top-line numbers suggest. The Qwen family dominated consistently across parameter sizes, with Qwen3-8B-Q4KM reaching the highest SLM Correctness (0.72) and Faithfulness (0.83), and Qwen3-4B-Instruct-2507-Q5KM achieving the best Answer Relevance (0.89) at a CPU latency of 70.9 seconds per query. These are the models the paper's authors selected for their own production system.

The performance floor is worth noting because it is low. Llama-2-7B-Chat-Q4KM scored 0.32 on Correctness, 0.46 on Answer Relevance, and 0.42 on Faithfulness. That is not a minor underperformance. DeepSeek-R1-Distill-Qwen-7B-Q4KM combined the worst quality-to-latency ratio of any 7B model: a Correctness score of 0.40 alongside a CPU latency of 297.4 seconds. The distillation process that made DeepSeek-R1 a compelling reasoning model appears to have extracted something that does not transfer cleanly to RAG generation tasks, at least in this setup.

The latency numbers for local CPU inference are real and worth sitting with. Seventy seconds per answer is not suitable for interactive chat interfaces expecting sub-second responses. It is entirely suitable for asynchronous document Q&A, overnight compliance review workflows, batch knowledge-base queries, and any application where the user submits a question and returns to the result rather than expecting a real-time exchange. The use case shapes whether the latency is a dealbreaker or irrelevant.

The Finding That Reframes the Retrieval Benefit

One of the more telling data points in the study is what happens to GPT-5-mini when you remove the retrieved context. In no-context mode, GPT-5-mini's Correctness score dropped from 0.73 to 0.47, an absolute decline of 0.26, representing a 55% relative fall. Answer Relevance dropped only marginally, from 0.88 to 0.86, which the authors note suggests that metric is largely insensitive to whether context is present.

The retrieved context, not the generative model's parameter count, is carrying most of the quality weight in a RAG system. This is the insight that reframes the infrastructure question. If the retrieval stage is doing its job, the generator's role is substantially constrained: synthesise the provided evidence into a coherent answer. The authors' phrasing captures it precisely. "Generating correct responses requires reliance on external context rather than merely leveraging inherent model knowledge." A model that is good at following retrieved evidence and staying grounded in it is what a RAG generator needs to be, not a model that can reconstruct facts from parametric memory.

This is also where the Faithfulness gap between SLMs and GPT-5-mini (0.83 vs. 0.89) deserves the most attention. Faithfulness measures whether the generated answer stays grounded in the retrieved documents rather than drifting into the model's own confident inferences. A 0.06 gap on this metric is meaningful in high-stakes applications. In a legal or compliance context, an answer that reads fluently but synthesises beyond the retrieved evidence is a liability, not a feature. Whether that 0.06 gap holds under prompt optimisation for individual SLMs is an open question the uniform-prompt methodology cannot answer.

What Executives Running Sensitive Workloads Should Reassess

This is a benchmark study, not a deployment recipe. The findings are specific to Russian-language tasks, use a single uniform prompt that likely understates what individual models can achieve, and report no statistical significance testing. The performance margins at the top of the SLM rankings are narrow enough that a different benchmark, a different prompt, or a different retrieval setup could reorder them. Take the architecture conclusion, not the model ranking, as the durable output.

The architecture conclusion is this: a fully on-device RAG system, running on CPU hardware within a standard 16 GB RAM budget, can now achieve generation quality that is competitive with frontier proprietary models on the metrics that matter most for document synthesis workloads. The residual gap is on Faithfulness, which is addressable through retrieval quality improvements and model-specific prompt engineering, two areas the paper explicitly flags as future work.

For executives running data-sensitive operations, the immediate strategic question is not which specific model to deploy. It is whether the privacy and cost calculus of cloud API dependency is still accurate given what local inference can now deliver. Organisations that benchmarked on-device AI quality 18 months ago and concluded it was not competitive enough may have reached a correct conclusion at the time. The same conclusion reached today, without revisiting it, is increasingly likely to be wrong.

The 4-billion-parameter model running on a CPU is not a compromise. It is an architecture decision with a specific trade-off profile, and for a meaningful share of enterprise workloads, that trade-off now resolves in favour of local deployment. The question is whether your organisation noticed.

Agents Applied covers applied AI research for executives and senior technologists. Each edition translates one paper into the operational implications that matter.