Sub-10B Local AI: Quantization and Edge Inference Come of Age

As of mid-2026, the minimum viable local inference stack — 16 GB system RAM, a modern CPU, and either 6+ GB VRAM or Apple Silicon — is standard consumer hardware. The dominant runtime is llama.cpp, written in pure C/C++ with no external dependencies; virtually every higher-level tool (Ollama, LM Studio) shells out to it. Ollama functions as the de facto developer-prototyping layer, while platform-native runtimes (Apple MLX, Meta ExecuTorch, NVIDIA TensorRT Edge-LLM) dominate production deployments.

The dominant quantization formats are:

• GGUF Q4_K_M / Q5_K_M: balanced CPU/GPU, ~92% quality retention vs FP16
• AWQ INT4: ~95% quality retention, hardware-friendly, best for creative/coherence tasks
• GPTQ INT4: up to 5× faster than GGUF on pure CUDA with Marlin kernels
• FP8: GPU-native, emerging on RTX 4090/5090, lossless for many models

Top models deployable on 8 GB RAM as of June 2026: Phi-4-mini (3–4B, leads MMLU at 67.3%), Qwen3.5-4B (outperforms GPT-OSS-120B on IFEval), Gemma 4 4B (Apache 2.0, multimodal), Llama 3.2 3B, Mistral 7B Q4_K_M (~4.5 GB VRAM). Local inference is 30–150× cheaper per token than cloud APIs once hardware is amortised, and end-to-end latency under 100 ms is routine on mid-range hardware.

Quantization frontier

Microsoft BitNet b1.58 2B4T (April 2025) is the first natively trained 1-bit (ternary) model at 2B parameters, trained on 4 trillion tokens under MIT license. Weights are quantised to {−1, 0, +1} at training time — not post-hoc — eliminating floating-point multiply-accumulate entirely. It achieves ~12× lower energy per inference vs Qwen2.5 (0.028 J vs 0.347 J) and competes with full-precision models 2–4× its memory size. A January 2026 CPU kernel update added 1.15–2.1× additional throughput. BitNet a4.8 extends this with 4-bit activations (INT4/FP4), matching b1.58 quality at lower inference latency.

Google TurboQuant (2026) compresses KV caches to 3 bits with zero measured accuracy loss — 6× memory reduction — directly attacking the long-context bottleneck. QuantSpec (Feb 2025) combines self-speculative decoding with a hierarchical 4-bit quantized KV cache, sustaining >90% token acceptance rates and delivering ~2.5× end-to-end speedups.

Hardware frontier

Apple M5 Neural Accelerators (integrated one per GPU core) show Qwen3-14B-4bit at 4.06× faster time-to-first-token vs M4, with 1.19× faster generation. Qualcomm Snapdragon X Elite Gen 2 targets 65 TOPS on its Hexagon NPU and claims local inference of models up to 13B parameters without cloud connectivity; volume OEM availability is slated for Q3 2026. NVIDIA Jetson Thor (announced Feb 2025) delivers >800 TOPS for robotics/industrial edge.

2022: GPTQ establishes the PTQ engineering baseline — post-hoc weight quantisation via approximate second-order optimisation, enabling 4-bit LLaMA on a single A100.

Jan 2023: llama.cpp ships; Georgi Gerganov's C/C++ port of LLaMA inference runs a 7B model on a MacBook CPU, sparking the local-AI movement. GGUF format emerges as a portable, self-describing container for quantised weights.

2024: AWQ (Lin et al., MLSys 2024 Best Paper) from MIT's Song Han lab demonstrates that protecting just 1% of "critical weights" — identified by activation magnitude — makes 4-bit quantisation nearly lossless and hardware-friendly without backpropagation. SqueezeLLM (ICML 2024) isolates outliers into a sparse matrix alongside non-uniform quantisation, achieving lossless 3-bit. QuIP# (ICML 2024) adds Hadamard incoherence and lattice codebooks for extreme compression. Collectively, these make sub-4-bit inference viable.

Nov 2024: BitNet a4.8 (Microsoft Research) enables 4-bit activations for 1-bit LLMs, accelerating inference without quality loss.

Apr 2025: Microsoft releases BitNet b1.58 2B4T, the first openly released natively trained 1-bit model; proof-of-concept for a train-time rather than post-hoc quantisation paradigm.

Oct 2025: ExecuTorch 1.0 GA ships — production-ready on-device PyTorch inference across CPU/GPU/NPU, already powering Meta's Instagram, WhatsApp, Quest 3, and Ray-Ban Smart Glasses.

Jan 2026: NVIDIA JetPack 7.1 ships TensorRT Edge-LLM, an open-source C++ runtime purpose-built for Jetson-class edge boards. Apple M5 ships. Qualcomm Snapdragon X Elite Gen 2 announced (65 TOPS). Google releases Gemma 4 (Apache 2.0, multimodal from 4B+) April 2026.

Market size

The global edge AI market is valued at $25–30B in 2025 across competing estimates, with CAGR projections of 20–29% pointing to $118–165B by 2033–2035. The AI chip for edge inference sub-segment was $12.4B in 2025, forecast $84.6B by 2034 at 23.7% CAGR. The edge AI hardware market (boards, modules, integrated systems) is separately estimated at $30.74B in 2026 growing to $68.73B by 2031.

Private investment

Edge AI startup funding grew from $1B (2022) to $4.35B (2025), with average deal size jumping from $33M (2023) to $84M (2025). Breakdown of the $7B+ raised across tracked categories:

• Humanoid/embodied AI: $2.9B (41%) — Figure ($1B), Skild AI ($500M), FieldAI ($405M)
• Edge AI chips/semiconductors: $1.4B (20%), most active across all four years
• Automotive edge AI: Wayve $1.2B Series D, Feb 2026, the single largest disclosed round

Public/government capital

Qualcomm announced a dedicated AI Engineering Center in partnership with HUMAIN (Saudi Arabia's national AI program, Nov 2025). Intel's Core Ultra NPU investment tracks its PC AI strategy directly; Intel achieved first full NPU support in MLPerf Client v0.6.

Runtime and tooling

llama.cpp (ggml-org): the de facto standard for local inference; powers Ollama, LM Studio, and scores of wrappers. Supports GGUF, AWQ, GPTQ, FP8; backends for CUDA, Metal, Vulkan, ROCm, SYCL. ~35% speed gains in recent releases via NVFP4, FP8, and GPU token sampling.

Ollama: developer-facing HTTP daemon; handles model pulling, GGUF management, and multi-model serving. Positioned as prototyping layer; not suited for embedded production runtimes.

Unsloth: fine-tuning + quantisation; ~70% less VRAM and 2× faster fine-tuning than stock PyTorch; produces higher-quality quantisations than standard pipelines at equivalent file size. Now integrates speculative decoding via MTP GGUFs.

Apple MLX: unified-memory framework; no data copies between CPU and GPU. Qwen3-14B-4bit at 50 tok/s on M3 Max; M5 accelerators push this further. Preferred runtime for Mac production deployments over llama.cpp.

ExecuTorch (Meta/PyTorch): production on-device runtime; Qualcomm Hexagon NPU delegate, ARM KleidiAI delegate. Powers 1B+ devices across Meta's app ecosystem.

Qualcomm AI Hub: 175+ pre-optimised models, 1,800+ enterprise customers including Meta, Samsung, Amazon; compiles and profiles models against physical Snapdragon devices in the cloud before on-device deployment.

NVIDIA TensorRT Edge-LLM (JetPack 7.1): open-source C++; targets Jetson Orin/Thor with hard memory budgets and sub-10 ms latency requirements for robotics perception stacks.

Competitive dynamics

The consumer edge inference market is rapidly commoditising at the runtime layer; differentiation is shifting to (a) hardware-native quantisation kernels (Qualcomm Hexagon vs ARM KleidiAI vs Apple ANE) and (b) model quality at fixed parameter counts — where Qwen3, Phi-4, and Gemma 4 compete aggressively. Chip incumbents (Qualcomm, Apple) have a structural advantage as NPU instruction sets are proprietary and inference optimisation is deep.

Key labs and groups

• MIT HAN Lab (Song Han): AWQ, SpAtten, StreamLLM — led activation-aware weight quantisation
• UC Berkeley ICSI / BAIR (Kurt Keutzer, Trevor Darrell): SqueezeLLM, QuIP#; focus on dense-and-sparse and incoherence-based quantisation
• Microsoft Research: BitNet series; the only group producing trained (not post-hoc) 1-bit models at non-trivial scale
• Apple ML Research: MLX framework, Neural Accelerator architecture; benchmarked in arxiv:2510.18921
• Qualcomm AI Research: Hexagon NPU kernel optimisation, ExecuTorch delegate contributions
• Google DeepMind: Gemma family, TurboQuant KV compression, LiteRT (mobile runtime)

Key papers

• AWQ (Lin et al., 2023/MLSys 2024): activation-aware 1% weight protection → near-lossless 4-bit
• SqueezeLLM (Kim et al., ICML 2024): dense-sparse 3-bit quantisation
• QuIP# (ICML 2024): Hadamard incoherence + lattice codebooks for extreme compression
• BitNet a4.8 (Nov 2024): 4-bit activations for ternary weight models
• QuantSpec (Feb 2025): hierarchical 4-bit KV cache + self-speculative decoding, ~2.5× speedup
• ITQ3_S (arxiv 2603.27914): 3-bit inference via interleaved ternary quantisation with rotation-domain smoothing
• Edge Deployment of SLMs: CPU vs GPU vs NPU comparison (Nov 2025): systematic benchmarking across RaiderChip NPU, Intel CPU, NVIDIA GPU

Open problems

1. Sub-3-bit post-training quantisation at 7B scale without perplexity degradation on domain-specific tasks
2. Reliable structured output (JSON, function-calling) at ≤3B parameter scale — Llama 3.2 3B fails ~44–52% of JSON parse attempts vs ~100% at 7B+
3. NPU-native quantisation kernels beyond INT4: no standardised FP4 / INT2 NPU ABI across Qualcomm, Intel, Apple
4. Long-context (>32K token) inference on constrained RAM: KV cache remains the primary bottleneck even at 3-bit compression

Near-term (0–12 months) — confidence: high

Snapdragon X Elite Gen 2 (65 TOPS) enters volume production Q3 2026, making 13B model inference a standard Windows-on-ARM feature. Expect benchmark pressure that forces 7B models toward 35–50 tok/s on consumer NPUs. GGUF-format FP4 kernels will land in llama.cpp following NVIDIA RTX 5000-series adoption. BitNet-style training pipelines will be integrated into Unsloth, lowering the barrier for custom 1-bit fine-tunes. Speculative decoding (via QuantSpec/MTP-GGUF) becomes default-on in Ollama and LM Studio. Google TurboQuant 3-bit KV cache reaches framework integration, enabling 64K+ context on 16 GB RAM.

Mid-term (1–3 years) — confidence: moderate

7B-class models hit ≥40 tok/s on mid-range NPUs (~$400 consumer hardware), making local inference indistinguishable in latency from fast cloud APIs for single-user workloads. Smartphone on-device inference goes mainstream: Samsung and Apple will ship 4–7B models as persistent system agents on flagship devices. The inference framework landscape consolidates around 2–3 winners (MLX on Apple, ExecuTorch on Android/embedded, llama.cpp for heterogeneous/Linux). Sub-3-bit post-training methods mature enough for production recommendation/coding tasks. Edge AI chip market approaches $40B as automotive and robotics deployments scale.

Long-term (3–10 years) — confidence: low

Purpose-trained ternary/binary models (BitNet successors) at 10–30B parameters close to within 5% of fp16 quality on general benchmarks, redefining the efficiency frontier. By volume, edge inference surpasses cloud inference as the dominant AI compute mode globally, driven by privacy regulation, latency requirements, and connectivity constraints in emerging markets. Bespoke edge inference ASICs become commodity in consumer electronics (TVs, cars, wearables). The line between "model" and "OS component" blurs: OS-level inference runtimes manage model sharding across CPU/NPU/GPU transparently. Uncertainty is high here: cloud incumbents have strong economic incentives to retain inference centrally, and hardware-software co-design timelines routinely slip.