Inference Economics: Speed and Cost Per Token as the New Competitive Moat

Token prices for GPT-4-class capability have collapsed from roughly $20 per million tokens in late 2022 to ~$0.40 per million in early 2026 — approximately a 1,000× reduction in three years. Despite this, enterprise generative AI spend grew from $1.7B (2023) → $11.5B (2024) → $37B (2025), a 22× increase coinciding precisely with the price collapse, textbook Jevons dynamics. Token consumption grew 450% in 2024–2025 even as per-token prices halved.

The inference provider landscape expanded from 27 vendors in early 2025 to ~90 by end-2025, then began consolidating — most notably NVIDIA's $20B licensing deal / acqui-hire of Groq in December 2025. Inference now accounts for roughly two-thirds of total AI compute spend, and the software stack has correspondingly matured: HuggingFace TGI moved to maintenance mode in March 2026, directing users to vLLM, SGLang, and llama.cpp.

On the model side, open-weight systems from DeepSeek, Moonshot, and Qwen now post coding scores within 2–3 percentage points of the best closed models, while maintaining a 6–7× price advantage on output tokens. The capability-per-dollar arbitrage has permanently reframed the question from "which model is smartest" to "which stack serves it most cheaply at scale."

Speed: hardware frontier

Cerebras CS-3 sustains ~2,100–3,000 tokens/second on Llama 70B-class models — roughly 6× faster than Groq's LPU on identical models, which itself was the previous speed leader. Cerebras achieved this via wafer-scale integration eliminating inter-chip communication overhead. Canadian startup Taalas claims 16,960 tokens/sec per user on Llama 3.1 8B — ~48× a B200 — though independent verification is pending. NVIDIA's Blackwell GB200 NVL72 delivers 30× faster LLM inference than H100 via NVLink rack-scale fusion; the next-gen Vera Rubin (GTC 2026) claims an additional 5× over B200 and 10× reduction in cost per generated token.

Efficiency: model architecture

DeepSeek's architecture is the most influential efficiency demonstration: 671B total parameters but only 37B active per token via MoE routing, plus multi-head latent attention (MLA) reducing KV-cache memory to 5–13% of prior methods. This yielded inference costs 14–29× cheaper than Western competitors on comparable outputs. DeepSeek V4 (April 2026) extended these gains, with Forbes noting the "next AI race is about efficiency."

Software: serving stack

Speculative decoding has matured from research to production standard — NVIDIA demonstrated 3.6× throughput improvement on H200s with EAGLE-style draft models achieving ~80% acceptance rates. TensorRT-LLM achieves 15–30% higher throughput than vLLM on H100s at the cost of compilation overhead; vLLM with PagedAttention holds 85–92% GPU utilization under concurrent multi-user load. Combined optimizations (quantization + speculative decoding + FlashAttention + continuous batching) routinely yield 5–10× cost reduction and 3–5× latency improvement versus naive serving.

2022 Q4 — GPT-4 API launches at ~$0.06/1K tokens output ($60/M), setting the initial commercial benchmark. Training cost and RLHF expertise are the moats.

2023 — Meta releases Llama 1 and 2; open-weight race begins. Community builds llama.cpp enabling CPU inference on consumer hardware. Inference-as-a-service providers (Together AI, Anyscale) emerge as middlemen arbitraging GPU access.

2024 — Groq's LPU demonstrates that purpose-built inference silicon can outrun GPUs on latency by >10×. Speculative decoding papers proliferate; AWQ becomes default quantization. Inference costs decline ~10× through the year. Enterprise GenAI spend hits $11.5B, double 2023, despite falling prices.

January 2025 — DeepSeek R1 releases and immediately benchmarks within striking distance of GPT-4o while running at a fraction of the cost. The "DeepSeek shock" triggers a single-day NVIDIA stock drop and forces a global reassessment of compute-cost assumptions. MoE architecture and MLA become the new templates.

Early–Mid 2025 — Fireworks AI closes $250M Series C at $4B valuation. Inference providers multiply to 90+; API prices drop 80% through the year. Cerebras files for IPO reporting $1B+ in 2025 revenues. Hyperscalers roll out captive ASICs: AWS Trainium 3, Google TPU Ironwood, Microsoft Maia 200.

Late 2025 — NVIDIA announces $20B Groq deal. HuggingFace TGI enters maintenance mode March 2026; vLLM and SGLang absorb its user base. Open-source benchmark parity is reached within 2–3 points on major coding evals.

2026 H1 — DeepSeek V4 continues the efficiency-first thesis. Fireworks AI hits $800M ARR and seeks $15B valuation. NVIDIA Vera Rubin announced at GTC 2026. Apple's AFM 3 marks a turning point for on-device inference boundary economics.

Inference market size: $103B in 2025, projected $255B by 2030, growing ~20% CAGR. Inference is now ~two-thirds of total AI compute spend, having overtaken training.

Private market leaders:

• Fireworks AI: $250M Series C (Oct 2025, $4B valuation, led by Lightspeed + Index + Sequoia); $800M ARR as of May 2026; in talks for new round at $15B valuation
• Cerebras: Filed IPO with $1B+ 2025 revenues; secured 750MW of compute supply to OpenAI through 2028
• Groq: Acquired/licensed by NVIDIA for $20B (December 2025)
• Baseten: Approaching decacorn status; growing "multiples on a $100M+ baseline" as of H1 2026
• Together AI, Modal, Parasail: Significant undisclosed rounds in the "abstract the asset" tier

Hyperscaler ASIC investment: AWS, Google, Microsoft, and Meta have each committed multi-billion dollar silicon programs. Google's TPU v6e Trillium available at $2.70/chip-hour (on-demand), claiming 4× better price-performance than H100 for LLM workloads. Microsoft Maia 200 (140B transistors, 3nm, deployed January 2026) claims 30% token cost reduction. These chips are captive — unavailable to outside customers.

Public markets: NVIDIA's data center revenue exceeded $100B in FY2026, with the inference shift to Blackwell architectures driving the upcycle. Goldman Sachs projects token consumption growing 24× by 2030.

Competitive tiers

The market has split into two structural strategies: "own the asset" (Lambda, Crusoe, Nscale, Nebius — building raw GPU/ASIC capacity) and "abstract the asset" (Baseten, Fireworks AI, Modal, Parasail, Gimlet Labs — making that capacity easier and cheaper to consume via software optimization layers).

Inference-as-a-service

DeepInfra hosts the widest open-model catalog at $0.039 input / $0.19 output per 1M tokens for Llama 70B (June 2026). Groq at $0.15 / $0.60 and Cerebras at $0.85 / $1.20 trade off speed vs. cost differently — Cerebras ships more tokens/sec but at higher per-token price, roughly equalizing cost-per-second-of-output. Chinese providers run 14–29× cheaper on input/output for equivalent quality, creating pricing pressure every Western provider must respond to.

Model providers

Closed-source labs (Anthropic, OpenAI) lead in funding ($37.5B vs. open-source's $14.9B since 2020) and revenue, but their raw model capability advantage is measured in single-digit benchmark points rather than order-of-magnitude gaps. The leverage is migrating to the application and infrastructure layers around the model.

Vertical AI

The shift from "base model as moat" to "vertical AI and inference infrastructure as moat" is accelerating. Inference startups that build software efficiency on top of commodity GPUs capture margin that base model providers cannot, because model weights commoditize but optimized serving stacks require sustained engineering investment.

On-device

Apple Silicon (M4/M5 with unified memory) offers the best performance-per-dollar for local inference for most developers: $0 marginal token cost, full privacy, no idle charges. The economic crossover point — where on-device beats cloud — is roughly 10–50% GPU utilization; below that, on-demand cloud becomes prohibitive; above it, owned cloud or on-device converge. Apple's AFM 3 has pushed the capability threshold high enough that on-device handles the majority of routine tasks.

Key labs

DeepSeek (Hangzhou Deepseek Artificial Intelligence) continues to produce the most influential efficiency-focused papers. Their MoE + MLA architecture stack is now the template every serious inference-efficiency researcher starts from.

Together AI has published foundational research on efficient inference at scale including FlashAttention variants and continual batching improvements. UC Berkeley's Sky Computing Lab (vLLM team) remains the gravitational center of open inference serving research.

Key papers and techniques (2025–2026)

• Speculative decoding + quantization — arxiv 2505.22179: QSpec, QuantSpec, ML-SpecQD integrate 4-bit quantization with speculative drafting, achieving near-full speculative speedup at quantized precision
• AdaSD (arxiv 2512.11280): adaptive speculative decoding that modulates draft length based on predicted acceptance rate, outperforming fixed-draft methods
• DFloat11 (arxiv 2504.11651): lossless 70% size compression via dynamic-length float encoding — identical accuracy at 30% smaller memory footprint
• SlimMoE (arxiv 2506.18349): structured MoE pruning enabling inference on smaller hardware without fine-tuning
• Cascade distillation (Mistral): produces a model family from a parent using 1–3T training tokens vs. 15–36T for full training; Ministral 3 outperforms Qwen 3 on MATH at equivalent size

Open problems

1. Long-context inference cost: quadratic attention still dominates; linear-time alternatives (Mamba, RWKV) struggle on quality parity
2. Speculative decoding for large batch sizes: acceptance rates degrade under high concurrency, limiting production applicability
3. Cross-device memory arbitrage: running large models across heterogeneous memory (CPU+GPU+NPU) without latency cliffs
4. MoE load balancing at inference time: uneven expert utilization wastes GPU memory; dynamic routing improvements are active research
5. Benchmark-to-production transfer: MMLU and HumanEval scores correlate poorly with per-token economics on agentic multi-step workloads

Near-term (0–12 months) — high confidence Token prices continue falling 3–5× per year, decelerating from the prior 10×/year pace as low-hanging optimization fruit is harvested. Fireworks AI's $15B round likely closes, signaling peak private market enthusiasm for inference middleware. NVIDIA Vera Rubin systems begin shipping; early adopters report 3–5× throughput gains over Blackwell. Speculative decoding becomes mandatory in all production vLLM/SGLang deployments. On-device inference crosses 50% of total AI inference requests on Apple devices as AFM 3 handles calendar, email, and document tasks locally. Chinese inference providers face access restrictions in EU and US government procurement, pushing buyers toward Western alternatives with equivalent efficiency.

Mid-term (1–3 years) — moderate confidence The "efficiency gap" between open and closed models reaches near-zero on most benchmark categories. Model weights become a commodity input, and the sustainable margin in the AI stack moves firmly to (a) optimized inference software stacks, (b) proprietary data pipelines, and (c) application-layer verticalization. Hyperscaler captive silicon (TPU Ironwood, Trainium 3, Maia 200) accounts for 30–40% of total inference compute, permanently reducing NVIDIA's pricing power in cloud inference. On-device inference SoCs from Apple, Qualcomm, and MediaTek become the primary deployment target for consumer AI, with cloud as fallback for long-context and reasoning-heavy tasks. A Jevons-driven demand surge produces a second GPU supply crunch around 2027 as agentic workloads (multi-step, multi-call) multiply effective token consumption.

Long-term (3–10 years) — speculative The arxiv paper "The End of the Foundation Model Era" argues that the model layer fully commoditizes and value accretes to world models, agent orchestration, and inference infrastructure. If correct, the analogy is cloud compute: AWS/Azure/GCP are enormously profitable not because they invented CPUs but because they built the cheapest, most reliable way to run them. Inference specialists who achieve and sustain 2–5× cost efficiency advantages over commodity GPU clouds have durable moats — hardware amortizes slowly, software stacks compound, and customer switching costs accumulate. The wildest card is optical compute and analog inference (Lightmatter, Rain AI) which could yield another 10–100× efficiency step change if manufacturing yields mature, but commercial viability before 2030 is uncertain.