When GPT-4 launched in March 2023, running a single long conversation through it cost roughly $0.06 per thousand output tokens. For a typical enterprise workflow — contract review, customer support triage, document summarization — that meant inference costs in the range of several dollars per user per day. Interesting as a demo. Economically implausible at scale.
Eighteen months later, models matching or exceeding GPT-4's benchmark performance on most tasks were available for a fraction of a cent per thousand tokens. Today, in mid-2026, the cost of that same compute has dropped by a factor of somewhere between 80 and 150, depending on the workload and provider. The trajectory is not flattening. If anything, it is steepening.
This is not a gradual trend. It is a collapse. And most enterprise software buyers, product teams, and investors are still pricing AI as though we are living in Q1 2023.
Three forces driving the free-fall
The cost drop is the product of three concurrent forces, each compounding the others.
Model architecture efficiency. The first generation of frontier LLMs were dense transformers — every parameter activated for every token. Mixture-of-Experts (MoE) architectures changed the equation fundamentally: only a fraction of a model's parameters engage for any given input. A 100-billion-parameter MoE model can match the output quality of a dense 70-billion-parameter model while consuming a third of the compute per inference call. Layered on top of this, techniques like speculative decoding, flash attention, and aggressive quantization have squeezed further efficiency out of the same hardware. The models themselves got dramatically cheaper to run before any hardware improvement was even factored in.
Hardware supply and competition. In 2023, H100 GPUs were months-backordered and effectively rationed. The combination of TSMC capacity expansions, AMD's MI300X and MI325X entering serious production, and hyperscalers deploying custom silicon at scale — Google's TPU v5, AWS Trainium2, Microsoft's Maia — fundamentally changed the supply picture. When inference hardware goes from scarce to abundant, and when three different vendors are competing for the same workload, per-token prices respond accordingly.
Open-weight model proliferation. Meta's Llama series, Mistral's releases, and a wave of fine-tuned derivatives changed the competitive floor. Any enterprise with modest engineering resources can now self-host a model that outperforms the frontier of three years ago, on hardware that costs a few hundred dollars a month to rent. The existence of that option caps what API providers can charge. Anthropic, OpenAI, and Google are not pricing against each other in isolation — they are pricing against the realistic cost of a customer hosting their own model.
What this breaks: the per-query pricing model
The first casualty of collapsing inference costs is the per-query, pay-as-you-go business model that AI startups spent 2023 and 2024 building around.
Per-query pricing made intuitive sense when compute was expensive and unpredictable. You charged customers a margin on your cost. Simple. But when the underlying cost approaches zero, per-query pricing loses its anchor. Customers cannot budget against it. It creates adversarial incentives — developers optimize prompts to minimize token counts rather than maximize output quality. And it makes the AI layer look like a cost center that scales with usage rather than a fixed-cost capability that scales for free.
The model that is winning is flat-rate seat-based pricing with AI capabilities bundled in. Not "AI add-on" pricing, not usage tiers — just: this product now includes AI, the same way it includes a database and a CDN. Companies still charging separately for AI features by mid-2026 are watching those line items become increasingly difficult to justify in procurement conversations.
What this unlocks: the embedded AI floor
The more consequential shift is on the demand side. At $0.06 per thousand tokens, AI inference was a feature you turned on carefully. At $0.0004 per thousand tokens, it is infrastructure you leave running.
This changes the architecture of software entirely. When the cost of an LLM call is indistinguishable from the cost of a database query, you stop asking "should we add AI here?" and start asking "why would we not?" Every form field becomes an opportunity to validate intent. Every report becomes an opportunity to surface an anomaly. Every user interaction becomes an opportunity to reduce friction. The unit economics that made these investments marginal three years ago now make them obvious.
Enterprise adoption numbers reflect this. The barrier was never primarily about model capability — it was about whether the cost-per-value calculation worked at production scale. For most use cases, that calculation flipped somewhere in late 2024 and has only improved since.
The curve is not done
The counterintuitive thing about exponential curves is that the steepest part is always ahead of you, not behind. Dedicated inference chips from multiple vendors are entering the market simultaneously. Model distillation is getting more sophisticated, making smaller models more capable. The open-weight ecosystem is accelerating. Hyperscaler infrastructure spending continues at record levels.
The teams and companies with an advantage right now are not the ones with the biggest AI budget — it is the ones who have already internalized that inference cost is not a meaningful constraint and are designing products accordingly. The cost curve did not just make AI affordable. It made AI invisible in the right way: present everywhere, justified everywhere, invisible in the line items.
That shift happened faster than almost anyone's roadmap anticipated. The companies adjusting fastest are the ones who will own the next phase.