The Inference Economy

Here's a number that should make you stop and think:

80-90% of AI's lifetime costs come from inference, not training.

Let me say that again, because most people—including most investors, most tech executives, and most policymakers—don't understand this yet.

Training a frontier AI model is expensive. GPT-4 reportedly cost over $100 million to train. DeepSeek's models cost a fraction of that, but still millions. These numbers make headlines. They're big, scary, impressive.

But they're also one-time costs.

Inference—the process of actually using the model, of serving billions of queries to millions of users—is a continuous cost. And it scales with adoption.

The more successful your AI product, the more expensive it becomes to run.

This is the hidden economics of AI that nobody talks about. The inference economy. And it's about to become the single largest driver of data center buildout, energy consumption, and infrastructure investment over the next decade.

Here's why: inference now accounts for 55-65% of all AI compute, up from 33% in 2023. By 2028, it will hit 70-80%. The inference-optimized chip market alone is projected to exceed $50 billion in 2026.

And unlike training—which you do once per model version—inference happens every single time someone uses your product.

Every ChatGPT query. Every Midjourney image. Every Claude code completion. Every AI agent action.

That's inference.

And the bill never stops coming.

Let me tell you a story that perfectly captures the inference trap.

A construction company built an AI predictive analytics tool. During development, it cost them under $200/month. Reasonable. Manageable.

Then they launched it to actual users.

First month's bill? $10,000.

They weren't hacked. They didn't misconfigure anything. Their users just… used it. And usage at scale is 40-60% more expensive than most teams estimate when moving from development to production.

Here's the math that kills startups:

The pricing that looked reasonable in testing—$3 per million input tokens, $10 per million output tokens for GPT-4 (now succeeded by GPT-5.2)—suddenly becomes catastrophic at scale.

And here's the kicker: output tokens cost 2-5x more than input tokens because generation is computationally expensive.

So that innocent-looking chatbot that helps customers with support tickets? Every response it generates is burning money. The longer and more helpful the response, the more it costs.

Monthly production costs scale from:

And this is just the API fees. If you're running your own infrastructure, you're paying for GPUs, electricity, cooling, and data center space 24/7, whether anyone is using your model or not.

Now, here's where it gets interesting.

Inference costs have dropped 1,000x in just three years.

In late 2022, running a GPT-4-class model cost approximately $20 per million tokens. In early 2026, equivalent performance costs $0.40 per million tokens—or less with providers like DeepSeek.

That's one of the fastest cost declines in computing history.

What drove this collapse?

1. Hardware efficiency gains
Each GPU generation delivers 2-3x more inference throughput per dollar. The H100 processes roughly 3x more tokens per second than the A100 at a similar price point.

2. Software optimization
Inference frameworks like vLLM, TensorRT-LLM, and SGLang improved GPU utilization from 30-40% to 70-80% through continuous batching, PagedAttention, and speculative decoding.

3. Model architecture efficiency
Mixture-of-Experts (MoE) models like Mixtral and DeepSeek V3 activate only a fraction of total parameters per token, delivering frontier-quality output at 3-5x lower compute cost.

4. Quantization and distillation
Running models at INT8 or INT4 precision reduces memory and compute requirements by 2-4x with minimal quality loss.

But here's the paradox: even with a 1,000x cost reduction, inference is still too expensive for most use cases to be profitable.

Why? Because demand is growing faster than costs are falling.

There's an economic principle called the Jevons Paradox: when you make something more efficient, people use more of it, not less.

Coal-fired steam engines got more efficient in the 1800s. Did coal consumption fall? No. It exploded—because suddenly steam power was economical for a thousand new uses.

The same thing is happening with AI inference.

At $20 per million tokens, only the highest-value enterprise applications justified LLM deployment.

At $0.40 per million tokens, every SaaS product, internal tool, and consumer app can embed AI.

The addressable market expands by orders of magnitude.

So even though per-token costs are falling, total inference spending is growing. And it's growing fast.

Total GPU-hours consumed for inference are increasing, supporting rental rates on GPU marketplaces. The inference market is projected to exceed $50 billion in 2026, growing faster than training for the first time.

This is the inference economy.

And it's going to consume more electricity, more GPUs, more data center space, and more capital than training ever did.

Let's talk about what this means for energy.

A single ChatGPT query uses roughly 10 times more electricity than a Google search according to IEA estimates.

Doesn't sound like much, right?

Now multiply that by billions of queries per day.

ChatGPT alone handles an estimated 1+ billion queries daily, leading to about 300 megawatt-hours (MWh) of electricity consumption per day and over 260,000 kilograms of CO₂ emissions per month from ChatGPT use alone.

And that's just one AI product.

Add in:

And you start to see the scale of the problem.

By 2028, over half of data center electricity will be used for AI, with inference accounting for 80-90% of that.

At that point, AI alone could consume as much electricity annually as 22% of all US households.

And here's the kicker: the energy used for inference is growing faster than renewable energy capacity.

Data centers now use electricity that is 48% more carbon-intensive than the US average, because they're increasingly powered by natural gas to meet immediate demand.

The math doesn't work.

We're building an economy on top of a technology whose energy requirements are growing exponentially, while our ability to provide clean energy for it is growing linearly.

Let me break down why inference economics are fundamentally different from training economics:

Training is CapEx. You spend a lot upfront, but it's a known, finite cost.

Inference is OpEx. It's a continuous burn that scales with success.

And here's the brutal reality: if your inference costs scale faster than your revenue, you go bankrupt.

This is why so many AI startups are struggling. They built products that users love, usage is growing, and they're… losing money on every query.

The unit economics don't work.

This brings us to the most important metric in AI today: cost-per-token.

If your inference cost per million tokens is $1.00 and you charge $2.00, your gross margin is 50%.

If inference costs drop to $0.40 per million tokens, the same pricing yields 80% margin—or you can cut prices to grow users.

This is why a price war is brewing.

DeepSeek's models cost 20-50x less than OpenAI's equivalent, with inference at roughly $0.14 per million input tokens and $0.28 per million output tokens for DeepSeek-V3.

Compare that to:

DeepSeek is 90% cheaper.

How? Through a combination of:

And here's the thing: OpenAI, Google, and Anthropic can't ignore this.

If a competitor offers 90% lower costs with comparable quality, enterprise customers will switch. Developers will switch. Consumers will switch.

So the incumbents have two choices:

Most likely, they'll do both.

This is the inference cost war. And it's just beginning.

Here's something most people don't understand: the H100 is optimized for training, not inference.

The H100 dominates AI training because training requires:

Inference has different requirements:

And for those requirements, the H100 is often overkill.

An L40S delivers comparable cost-per-token at one-quarter the price:

For pure inference workloads, the L40S is the better choice. You get similar cost-per-token at a fraction of the capital cost.

This is why the inference hardware market is fragmenting:

The H100's dominance in training doesn't guarantee dominance in inference.

The hardware wars are just beginning.

Inference doesn't just happen in centralized data centers. It happens everywhere users need it.

Tier 1: Hyperscale Cloud (AWS, Azure, GCP)

Tier 2: GPU Marketplaces (Vast.ai, RunPod, CoreWeave)

Tier 3: On-Premise / Co-located

Tier 4: Edge Inference (devices, mobile, local)

The "right" deployment depends on latency requirements, not just cost.

A medical imaging AI that needs 10ms response time cannot use a remote cloud API, regardless of price.

An autonomous vehicle cannot wait for a round-trip to a data center to decide whether to brake.

The inference supply chain is stratifying by latency tier, creating distinct GPU demand for each.

The shift from training-centric to inference-centric AI infrastructure is not just a technical trend. It's an economic and geopolitical restructuring.

Here's what I'm watching:

1. Data center design shifts
Engineers are redesigning data centers to use less copper per megawatt, with:

2. Geographic distribution
Training concentrates in locations with the most compute. Inference benefits from distribution to reduce latency. Expect more regional data centers closer to population centers, not fewer megaclusters.

3. The profitability crisis
Most AI companies are not profitable on inference. OpenAI reportedly loses money on each ChatGPT query. This is unsustainable. Either:

4. The agent explosion
AI agents use 10-100x more tokens than simple Q&A chatbots because they:

As AI shifts from chatbots to agents, inference costs will explode even as per-token prices fall.

5. Energy becomes the bottleneck
We're already seeing this play out:

By 2028, electricity may be the dominant variable cost for inference, not hardware.

Let's apply my core analytical framework here.

What's becoming abundant:

What's becoming scarce:

The collision:

We're building an abundant AI capability on top of scarce physical infrastructure.

Every efficiency gain makes AI more accessible, which increases adoption, which increases total resource consumption, which hits physical limits (energy, cooling, space, materials).

This is the inference paradox.

The better AI gets, the more we use it. The more we use it, the more it costs—in aggregate—even as per-unit costs fall.

And eventually, we hit a wall. Not a software wall. A physics wall.

You can optimize code infinitely. You can't optimize the laws of thermodynamics.

Every computation generates heat. Every watt of compute requires a watt of cooling. Every data center requires copper, concrete, steel, and rare earth elements.

The atoms matter.

And the inference economy is about to consume more atoms than any computing paradigm in history.

Let me walk through some non-obvious consequences of the shift to an inference-dominated AI economy:

1. The death of "AI for everything"
Right now, every SaaS product is adding AI features. But if inference costs don't fall fast enough, only high-value use cases will survive. We'll see a shakeout where:

2. The rise of inference-optimized models
Training will optimize for inference efficiency, not just benchmark performance. Expect:

3. The subscription model wins
Per-query pricing creates unpredictable costs for users and providers. Expect shift to:

This shifts inference cost risk from users to providers—who can then optimize infrastructure to manage it.

4. The vertical integration of AI companies
To control inference costs, AI companies will:

Inference cost control becomes a competitive moat.

5. The geopolitics of inference
Countries will compete to host inference infrastructure, not just training. Why?

Expect national policies to incentivize domestic inference capacity.

6. The energy arbitrage opportunity
Inference workloads can be geographically distributed and time-shifted. This creates opportunities for:

Energy-aware inference scheduling could reduce costs by 30-50%.

7. The privacy/cost trade-off
Cloud inference is cheap but requires sending data to third parties. Edge inference is private but expensive. This creates a market segmentation:

Privacy regulations will increase inference costs by forcing local deployment.

WINNERS:

1. Inference-optimized hardware companies

2. GPU cloud marketplaces

3. Inference optimization software

4. Energy providers with 24/7 capacity

5. Companies with pricing power

LOSERS:

1. Undifferentiated AI wrappers
If you're just calling OpenAI's API and adding a thin UI layer, you have:

Your margins will compress to zero.

2. Training-focused infrastructure

The infrastructure that powered the training boom is mismatched for the inference economy.

3. Freemium AI products with no path to profitability
Offering unlimited free AI usage was viable when inference was cheap and VCs were funding growth-at-all-costs. Now:

Expect mass shutdowns of free AI tools or aggressive feature limitations.

4. AI companies in high-electricity-cost regions
If your inference infrastructure is in California, Germany, or Japan (expensive electricity), you're at a permanent cost disadvantage vs. competitors in:

Geography is destiny in the inference economy.

5. Renewable-only data centers
Sounds counterintuitive, but here's why:

Baseload power (nuclear, geothermal, hydro) wins for inference.

Let's do some uncomfortable math.

OpenAI's reported metrics (estimated):

Revenue:

Even with aggressive assumptions, OpenAI may be losing money on inference.

And that's with the most advanced infrastructure, custom optimizations, and economies of scale that no competitor can match.

If OpenAI can't make inference profitable, who can?

Here are the only paths I see:

Path 1: Inference costs fall 10x more
Requires breakthrough innovations:

Possible, but not guaranteed.

Path 2: Willingness-to-pay increases
Users/enterprises pay more because AI becomes more valuable:

Already happening in enterprise, less clear for consumers.

Path 3: Business model shifts
Move away from per-query pricing:

Most likely path for profitability.

Path 4: Consolidation
Only 2-3 companies survive with:

Everyone else becomes a customer, not a competitor.

My bet? All four happen simultaneously.

Costs fall (but not fast enough). Prices rise (but not too much). Business models shift to subscriptions. Market consolidates to a few winners.

And even then, profit margins will be thin compared to traditional software (80%+ gross margins).

The inference economy will be a volume game, not a margin game.

If you're building an AI product:

If you're investing:

If you're in policy:

If you're a consumer:

Here's the story in one paragraph:

AI's training costs are falling. AI's inference costs are falling too—but total inference spending is exploding because usage is growing faster than efficiency. By 2028, inference will consume 80-90% of AI's energy, 70%+ of AI compute, and the majority of AI infrastructure investment. This creates a paradox: the better and cheaper AI gets, the more we use it, and the more total resources it consumes. We're building an economy on a technology whose marginal costs are near-zero but whose aggregate costs are approaching the scale of entire industries.

The inference economy is here.

And it's going to reshape:

The companies that master inference economics will dominate the AI era.

The ones that don't will burn through billions and collapse, no matter how good their models are.

Because here's the hard truth:

Training a great model is impressive. Running it profitably at scale is what actually matters.

And right now, almost nobody has figured that out.

The race is on.

Primary Sources (Accessed March 2026):

Follow @SloneSterling on X.com for daily research on AI, energy, commodities, and the collision of abundance and scarcity.

Precision in a world of noise.

Analysis by Slone Sterling