You've seen the headlines. OpenAI, Google, Meta — they're all racing to build bigger models, faster chips, and more capable AI. But there's a dirty secret nobody on the earnings calls wants to admit: the economics are broken.
I spent the last week talking to cloud architects, startup founders, and a couple of disillusioned engineers who've seen the spreadsheets. What I found is a house of cards propped up by venture capital and wishful thinking.
One API call costs more than a cup of coffee
Let's get specific. Running a single query on GPT-4 costs roughly $0.06 in compute. For a heavy enterprise app doing a million calls a day, that's $60,000 daily — or $1.8 million a month. For one application.
Startups building on these models are hemorrhaging cash. One founder told me his AWS bill hit $200,000 in month three. He'd raised $2 million. Do the math.
And the problem's getting worse, not better. Each new model generation demands more parameters, more context tokens, and more GPU cycles. The open-source Llama 3 70B costs roughly $0.70 per million tokens to run — that's before you factor in the infrastructure to keep it online.
"We're seeing companies raise Series A rounds just to pay their inference bills. That's not a business — it's a Ponzi scheme." — Cloud architect at a major AI startup
The inference tax is strangling innovation
The real cost isn't just training — it's inference. Every time someone talks to a chatbot, generates an image, or uses a code assistant, somebody pays for compute. And the margins are brutal.
Take customer support bots. A company replaces 10 human agents with an LLM. Each human cost $40,000/year. The bot costs $0.10 per conversation. If the bot handles 10,000 conversations a day, that's $1,000 daily — $365,000 annually. Cheaper than humans? Barely. And the bot still needs humans to fix its mistakes.
This doesn't scale. The entire AI industry is currently running on subsidized cloud credits from AWS, Azure, and Google Cloud. Those credits will dry up. And then the math gets really ugly.
Hyperscalers are the real winners
Here's what nobody says: the only companies making money in AI right now are the infrastructure providers. Nvidia's GPU sales are through the roof. AWS's AI-related revenue hit $50 billion last quarter. Meanwhile, the startups building on top are barely surviving.
It's the classic gold rush playbook. In the California gold rush, the people who got rich weren't the miners — they were the ones selling picks and shovels. Today, the picks are GPUs and the shovels are cloud credits.
Meta is spending $35 billion on AI infrastructure this year. Google's capex is $45 billion. Microsoft won't even say. These are bets that the usage will eventually pay off. But what if users don't want to pay $20/month for a chatbot?
Open-source might save us — but it's not free either
The open-source community offers a potential escape. Models like Llama 3, Mistral, and Mixtral let you run inference on your own hardware. No per-token fees. No API bills.
But here's the catch: you need the hardware. Running a decent-sized model requires at least an A100 GPU, and preferably an H100. Those cost $10,000 to $30,000 each. Then you need the electricity, cooling, and maintenance. For most startups, building their own infrastructure is even more expensive than paying for API calls.
And open-source models are getting bigger. The latest MoE models need something like 300GB of VRAM for inference. You're not running that on a laptop.
So we're stuck: pay the API tax or pay the hardware tax. Neither is cheap.
The bubble will pop — but what comes after?
I've been around tech long enough to recognize a hype cycle. Right now we're at the peak of inflated expectations. The crash will come when VCs stop funding inference costs and startups actually have to turn a profit.
When that happens, you'll see a wave of consolidation. The API prices will drop — they already have, by about 70% over the past two years. But they'll need to drop another 90% to make most applications viable at scale.
The survivors will be those who figure out how to run models efficiently. Maybe that's more quantization, better hardware, or smaller, specialized models that don't need 400 billion parameters to answer a customer question.
What you should do right now
If you're building on LLMs, stop assuming costs will magically drop. Model for the current prices and plan for them to stay flat for at least two years. Negotiate hard with your cloud provider. Optimize like your business depends on it — because it does.
And if you're an investor, for god's sake, look past the revenue numbers and ask about gross margins on inference. You might not like the answer.
The AI revolution is real. But the bill for the revolution just arrived. And it's a lot higher than anyone expected.
