Skip to main content
HashnodeJay Tank's Engineering BlogJay Tank's Engineering Blog

Command Palette

Search for a command to run...

Your GPU bill doesn't work like your cloud bill - stop FinOps-ing it like one

Updated
•11 min read•View as Markdown
Your GPU bill doesn't work like your cloud bill - stop FinOps-ing it like one
J
JAY TANK
Cloud & Platform Engineer focused on building systems that scale and endure. I explore how infrastructure, automation, and engineering practices come together to support modern software teams.

A team ships a new "AI assistant" feature in March. By June, the GPU line item on the cloud bill is bigger than the entire rest of the infrastructure budget combined, nobody signed off on that, and finance is asking engineering to explain a number that engineering can't actually break down - because the bill shows an instance type and a region, not which endpoint, which customer, or which feature burned the tokens. This isn't a hypothetical. It's the modal story behind a stat that should stop you: 73% of organizations running AI workloads blew their AI budget in the past year. That's not a rounding error in forecasting. That's a sign the forecasting model itself is wrong.

Traditional cloud FinOps was built on an assumption that mostly held for a decade: cost tracks usage, usage tracks demand, and utilization is a knob you tune with reserved instances and autoscaling. AI infrastructure spend doesn't play by that rulebook. It has a bimodal cost structure (a huge sunk capital event, then an open-ended operating cost that scales with product success), it has utilization numbers that would get a traditional workload paged at 3am, and it has a chargeback problem that's structurally harder because the unit of consumption isn't a VM-hour, it's a token, a request, or a batch - and most billing systems still don't speak that language.

This is about what's actually different, backed by numbers that are current as of this year, not a general "cloud cost optimization" retread.

The scale of the shift, in one survey

The FinOps Foundation's sixth annual State of FinOps survey (1,192 respondents, representing more than $83B in managed annual cloud spend, released February 2026) is the cleanest primary source here, and the numbers move fast year over year:

  • 98% of FinOps teams now manage AI spend, up from 63% a year earlier and just 31% two years earlier. This went from a niche concern to universal in about 24 months.

  • AI now accounts for 18% of cloud budgets at AI-active enterprises, up from 4% in 2023. A 4.5x jump in a category that used to round to zero.

  • 73% of organizations report blowing their AI budget against original plans.

  • GPU/AI spend management is now the top forward-looking FinOps priority, ahead of the SaaS and licensing concerns that used to dominate the conversation.

None of that is surprising once you look at why teams can't forecast this spend the way they forecast EC2 or S3. The answer is that AI cost has a different shape entirely.

Training vs. inference: two different cost animals

This is the comparison that gets flattened in most "AI cost" conversations, and it's the one that actually matters for how you budget.

Training is capital-expenditure-shaped even when it's billed as opex:

  • Bounded in time - a training run ends.

  • Predictable-ish in advance - you can estimate GPU-hours from model size, dataset size, and epoch count before you start.

  • Concentrated - a small number of large, visible jobs, usually owned by one ML team, easy to attribute.

  • The classic "AI cost" everyone budgeted for circa 2023.

Inference is opex-shaped in the worst way - unbounded and success-correlated:

  • It never ends. It scales with usage, meaning it scales with the thing you actually want (adoption).

  • It's diffuse - thousands or millions of small requests fanned out across every feature, customer, and API caller, rather than a handful of big visible jobs.

  • Agentic and reasoning-model patterns make a single user action fan out into multiple model calls, tool calls, and long hidden reasoning chains the user never sees but the bill reflects.

  • It's the piece most orgs under-modeled, because their FinOps muscle memory was built for training-shaped, bounded spend.

And inference has now overtaken training in absolute dollars. Depending on the source, the split lands somewhere around 55-80% of AI infrastructure spend going to inference rather than training, with the FinOps Foundation's own data putting inference at the high end of that range for mature AI deployments, and industry compute-market analysis (Gartner-cited figures) showing inference crossing 55% of AI cloud spend for the first time this year. The exact percentage varies by source and by how "training" vs. "fine-tuning" vs. "inference" get bucketed, but the direction is not in dispute anywhere: the expensive part of AI is no longer the part your model card brags about. It's the part that runs every time a user hits enter.

That matters for budgeting because a training run is a project you can gate with an approval. Inference is a running cost that grows automatically with the success of the product - which means the teams best at their jobs (the ones driving adoption) are also the teams blowing the AI budget, and punishing that with a hard cap is exactly backwards.

The utilization number nobody wants to say out loud

In traditional cloud FinOps, low utilization is treated as a solvable inefficiency - right-size the instance, buy a savings plan, add autoscaling, done. GPU utilization is a different, uglier problem, for structural reasons:

  • GPUs are provisioned in large, indivisible units (a full H100 or a full node), so there's no equivalent of "downsize to a smaller instance type" without re-architecting the workload.

  • Inference traffic is bursty and unpredictable by nature - you provision for peak or you eat latency, and most teams provision for peak.

  • Reported utilization across real fleets is low: one large-scale analysis across tens of thousands of Kubernetes clusters found average GPU utilization sitting in the single digits, and independent audits of AI teams specifically have found numbers more commonly in the 15-30% range for inference serving, occasionally reaching the low 20s.

  • The gap between "idle" and "utilized" isn't a rounding error at GPU prices. An idle high-end GPU instance can burn thousands of dollars a month doing nothing, and because GPU capacity is provisioned in whole-unit chunks, idle time isn't a tuning problem you fix with a percentage adjustment - it's wasted spend at the same rate as active spend.

This is the single biggest reason a GPU bill and an EC2 bill need different FinOps playbooks. On EC2, a right-sizing exercise gets you incremental savings. On GPU inference fleets, the gap between typical utilization and what's achievable with batching, request coalescing, and model-serving optimizations (routinely cited as lifting utilization from the 20-30% range toward 60-70%+) is often the difference between the budget holding and the budget blowing by 2-3x.

Why chargeback breaks, and what to charge back on instead

Traditional cloud chargeback allocates cost by tag: this VM belongs to team X, this S3 bucket belongs to product Y. It works because the unit of cost (an instance-hour, a GB-month) maps cleanly to something a cost-center owner recognizes and can act on.

AI inference doesn't map that cleanly, for a few reasons:

  • The unit of consumption isn't the unit of billing. You're billed in GPU-hours; the business cares about cost per request, per user session, or per resolved ticket. Translating between those requires instrumenting the application layer, not just the infrastructure layer - and most orgs' cost tooling stops at the infra layer.

  • Shared model-serving infrastructure defeats simple tagging. A single deployed model instance often serves multiple product features or multiple customers concurrently. There's no clean "this GPU belongs to team X" tag when the same running model is fielding requests from five different callers.

  • Token-level variance means average cost lies. A support-bot conversation and an agentic workflow that fans out into a dozen tool calls hit the same model endpoint but cost wildly different amounts. Chargeback on "requests" without weighting for token volume and call depth systematically misattributes cost.

  • Provisioning cost and consumption cost diverge. If a team reserves capacity that then sits at 20% utilization, do you charge them for what they reserved or what they used? Charge by reservation and you incentivize hoarding; charge by consumption and nobody owns the idle-capacity cost - which somebody still has to pay.

The practical fix teams are converging on is a showback-before-chargeback sequence: instrument cost per token/request at the application layer first, get teams looking at their own numbers without punitive billing attached, and only move to hard chargeback once the attribution is trustworthy enough that a team can actually act on the number instead of disputing it.

What to actually track (the short list)

Cut through the dashboards and this is the set of numbers that matters for AI-specific FinOps, distinct from what you'd track for general cloud spend:

  • Cost per unit of value, not cost per GPU-hour. Cost per resolved support ticket, per completed agent task, per active user - whatever your product's actual unit of work is. GPU-hour cost tells you what you're paying the cloud provider; it tells you nothing about whether that spend is working.

  • Inference-to-training spend ratio, tracked over time. If this ratio is climbing and nobody planned for it, that's your early warning that the budget model built for training-era AI costs is stale.

  • GPU utilization at the fleet level, split by training vs. serving workloads. These have different acceptable baselines - training runs should be near-saturated; serving fleets realistically won't be, but you want to know if you're at 20% or 60%.

  • Idle capacity cost, as its own line item. Not "utilization percentage" buried in a dashboard - an actual dollar figure for what reserved-but-unused GPU time cost this month. Money gets attention that a percentage doesn't.

  • Cost attribution coverage - what fraction of GPU spend can actually be traced to a team, feature, or customer, versus sitting in an unattributed shared pool. Low coverage here is the leading indicator of a chargeback model that will fail when finance asks for it.

  • Budget variance against forecast, reported monthly, not quarterly. Given that inference cost is usage-correlated and usage can spike overnight (a feature goes viral, an agent workflow gets looped into a retry storm), quarterly variance review is too slow to catch it before the bill lands.

The honest limits here

This isn't a call to over-engineer cost attribution before you need it. A five-person startup running one fine-tuned model behind one API doesn't need a showback pipeline - tag the project, watch the total, move on. The complexity described above earns its keep once you have multiple teams, multiple models, and inference traffic that's genuinely shared infrastructure.

The utilization numbers cited here also vary a lot by methodology - "average GPU utilization" measured across a Kubernetes cluster fleet is not the same measurement as utilization on a single dedicated inference server, and vendors publishing these numbers have an interest in making the problem look large. Treat the specific percentages as directional (low utilization is real and structural) rather than as a number to reproduce exactly in your own environment - measure your own fleet before you act on someone else's benchmark.

And the inference-vs-training split is genuinely contested in the exact number, not just imprecisely reported - different analyses bucket fine-tuning, RAG-adjacent retrieval compute, and agentic tool-call overhead differently, so you'll see figures anywhere from 55% to 80%+ for inference's share depending on the source and the year. What's not contested is the direction and the consequence: the operating cost line is now bigger than the capital cost line, and it's the one most FinOps practices weren't built to handle.

Sources

#finops#ai#machinelearning#cloudcomputing#devops

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

J

Jay Tank's Engineering Blog

62 posts

Deep dives on running fintech & Web3 infrastructure at scale - AWS, Kubernetes, CI/CD, edge security, observability, and Bitcoin Lightning. Practical architecture breakdowns and open-source DevOps tools from a senior platform engineer.