AI Cost Calculators
AI GPU Inference Cost Calculator
Estimate the complete cost of serving an AI model on GPUs using model memory, measured throughput, batching, target utilization, minimum replicas, infrastructure costs, and a managed API comparison.
Enter Your GPU Inference Plan
Use measured throughput from the same model, quantization, GPU count, and serving configuration planned for production.
Monthly Inference Workload
Model Memory and Measured Throughput
Capacity and Availability
Self-Hosted Infrastructure Prices
Implementation, API Baseline, and Budget
Estimated inference deployment shape
GPUs required per replica: 2
Allocated usable GPU memory: 144 GB
Memory headroom: 24 GB
Effective compute time per request: 1.72 seconds
Busy replica-hours: 491.35
Billable replica-hours: 907.1
Billable GPU-hours: 1,814.21
Average running replicas: 1.24
GPU Inference Cost and Capacity
The estimate applies measured throughput, batching, utilization, reserve capacity, minimum replicas, and infrastructure prices.
Monthly self-hosted planning cost
Cost per request
—
Cost per 1M tokens
—
Effective utilization
54.17%
GPU compute
1,814.21 GPU-hours · 2 GPUs per replica
—
Host, CPU, and memory
907.1 billable replica-hours
—
Storage
Model files, container images, logs, and cached assets
—
Network and data transfer
Ingress, egress, load balancing, and private networking
—
Monitoring and serving platform
Metrics, tracing, orchestration, and endpoint management
—
Amortised implementation
$0.00 spread across 12 months
—
Monthly operating cost: —
Idle replica-hours: 415.76
Extra cost from minimum replica floor: —
Estimated request capacity at entered settings: 1,000,000 per month
Managed API monthly baseline: Enter both API token prices
Monthly planning comparison: Enter self-hosted and API prices
First-year comparison: Enter self-hosted and API prices
Approximate API break-even volume: Enter both API token prices
Implementation payback: Enter both API token prices
Self-hosted price inputs entered: 0 of 6
Budget status: Add a budget to compare
* Important: This calculator stores no GPU, host, managed API, storage, network, monitoring, or implementation price. Enter current rates for the exact region and deployment. Throughput must be measured on the same model, quantization, GPU count, serving engine, batch settings, and context profile. Blank optional price fields are treated as zero.
Self-hosted inference cost depends on more than the GPU hourly rate. Model memory determines the GPUs needed per replica, token throughput determines busy time, and availability requirements create idle capacity. This calculator combines those factors.
Calculating GPU and Replica Capacity
Enter model and runtime memory, usable memory per GPU, and measured prompt and generation throughput for one complete replica. The calculator estimates the minimum GPUs needed by memory and the compute time used by one request.
Monthly busy replica-hours are adjusted for retries, batching, target utilization, and capacity reserve. The result then applies the minimum replica floor to estimate billable replica-hours and GPU-hours.
Throughput must come from the same deployment shape used for the estimate. A benchmark from one GPU should not be entered as replica throughput when the production replica uses several GPUs.
Including Batching, Utilization, and Idle Capacity
NVIDIA Triton documents dynamic batching as a way to combine requests and typically increase throughput for stateless models. The actual improvement depends on request arrival patterns, sequence length, queue delay, and latency targets.
Target utilization leaves headroom for traffic variation. Capacity reserve adds another planning margin. Minimum replicas can create idle GPU-hours when the monthly workload is lower than the always-on capacity.
The calculator reports effective utilization, idle replica-hours, average running replicas, and the share of total cost caused by the minimum replica floor.
Planning Autoscaling and Endpoint Availability
Managed inference platforms can add and remove instances in response to workload. Scaling rules, cooldown time, startup delay, minimum replicas, and scale-to-zero support vary by endpoint type and provider.
Amazon SageMaker AI documents endpoint autoscaling that adjusts provisioned instances as workload changes. Vertex AI also supports inference-node autoscaling with configured minimum and maximum replicas.
Use the minimum-replica field to represent the availability floor that applies to the exact deployment option being planned.
Comparing Self-Hosting With a Managed API
Enter managed API input and output prices for a comparable model or service. The calculator compares token-based API usage with self-hosted GPU, host, storage, networking, monitoring, and amortised setup cost.
Results include monthly and first-year differences, implementation payback, and the approximate request volume where self-hosting becomes cheaper under the entered assumptions.
Practical Decisions This Tool Supports
- Estimate GPU inference cost before deployment.
- Calculate GPUs required by model memory.
- Turn measured token throughput into replica-hours.
- Measure idle cost from minimum replicas.
- Test batching and utilization assumptions.
- Calculate cost per request and per million tokens.
- Compare self-hosting with a managed API.
- Find payback and break-even request volume.
Costs and Performance Risks Outside the Estimate
The result does not automatically include engineering support, failed deployments, reserved-capacity commitments, spot interruptions, regional premiums, taxes, egress, database cost, security review, or the business cost of latency and downtime unless entered.
Throughput can change with context length, output length, quantization, tensor parallelism, batching, model architecture, serving software, and latency requirements. Production-like load testing remains essential.
Frequently Asked Questions
What numbers should I benchmark before using this calculator?
Measure prompt-processing tokens per second and generation tokens per second on the same model, quantization, GPU count, serving engine, batch settings, and context profile planned for production.
Why are prompt and generation throughput separate?
Processing input tokens and generating output tokens can have very different performance. Separating them produces a more realistic compute-seconds estimate for each request.
How does batching affect inference cost?
Batching can increase throughput by combining compatible requests, but the real gain depends on traffic, sequence lengths, latency targets, and the serving engine. Enter a measured or tested uplift rather than a guessed maximum.
Why include a minimum replica count?
Real-time endpoints may keep capacity running for availability and low latency even when traffic is low. Minimum replicas can create paid idle hours that dominate cost at smaller volumes.
How is the managed API baseline calculated?
The calculator multiplies monthly input and output tokens by the custom API rates entered. It compares that usage cost with self-hosted GPU, host, platform, and amortised setup costs.
What is the break-even request volume?
It is the approximate monthly request volume where the calculated self-hosted planning cost becomes equal to or lower than the entered managed API cost.