We had a client spending $12,400 per month on OpenAI API calls. Their app handled 2.3 million requests a month, mostly summarizing customer support tickets and generating product descriptions. The problem was they treated every prompt like a one-off conversation.
After analyzing their call patterns, we found that 68% of their input tokens were identical across requests. Cache hits were near zero. They were paying for the same work over and over. Six weeks of refactoring later, their monthly bill dropped to $3,400 without any noticeable drop in response quality.
This is not an outlier. In our experience, most teams are overpaying by 3-10x on AI inference, regardless of whether they use OpenAI, Anthropic, or self-hosted models. The fixes are not exotic — they just require a shift from 'fire and forget' to 'think before you call'.
- Input token waste: 60-75% of input tokens are repeated across requests in most SaaS apps. Prompt caching can eliminate this entirely.
- Model overkill: 40-50% of queries can be handled by smaller, cheaper models (gpt-4o-mini vs gpt-4o) without quality loss.
- Idle GPU time: Self-hosted models run at 15-30% utilization on average. Batch processing can push this to 80%+.
- Latency vs cost tradeoff: Increasing batch wait time from 100ms to 1 second can reduce per-token cost by 60% on GPU clusters.
Where Most Teams Leak Money (And How to Plug It)
The biggest leak is repeated context injection. Many apps send the entire conversation history or document context with every request, even when the model only needs the last turn. We saw one system sending a 12,000-token system prompt on every call where only the user's question changed.
Fix: Static system prompts should be cached at the API level. OpenAI and Anthropic both support prompt caching on their latest models. You just need to structure your requests so the first N tokens are exactly identical. We prefix every call with a deterministic header that includes the model version, temperature, and system prompt hash.
The second leak is over-provisioning GPUs for self-hosted models. Teams buy A100s or H100s because they might need the capacity for peak load, then let them sit idle 70% of the time. Solution: use a spot instance pool with automatic scaling and preemption handling. We run Mistral 7B on 4x L4 GPUs from GCP Spot, costs dropping from $4.20/hr to $1.10/hr.
The third leak is treating all requests equally. A simple 'yes/no' classification should not pay the same per-token cost as a complex summarization. Build a cost-conscious routing layer that assigns each request to the cheapest model that can handle it. We use a decision tree with 5 features: token count, expected complexity (from a small BERT classifier), latency budget, required accuracy, and user tier.
- 1Audit your current AI spend for the last 30 days. Break down by model, endpoint, and user.
- 2Enable prompt caching on OpenAI or Anthropic. Measure hit rate after 48 hours.
- 3Deploy a model router that classifies requests by complexity and routes to the cheapest model.
- 4Set max_tokens to the 95th percentile of actual output length for each endpoint.
- 5Switch to spot instances for self-hosted inference. Add preemption handling with checkpointing.
- 6Implement request batching with a 200ms window. Measure throughput improvement.
- 7Review your system prompt. Eliminate anything that does not change between calls.
- 8Add cost alerts. Set up a webhook that notifies your team when daily spend exceeds 2x the baseline.
Real Numbers From Production Systems
We track these metrics across all the AI systems we operate. After the first month of optimization, the typical profile looks like this: pre-optimization spend of $8,200/month on 3.1M requests. Post-optimization spend of $2,450/month on 3.4M requests (traffic grew). Cache hit rate: 72%. Model router accuracy: 97.4%. Average latency increase from routing: 38ms. User satisfaction: unchanged.
The key insight is that cost reduction and performance are not in tension. Caching reduces latency by 40-60% because the model skips the computation for cached tokens. Batch processing increases throughput per GPU by 3-5x. Model routing improves response time for simple queries because smaller models generate tokens faster.
One client in fintech was skeptical that cheaper models could handle their compliance reviews. We ran a blind A/B test: 500 reviews handled by gpt-4o, 500 by gpt-4o-mini with a structured prompt. Accuracy difference: 0.3% (not statistically significant). Cost difference: 87%. They switched that night.
Stop Overpaying for AI Inference
The AI gold rush has produced a lot of brilliant products and a lot of wasteful spending. For most teams, the path to 60-70% cost reduction is clear: cache aggressively, route intelligently, and batch relentlessly. These are not research projects. They are engineering decisions that can be implemented in days, not months.
If you are running an AI-powered product and your monthly inference bill makes you wince, you are probably overpaying. IRPR has optimized dozens of AI pipelines, from early-stage startups spending $2,000/month to enterprise systems burning $150,000/month on GPU clusters. We know exactly where the waste hides and how to eliminate it.
Book a discovery call. We will audit your last 30 days of AI spend and deliver a specific plan to reduce costs by at least 40% — or we will tell you honestly if you are already optimized. No fluff, no marketing. Just engineering data.
The IRPR engineering team ships production software for 50+ countries. Idea → Roadmap → Product → Release. 200+ products live.
About IRPR