Everyone tells you that AI costs a fortune. They point to Anthropic's $100M compute budget or OpenAI's datacenter buildouts and say 'that's the price of progress.' They're wrong if you're building a product, not training a foundation model.
At IRPR, we've shipped 30+ AI-powered features over the last two years - RAG chatbots, document analyzers, automated review systems - and the cost per deployment rarely exceeds $500/month in production. That number includes inference, vector storage, embedding generation, and all the infrastructure glue.
This post walks you through exactly how we keep those numbers low. You'll learn which models to pick, when to self-host vs. use APIs, and how to design retrieval pipelines that don't burn cash on unnecessary context.
The Three Budget Levers: Model Selection, Infrastructure, and Caching
Most teams blow their AI budget on three things: over-provisioned GPU instances, unnecessary context tokens in every API call, and zero caching discipline. Fix each of these and you cut your bill by 60-80% without sacrificing quality.
We categorize every AI feature into one of three tiers: classification (fast, cheap, small models), generation (moderate cost, needs some reasoning), and agentic workflows (expensive, rare use). The best teams force 80% of their features into the first tier.
The budget breakdown we use: model inference costs should be under 40% of total spend, vector storage under 20%, and caching/infrastructure overhead under 30%. The rest goes to evaluation and monitoring.
- Model API: OpenAI gpt-4o-mini at $0.15/1M input tokens, or self-hosted Llama 3.1 8B on a single T4 GPU for $150/mo
- Vector Database: pgvector on a $20/mo PostgreSQL 16 instance from Crunchy Bridge or Supabase
- Embedding Service: text-embedding-3-small at $0.02/1M tokens, or free open-source models like BGE-small via API
- Caching Layer: Redis on a $15/mo Upstash instance; caches identical queries, saves 50-70% on inference costs
- Orchestration: LangChain or LlamaIndex on a free Vercel hobby plan; zero additional cost for low traffic
Self-Host or API: The Decision Framework
The API vs. self-host decision comes down to traffic patterns. If you're serving under 100K requests/month, APIs win every time. OpenAI gpt-4o-mini costs $15 for 100K requests (assuming 1K input + 500 output tokens each). A self-hosted Llama 3.1 8B on a T4 GPU costs $150/mo plus bandwidth - ten times more for the same volume.
Self-hosting makes sense at higher volumes. At 1M requests/month, the API cost balloons to $150, while the GPU cost stays fixed at $150. Cross that threshold and the savings compound. Use vLLM with continuous batching to maximize throughput - we've seen 50 req/s on an A10G with Llama 3.1 8B.
For fine-tuning, the math is equally brutal on small volumes. Fine-tuning gpt-4o-mini costs $25 for 100K training tokens. Running QLoRA on a RTX 4090 locally costs $0 in compute but takes 8 hours of your time. If you're doing it once, pay the API fee. If you're iterating weekly, buy the GPU.
A Real-World Budget: RAG Customer Support Bot
Here's a concrete example from a recent IRPR build. We deployed a customer support chatbot for a B2B SaaS company with 200K documents in the knowledge base. Traffic averaged 3,000 queries/day with spikes to 10K.
Monthly breakdown: OpenAI gpt-4o-mini at $140 (use prompt caching and 4K tokens/query), pgvector on a $20/mo Supabase instance, embeddings via text-embedding-3-small at $15, Redis caching at $12/mo Upstash, and observability via LangSmith at $25/mo. Total: $212/mo.
That $212 handled 90K queries with 94% answer accuracy. The client had budgeted $2,000/mo based on vendor quotes. They redirected the savings to building a feedback loop that improved retrieval accuracy by 5% month over month.
- 1Audit current AI spend and categorize by feature and model
- 2Replace gpt-4 with gpt-4o-mini or similar cheaper model for non-critical tasks
- 3Implement Redis caching with 1-hour TTL for query embeddings
- 4Set per-user rate limits (100 req/min) and per-session token caps
- 5Switch from dedicated vector DB to pgvector if index is under 10M vectors
- 6Add stale document eviction (TTL: 90 days, with 30-day query recency check)
- 7Configure prompt caching with 80% static prefix structure
- 8Reduce embedding dimensions to 512 if accuracy remains above 85%
- 9Batch embedding generation in groups of 100
- 10Monitor token usage per user and set alerts for anomalies
Smart AI Doesn't Mean Expensive AI
The narrative that AI requires venture-scale compute budgets is perpetuated by companies selling GPU instances and massive API plans. For 90% of production use cases - customer support, document QA, classification, summarization - a budget under $500/month is not just achievable, it's optimal.
The strategies here aren't theoretical. IRPR has shipped over 30 AI features using these exact patterns. Every time a new client comes in saying they need $5K/mo for AI, we show them the numbers and they leave with a $300/mo plan that works better.
If you're evaluating an AI feature for your product and the costs feel prohibitive, look closer. Chances are you can ship it for under $500/mo with the right model selection, caching, and infrastructure choices. And if you need a partner to help design that system, we're here.
The IRPR engineering team ships production software for 50+ countries. Idea → Roadmap → Product → Release. 200+ products live.
About IRPR