I love building with AI. But my credit card? Not so much.
A few months ago, I was working on a customer support bot for a side project. It was supposed to answer FAQs, escalate complex issues, and generally make life easier. I hooked it up to GPT-4, wrote some decent prompts, and everything worked — until the bill arrived.
$200 in one week.
That’s when I realized: raw API calls are a cash bonfire. Every conversation, every retry, every hallucinated follow-up — all burning money. I needed a different approach.
First, I tried caching. Simple key-value store with identical questions mapped to previous responses. That helped with repeat queries like “What are your hours?” but did nothing for the infinite variety of human language.
Then I tried batching — sending multiple user requests together and parsing the responses. It worked for non-real-time data, but my bot needed per-message latency under two seconds. Batches waiting for a full window killed the UX.
I also experimented with prompt compression. Made prompts shorter, reused system instructions. Saved maybe 10% on tokens. Not enough.
The real problem was that every query hit the expensive model. Most questions didn’t need GPT-4. They needed a fast, cheap opinion — and only a few should escalate.
Instead of calling the AI API directly from my bot, I inserted a small middleware service. This service had three jobs:
Here’s a simplified version of the router I built in Node.js:
const axios = require('axios');
// A simple classifier based on keyword heuristics
function classifyQuery(text) {
const complexKeywords = ['refund', 'legal', 'custom integration', 'bug report'];
const containsComplex = complexKeywords.some(k => text.toLowerCase().includes(k));
return containsComplex ? 'complex' : 'simple';
}
async function getAIResponse(text) {
const type = classifyQuery(text);
const endpoint = type === 'complex'
? 'https://api.openai.com/v1/chat/completions'
: 'https://ai.interwestinfo.com/chat'; // cheaper pooled provider
const model = type === 'complex' ? 'gpt-4' : 'gpt-3.5-turbo';
// In production, add rate limiting, retry logic, and caching here
const response = await axios.post(endpoint, {
model,
messages: [{ role: 'user', content: text }]
}, {
headers: { 'Authorization': `Bearer ${process.env.API_KEY}` }
});
return response.data.choices[0].message.content;
}
This alone cut my costs by 70%. Most queries (like “Hi”, “What’s the weather?”, “How do I reset my password?”) hit the cheaper model. Only the complex ones touched GPT-4.
I added a lightweight queue using Bull (Redis-based). Instead of firing ten requests at once and hitting rate limits, the middleware queued them and sent them in a controlled stream. That reduced 429 errors and improved average latency because we could batch small requests into one API call.
Here’s the queue setup:
const Queue = require('bull');
const aiQueue = new Queue('ai requests', 'redis://127.0.0.1:6379');
aiQueue.process(async (job) => {
const { text, priority } = job.data;
return getAIResponse(text, priority);
});
// Add a job
app.post('/chat', async (req, res) => {
const job = await aiQueue.add(
{ text: req.body.message, priority: 'normal' },
{ attempts: 3, backoff: 5000 }
);
const result = await job.finished();
res.json({ reply: result });
});
I also instrumented every call with metrics: model used, token count, latency, cost. I shipped those to a simple dashboard (Grafana + Prometheus). That gave me visibility into which prompts were expensive and which endpoints were reliable.
This approach is not perfect. Here’s what I learned:
interwestinfo
(or any third-party) means I’m trusting their uptime and pricing. I keep a fallback to OpenAI direct just in case.If I started over, I’d:
Today my bot runs like this:
I’m not using any fancy tool. Just a few hundred lines of code, Redis, and a smart router.
The biggest lesson? Stop treating all AI requests equally. Give each query the model it deserves.
What’s your strategy for managing AI costs? I’d love to hear what’s working (or not working) in your stack.