How I Cut My LLM API Bill by 60% Without Touching My Agents

My LLM bill in January 2026 was $340. In April it was $136. Same agents, same workload, roughly the same number of tasks processed per day. I did not swap models for cheaper ones across the board, and I did not cripple any features to get there.

What I did was build a proper cost audit loop — the same kind of loop I now package into every production agent system I ship for clients.

Here's what actually moved the numbers.

The Moment I Realized I Had a Waste Problem

I built LLM Cost Lens as a side tool for myself. Not as a client product initially — just as a dashboard so I could actually see where tokens were going. I had four agents running in parallel at the time: the AI Mafia multi-agent system, a WhatsApp automation bot running on Cloud Run, a context-window auditor I call ctxauditor, and an early version of a document-intelligence agent.

What the dashboard showed me was embarrassing. About 38% of my total token spend was going to system prompts being re-sent on every single call. Not cached. Not chunked. Just blasted wholesale at the model every request. Another 14% was coming from tool-call responses that included full JSON blobs when only 3 fields were ever read downstream.

That is more than half my spend sitting in fixable inefficiency. Nothing exotic. Just inattention.

Token Caching: The First Fix That Actually Sticks

The single highest-ROI change was prefix caching. Anthropic's prompt caching on Claude cuts the cost of re-used prompt prefixes by 90%. For my WhatsApp bot — which shares a long system prompt across every user session — that alone saved $47 in a single month.

The implementation is not complex. You mark your static system prompt sections with cache_control set to ephemeral. The model caches that prefix for up to 5 minutes. If the next request hits within that window, you pay cache-read rates (0.3 cents per million tokens on Sonnet) instead of full input rates (3 cents per million on Sonnet). A 10x difference.

The catch — and I got burned by this for about a week — is that caching only kicks in if the cached content is at the front of the prompt. If your system prompt structure changes per request (say, you are dynamically injecting user data into the system prompt itself rather than the user turn), your cache hits drop to zero. I had to restructure three agents before caching started working reliably.

Scratch that earlier assumption I made that caching would just work once I turned it on. It needs architecture discipline, not just an API flag.

Model Routing: The Part Most Builders Skip

Not every call needs GPT-4o or Claude Sonnet. I know that sounds obvious. But when you are building fast, you default to the best model because you trust it, and routing feels like premature optimization.

Here is the framing that finally got me to actually build it: classify your calls by decision stakes, not by default habit.

I split agent calls into three tiers:

Tier 1 — Extraction, formatting, simple classification. Low stakes. A smaller model handles this fine.

Tier 2 — Reasoning over ambiguous inputs, multi-step planning, tool selection. Medium stakes. Sonnet or equivalent.

Tier 3 — Creative synthesis, novel problem-solving, nuanced judgment calls. High stakes. Only here do I send to the flagship model.

In my ctxauditor agent, about 70% of calls were Tier 1 work — literally just pulling structured fields out of documents. I moved those to a smaller, faster model. Cost per call dropped by 85% for that tier. Total agent cost dropped 41%.

The routing logic itself is simple: a lightweight classifier agent (Tier 1 cost, obviously) decides which model tier handles each incoming task. The overhead of the classifier is about 200 input tokens per call. At scale, that pays for itself within 10 calls.

The Tool-Response Problem Nobody Talks About

This one surprised me. When an agent calls a tool and gets a response back, that response goes into context. If you are not careful, it stays there for the entire conversation, growing the context window with every turn.

In my AI Mafia system — which has multiple sub-agents communicating back and forth — I was seeing context windows balloon to 180,000 tokens mid-conversation. Almost all of it was stale tool responses from earlier in the workflow. Results that had already been used, processed, and were never going to be referenced again. Still paying to re-send them with every new turn.

The fix was context pruning: after a tool result gets consumed by the agent, flag it. On the next turn, replace it with a 1-2 sentence summary rather than the full JSON. For long tool responses (API calls that return 50-field objects), I now write explicit compressor prompts that extract only the fields the downstream logic will use before the result ever enters context.

This brought my AI Mafia context overhead down from roughly 180K tokens per workflow to roughly 42K tokens. Per day, at the volume this system runs, that is meaningful.

The Budget Loop Pattern

One pattern I have standardized across every new agent I build: the token budget loop.

The idea is simple. Before each LLM call, check the running token cost for this session against a soft cap. If you are within 20% of the cap, switch to a frugal mode — shorter system prompt, smaller model tier, less verbose tool calls. If you hit the cap, the agent surfaces a summary to the user and pauses rather than running indefinitely.

I implemented this first in the Cloud Run WhatsApp bot. The agent now tracks cumulative token spend per user per day using a lightweight Redis counter. When a user's session approaches the budget threshold, the agent naturally compresses its responses. Heavy users never notice (the quality is still there), but they stop triggering $2-per-session outlier costs.

Total spend on that bot: down 28% month-on-month after the budget loop shipped.

What LLM Cost Lens Actually Showed Me

The dashboard I built — LLM Cost Lens — surfaces four numbers I now consider essential for any production agent:

1. Token distribution by prompt section — what percentage of your spend is system vs. context vs. tool responses vs. assistant output

2. Cache hit rate — are you actually benefiting from prefix caching or just thinking you are?

3. Model tier allocation — is your routing logic working or are Tier 1 tasks still going to your flagship model?

4. Per-user and per-workflow cost variance — which specific users or workflow paths are outliers?

That fourth number is underrated. In every system I have audited, there are 5-10% of users or workflow types that generate 40-60% of total cost. Identifying them lets you build targeted compression logic rather than blunt cuts that hurt everyone.

The Harder Truth About Cost Control

Here is what I tell clients when they come to me wanting to cut their LLM bills: the cheapest call is the one you architect correctly the first time.

Every cost optimization I have described is a retrofit. I was fixing architectures I had built fast, under pressure, without cost modeling. The real leverage is building a cost-aware system from day one — which means doing a token budget analysis during design, not after your first invoice shock.

For new projects, I now run a two-hour cost modeling session before writing a line of agent code. We estimate call volume, average context size, model tier split, and tool response overhead. It is not perfect — real usage always differs from estimates — but it gets clients to within 30% of actual costs, which means no surprises and no emergency optimization sprints.

The $204 I saved between January and April was not a clever hack. It was the result of having data (LLM Cost Lens), having a routing framework (tier classification), and having the discipline to retrofit my systems properly rather than just adding more compute.

For any AI builder doing serious production work: instrument your spend before you instrument anything else.

---

FAQ

What is the biggest driver of high LLM API costs in production agents?

System prompts re-sent in full on every call without caching are typically the largest single driver. In most agent architectures I have audited, uncached static prompts account for 30-40% of total token spend. Enabling prefix caching on your static prompt sections and restructuring prompts to place static content first can cut this component by up to 90%.

How does prompt caching work on Anthropic's Claude API?

You mark static sections of your prompt with cache_control set to ephemeral. If the same prefix is re-used within a 5-minute window, subsequent calls pay cache-read rates — roughly 10x cheaper than full input rates. The cached content must appear at the start of the prompt; dynamic content injected into the static section breaks cache hits entirely.

What is model routing in AI agent development?

Model routing means directing each LLM call to the cheapest model capable of handling that specific task. You classify calls by decision complexity — simple extraction goes to a lightweight model, nuanced reasoning goes to a flagship model. A classifier agent (itself running on the cheapest tier) makes the routing decision. The overhead is around 200 input tokens per call, which pays back within a few routed calls.

How do you prevent context windows from growing too large in multi-agent systems?

After a tool response is consumed by the agent, replace the full response with a compressed 1-2 sentence summary in subsequent turns. For API responses returning large objects, write a compressor prompt that extracts only the fields the downstream logic will reference. This can reduce per-workflow context from 150K or more tokens to under 50K tokens.

What is a token budget loop and why does it matter?

A token budget loop checks cumulative session spend before each LLM call and switches the agent to a frugal mode — smaller model, shorter prompts, less verbose outputs — when spend approaches a soft cap. It prevents runaway costs from long sessions or heavy users without degrading quality for normal usage patterns.

How much can model routing realistically save?

In my ctxauditor agent, routing 70% of calls to a smaller model reduced per-call cost by 85% for that tier and cut overall agent cost by 41%. Results vary by workload, but any agent with a significant portion of structured extraction or simple classification calls will see large savings from routing those calls away from flagship models.

Is it worth building a custom cost dashboard like LLM Cost Lens?

For any agent processing more than a few hundred calls per day, yes. Without granular data on token distribution by prompt section, cache hit rate, and per-workflow cost variance, you are optimizing blind. Even a basic logging layer that captures prompt tokens, completion tokens, model used, and session ID per call gives you the data you need to find the 5-10% of workflows generating 40-60% of cost.

Should I optimize costs before launching an AI agent or after?

Before — even a two-hour cost modeling session during design is worth it. Estimate call volume, average context size, and model tier split. Real usage will differ, but you will land within 30% of actual costs, avoiding invoice shock and emergency retrofits. Retrofitting cost controls into a live production system is 3-4x more work than building them in from the start.