How to Reduce AI Agent Token Costs Without Losing Quality

To reduce LLM costs for AI agents, stop looking at cost per call. Look at cost per completed task.

A cheaper call that fails more often is not a saving. It moves the cost to retries and human review.

This post covers where agent tokens go, how to measure the real cost, and four levers: caching, trimming context, routing simple steps to smaller models, and capping loops. Each lever gets a check so you know quality held.

I’m not quoting prices. They change often. Use your own invoice.

Where tokens go in an agent

A chat reply is one call. An agent is a loop. Each turn sends the whole conversation back to the model, plus tool definitions, plus every tool result so far.

That is why agents cost more than chat. Anthropic reported that agents use about 4 times more tokens than chats, and multi-agent systems about 15 times more (June 2025). In the same write-up, token usage alone explained 80% of the variance in performance on their research benchmark. So tokens are not only a cost. They are part of how the system gets its results. Cut blindly and you cut quality.

In a typical agent run, the tokens come from five places:

Source	Why it grows
System prompt and instructions	Sent on every turn
Tool definitions	Every tool’s schema is sent every turn
Conversation history	Grows with each step
Tool results	Raw API or document output pasted in
Retries and loops	Failed steps run again

The first two are repeated and stable. That makes them good cache targets. The last three grow and need trimming or caps.

Anthropic’s own guidance says to find the simplest solution possible and add agentic steps only when simpler ones fall short. They also note that agentic systems trade latency and cost for better task performance. Ask first whether the workflow needs a loop at all. A single prompt with retrieval costs a fraction of a loop.

Cost per completed task

Here is the number I’d track:

Cost per completed task = (all model spend + tool spend + human review time) / tasks completed correctly

Three details matter.

Count all runs, including failures. A run that loops ten times and gives up still cost money. It goes in the numerator.

Count only correct completions in the denominator. A task the agent marked done but a person had to redo does not count.

Include review time. If a cheaper setup sends more cases to a person, price that person’s time.

A hypothetical example. Say an invoice agent runs 1,000 tasks a month.

	Before	After a cheaper model
Model spend	$400	$150
Correct completions	900	780
Cases needing human fix	100	220
Review cost at $4 per fix	$400	$880
Total cost	$800	$1,030
Cost per correct task	$0.89	$1.32

The model bill dropped 62%. The cost per correct task rose 48%. These numbers are invented to show the math, not measured. The pattern is real, though. Splunk’s routing write-up (July 2026) makes the same point: a model can cost far less per call and still lose money if it raises failure rates or sends more work to human reviewers.

Log per run: tokens in and out, cached tokens, tool calls, loop turns, outcome, and whether a human changed it. My post on AI agent running cost per month covers the full budget, including the non-token costs.

Prompt caching

Does prompt caching help agents? Yes, more than it helps most apps, because agents resend the same prefix on every turn.

The idea is simple. The provider stores the processed form of a prompt prefix. If your next call starts with the same prefix, it reuses it at a lower price and with less delay.

Two providers announced it with different mechanics:

• Anthropic’s prompt caching was described as cutting costs by up to 90% and latency by up to 85% for long prompts. It lists agentic search and tool use among the use cases, where each step needs a new API call.
• OpenAI’s prompt caching (October 2024) applies automatically to prompts longer than 1,024 tokens. It caches the longest matching prefix and, at launch, gave a 50% discount on cached input tokens.

Those figures are from the announcements. Discounts, minimum lengths, and cache lifetimes have changed since, and they differ by provider and model. Read your provider’s current documentation before you plan around a number.

How to get the most from it:

1. Put stable content first. System prompt, tool definitions, and fixed reference material go at the top. Anything that changes per request goes at the end. A single changed word near the top breaks the match for everything after it.
2. Keep tool definitions stable. Reordering or editing tools mid-session invalidates the prefix.
3. Keep timestamps and request IDs out of the header. They look harmless and kill the cache.
4. Watch the time window. Caches expire. A workflow that calls every few seconds benefits. One that calls hourly may not.
5. Log cache hits. Providers report cached token counts in usage data. A low hit rate means your prefix is less stable than you think.

How to verify: caching should not change outputs, so run your eval set before and after. Differences point to a bug in how you changed the prompt, not to caching itself.

Context trimming

Caching makes repeated tokens cheaper. Trimming removes tokens you did not need.

Anthropic’s engineering team describes context rot (September 2025): as the number of tokens in the window grows, the model’s ability to recall information from it drops. Every extra token also draws on a finite “attention budget.” So trimming can cut cost and raise quality at once. This is the one lever where both can move the right way.

Ranked by how safe they are:

1. Clear old tool results. Once a tool has been called deep in the history, the agent rarely needs the raw output again. The same post calls tool result clearing one of the safest, lightest-touch forms of compaction.

2. Return less from tools. Don’t let a tool return a 5,000-token JSON blob when the agent needs three fields. Fix this at the tool, with filters and field selection.

3. Retrieve just in time. Instead of loading every possibly relevant document up front, keep lightweight references and fetch content when the agent needs it.

4. Summarize history (compaction). When a conversation nears the window limit, summarize it and start a fresh window with the summary. This is more aggressive. A bad summary drops a detail that matters, so test it hardest.

5. Use sub-agents for noisy work. A sub-agent reads lots of material in its own window and hands back a short summary, often 1,000 to 2,000 tokens in Anthropic’s description. Multi-agent setups use more tokens overall, so measure it.

For the full picture, see my guide to context engineering for AI agents.

How to verify: run the eval set with the trimmed context. Look especially at cases that need information from early in the run. Those are the ones trimming breaks.

Model routing for simple steps

When should you route to a smaller model? When a step is simple, well defined, and easy to check.

Agents do many kinds of work in one run. Classifying an email, extracting a date, and picking a tool are narrow jobs. Writing a nuanced reply or planning a multi-step fix is harder. A cheap model may handle the first group and not the second.

The research on this is real but mixed:

• The FrugalGPT paper (2023) reported matching the best single model’s performance with up to 98% lower cost on its benchmarks, using a cascade that tries cheaper models first.
• RouteLLM (July 2024) reported cost cuts of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K while keeping 95% of GPT-4’s quality. The authors also noted that on MMLU their routers performed near random until they added in-distribution data.

Read the second point again. A router trained on one kind of data may fail on yours. These results come from public benchmarks, not agent workloads. Treat them as proof the approach works, not as a forecast of your savings.

A practical way to start:

1. List the steps in your agent and tag each as simple or hard.
2. Run each simple step on the smaller model against your eval set, one step at a time.
3. Add a fallback. If the small model’s answer fails a check (bad format, low confidence, failed validation), escalate to the larger model. The check is the hinge. Splunk’s write-up warns that if the scorer approves weak answers or rejects good ones, quality or cost suffers.
4. Keep planning on the strong model. Splunk also notes that small models get less reliable when they must hold an ambiguous plan across a long chain of steps.

Open-weight models are another option for simple steps. See open-source vs commercial LLMs.

How to verify: compare per-step accuracy, then end-to-end task success. A step can score fine alone and still hurt the whole run.

Loop and step caps

How do you cap loops? With hard limits, set before launch.

Anthropic’s agent guidance says it is common to include stopping conditions such as a maximum number of iterations to maintain control. Without them, one stuck run can burn more tokens than a hundred normal ones.

Set these caps:

Cap	What it stops	Starting point
Max turns per task	Endless loops	Set from your longest good run, plus a margin
Max tool calls per task	Repeated calls to the same tool	Same method
Max tokens per task	Runaway context growth	A budget per task type
Repeat detector	Same call with same arguments twice	Stop or escalate on the second repeat
Wall-clock timeout	Stalled jobs	Based on the workflow SLA

Set the starting values from data. Run 100 real tasks, look at turns and tokens for the correct ones, and set the cap above the high end. Too low truncates good runs. Too high protects nothing.

When a cap fires, don’t just fail. Hand off to a person with the run log attached, or return a clear “could not complete” status. Count it as an incomplete task in your cost metric.

How to verify: track how often caps fire. A rising rate means something else changed, such as a tool failing or a prompt regression.

Prove quality held with evals

How do you know quality held? You measure it on a fixed set of cases, before and after every change.

Build the set once, then reuse it:

1. Collect 50 to 200 real cases from the workflow, including hard ones and known failures.
2. Define pass or fail for each: the right answer, the right tool call, the right escalation.
3. Record a baseline with the current setup: pass rate, tokens, cost per completed task.
4. Change one thing at a time. Cache, trim, route, or cap. Not all four at once.
5. Rerun and compare. Accept a change only if pass rate holds within your tolerance and cost per completed task falls.

Use a judge model for fuzzy outputs, with spot checks by a person. My guides to the AI agent evaluation approach and LLM-as-a-judge cover the setup.

Order of attack, lowest risk first: fix tool outputs and prompt order, turn on caching, add loop caps, clear old tool results, route simple steps, then compaction and sub-agents.

Bottom Line

Track cost per completed task, not cost per call.

Cache what repeats, trim what you don’t need, route simple steps down, and cap every loop. Change one thing at a time and rerun your eval set after each.

My recommendation: start with caching and tool output cleanup. They carry the least quality risk and often give the quickest drop.

FAQ

Why do agents burn so many tokens? An agent runs in a loop, and each turn resends the system prompt, tool definitions, history, and tool results. Anthropic reported that agents use about 4 times the tokens of chat, and multi-agent systems about 15 times. Retries and long tool outputs add more.

Does prompt caching help agents? Yes. Agents resend the same prefix on every turn, which is what caching rewards. Put stable content first, keep tool definitions fixed, and avoid timestamps in the header. Check your provider’s current docs for discounts and cache lifetimes, because both have changed over time.

When should I route to a smaller model? Route steps that are simple, narrow, and easy to validate, such as classification or field extraction. Keep planning and ambiguous reasoning on the stronger model. Add a fallback that escalates when a check fails, and test each step against your eval set.

How do I cap loops? Set limits on turns, tool calls, tokens, and wall-clock time, using the distribution from 100 real runs. Add a repeat detector for identical calls. When a cap fires, escalate to a person with the log instead of failing silently.

How do I know quality held? Run the same eval set before and after each change and compare pass rate and cost per completed task. Change one lever at a time. Accept the change only if the pass rate holds within your tolerance.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect