Why AI Agents Are Slow and How to Fix It

AI agent latency is the sum of every step the agent takes before the user sees a useful answer. Model calls, tool calls, retries, and waiting between them. They stack.

The fix is not one trick. It is a budget. Decide how long the whole task may take, split that time across steps, and measure each step against its share.

Below: where the time goes, how to set the budget, and the changes that cut the most delay.

Where the time goes

A simple chatbot makes one model call. An agent makes many. Each loop is: send context to the model, wait for it to choose an action, run a tool, send the result back. Five loops means five model calls plus five tool calls, one after another.

Time hides in six places:

Input processing. The model reads your prompt before it writes anything. Long prompts and large tool results make this slower.
Output generation. Models write token by token. More tokens, more time. OpenAI’s latency guide says cutting 50% of your output tokens may cut about 50% of your latency, while cutting 50% of your prompt may only improve latency by 1 to 5%.
Tool calls. A database query, a CRM lookup, or a search can take longer than the model call itself.
Sequential loops. The agent waits for tool A before it asks for tool B, even when B does not need A.
Retries and errors. A failed call plus a retry doubles that step.
Overhead. Network hops, auth, queues, and cold starts.

Anthropic’s guide to building agents is honest about the tradeoff: agentic systems “often trade latency and cost for better task performance.” So some delay is the price of capability. The job is to spend it where it pays.

LangChain’s latency primer from March 2025 lists the same levers: find where the latency comes from, change the UX, make fewer model calls, speed up the calls you keep, and run calls in parallel. I follow that order. Measure first.

Measure before you tune

You cannot cut what you cannot see. Log a timestamp at the start and end of every model call and tool call, with a shared run ID. Then look at a few slow runs and ask: which step ate the time? Is it one slow tool, or many small calls? Our observability tools guide covers the tracing setup.

Look at the 90th percentile, not the average.

Latency budget per step

A latency budget works like a money budget. You set a total, divide it across steps, and flag anything that goes over.

Step 1: Set the total from the user. Start with how long the person will wait (next sections). Say the answer is 8 seconds.

Step 2: List the steps. Look at your typical run. For example, a support agent might: classify the request, look up the customer, search the knowledge base, draft a reply, check the reply.

Step 3: Assign a share to each. Here is a hypothetical budget for that 8-second target. The numbers are my starting points, not benchmarks:

Step	Budget	Notes
Classify request	0.7 s	Small, fast model
Look up customer	0.5 s	Cached or indexed query
Search knowledge base	1.0 s	Run in parallel with lookup
Draft reply	4.0 s	Largest model call, streamed
Check reply	1.0 s	Rules first, model only if needed
Overhead	0.8 s	Network, auth
Total	8.0 s	Lookup and search overlap, so real wall time is lower

Step 4: Enforce it. Add a timeout to each step. When a step exceeds its budget, log it, and decide in advance what happens: retry once, use a fallback, or tell the user.

Step 5: Review weekly. Find the steps that blow their budget most. Fix those first.

Budgets expose bad designs early. If your draft step alone needs 12 seconds, tuning elsewhere will not hit 8. Our agent architecture guide covers how to pick structures that match the time you have.

Parallel tool calls

This is often the biggest win and the easiest to find.

If two tool calls do not depend on each other, run them at the same time. Wall time becomes the slowest call, not the sum. OpenAI’s guide says the same: if steps are not strictly sequential, split them into parallel calls.

Research supports it. The LLMCompiler paper built a system that plans function calls and runs independent ones in parallel. Against the ReAct pattern it reported up to 3.7x faster latency and up to 6.7x lower cost on its benchmarks. Your gains will vary. But the direction is clear.

Use this check for every pair of tool calls:

Question	If yes
Does call B need the output of call A?	Keep sequential
Do both only need the original request?	Run in parallel
Does either call write data?	Be careful, order may matter
Could one result make the other call unnecessary?	Consider running sequentially to save cost

Three practical tips:

Tell the model it may call tools together. Many model APIs support returning several tool calls in one turn. Check the docs for your provider and test it, since models differ on how readily they do this.
Run known fan-out in code. If you always fetch customer and orders, do it before the model sees the request. That also removes a model decision.
Cap concurrency. Parallel calls can hit rate limits on your tools.

Anthropic’s guide calls this pattern “sectioning”: break a task into independent subtasks and run them in parallel. It also describes “voting,” running the same task several times, but that spends more to gain accuracy, not speed.

Caching and smaller models

Cache what repeats

Most agent prompts begin with the same thing: instructions, tool definitions, policies. Prompt caching lets the provider reuse its work on that shared start. Anthropic’s prompt caching docs describe caching prompt prefixes to cut cost and latency, with a short default lifetime that refreshes on use.

A January 2026 study of caching for long-running agent tasks tested several providers. It found API costs fell 41% to 80% and time to first token improved 13% to 31%. It also found a trap: caching everything can backfire. Dynamic tool results trigger cache writes that never get reused, and in some setups that raised latency. The lesson is to keep the stable parts first in the prompt, put changing content last, and do not cache tool results that will not repeat. Short prompts below the provider’s minimum size also will not cache.

Other caches worth adding:

Tool result caching. If the same lookup repeats within a session, store it.
Answer caching. For common questions with stable answers, skip the model.
Precomputed data. Compute slow summaries ahead of time, not at request time.

Use smaller models for simple steps

Not every step needs your strongest model. Classification, routing, extraction, and format checks often run fine on a smaller, faster one. OpenAI’s guide notes that model size is the main driver of inference speed, and that smaller models can sometimes match larger ones when used well.

My approach: use the big model for the step that needs judgment, and a small one for everything around it. Then test. Run the same set of cases through both and compare accuracy before you switch. A faster wrong answer is not a win. If you do not have an eval set yet, build one first.

Cut tokens

Trim prompts. Remove old instructions nobody needs.
Return only the fields the agent needs from tools. A tool that returns a 5,000-word record slows every later step. Our tool design guide covers shaping tool output.
Ask for concise output where it helps. Output length drives latency most.
Don’t default to a model. OpenAI lists this as a principle: hard-code, precompute, or use the UI when you can.

Streaming progress

Streaming does not make the agent faster. It makes the wait feel shorter. That counts.

OpenAI’s guide says streaming cuts the waiting time to a second or less, because the user sees words appear while the rest is generated. Use it for the final answer.

For long agent runs, stream the progress too:

“Looking up your account…”
“Found 3 matching documents. Reading…”
“Drafting your reply…”

The research on waiting is old but still holds. Nielsen’s response time limits note that for delays over 10 seconds users should get feedback on when the work will finish, and a clear way to stop it. NN/g’s progress indicator article cites a University of Nebraska-Lincoln study where people shown a continuous progress bar were willing to wait about three times longer than those with no indicator.

Be honest in the progress text. Don’t fake percentages. Show real steps. And let users cancel.

For work that takes minutes, drop the spinner. Run in the background and notify the user when it is done.

User tolerance by use case

How long will people wait? It depends on what they expect. NN/g’s classic limits: about 0.1 seconds feels instant, 1 second keeps a user’s flow of thought, and 10 seconds is about the limit of keeping their attention.

Agents rarely hit 1 second. So the question is what people accept for each job. Here are my starting targets. Treat them as hypotheses to test with your own users, not as data:

Use case	Target for first useful output	Total time they will accept	Notes
Live chat with customers	Under 2 s	10 to 15 s	Stream the first words fast
Internal copilot, one question	2 to 3 s	10 to 20 s	Show tool steps
Voice assistant	Under 1 s	A few seconds	Hardest case; avoid long loops
Document review or research	5 s to start	Minutes	Show progress, allow background
Overnight batch work	Not applicable	Hours	Optimize cost, not speed

Ask the real users. A claims adjuster who gets a full file review in 40 seconds may be delighted if it used to take an hour. A customer on live chat may leave after 15. The same agent can be fast or slow depending on the job it replaces.

Measuring end to end

Measure what the user feels, not just what the model does.

Track these:

Time to first visible progress. What the user waits before anything happens.
Time to complete, at P50 and P90.
Per-step time, with the budget next to it.
Steps per run. Watch for runs that wander.
Timeout and retry rate. Retries hide inside the average.
Cache hit rate. It shows whether your caching plan works.

Set an alert when P90 total time crosses your target for a day. Then pull the slowest five traces and find the step that blew its budget.

Watch for the quality side too. Every speed change can hurt accuracy. After you add parallel calls, cheaper models, or tighter prompts, rerun your evals. Our testing strategies guide covers how to keep that safety net in place.

Bottom Line

Agents are slow because steps stack. Set a latency budget from what users will accept, measure each step against it, and fix the worst offender first.

The biggest levers, in my order: run independent tool calls in parallel, trim tokens, cache stable prompt prefixes, use smaller models for simple steps, and stream progress so the wait feels honest.

Next step: pull your five slowest recent runs and write down the time each step took. That list is your plan.

FAQ

Why are AI agents slow? They chain steps. Each step is a model call, a tool call, or both, and they usually run one after another. Long prompts, large tool outputs, and retries add more. The model also writes output token by token, so long answers take longer.

How do I reduce latency? Measure each step first. Then run independent tool calls in parallel, trim prompts and tool outputs, cache stable prompt prefixes, use a smaller model for simple steps, and stream results. Retest accuracy after every change.

Does streaming help? It helps perceived speed, not total time. Users see output sooner, which makes the wait easier. For long runs, stream real progress steps, not just the final text.

Which steps can run in parallel? Any tool calls that only need the original request and do not depend on each other. Lookups of different data sources are common examples. Keep calls sequential when one needs the other’s output, or when order affects writes.

What delay will users accept? It depends on the job. Research from Nielsen Norman Group says about 1 second keeps a user’s flow of thought and 10 seconds is the limit of keeping attention. Past that, show progress. For deep research or document review, people accept minutes if you show what is happening.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect

Why AI Agents Are Slow and How to Cut Latency