Context Engineering for Enterprise AI Agents

Context engineering for AI agents means deciding what the model sees at each step, and nothing more. The prompt is one piece. Tool results, retrieved documents, message history, and notes all count.

Most agent failures I’d call “the model got dumber” are really “the context got messy.” Fix the context before you change the model.

This post is the narrow, practical version for enterprise ops agents: what each step needs, what to leave out, and how to budget tokens.

Context engineering for AI agents: from prompts to context

Prompt engineering is writing and organizing instructions. Anthropic defines context engineering as the strategies for curating and maintaining the optimal set of tokens during inference, including everything that lands there beyond the prompt.

That difference matters for agents. A chatbot gets one prompt and answers. An agent runs a loop. It calls a tool, reads the result, calls another, and keeps going. Every pass adds tokens. By step eight, your carefully written prompt is a small part of what the model reads.

How is it different from prompt engineering?

Prompt engineering asks: what should the instructions say?
Context engineering asks: what should be in the window right now, from which source, and in what form?

You still need good instructions. But for an agent, the bigger lever is what gets loaded around them. If you want the plain definition of the window itself, see the LLM context window glossary entry.

Why more context hurts

It feels safe to load everything. It isn’t. Three findings explain why:

Anthropic says context is a finite resource with diminishing returns. Models have an “attention budget,” and every added token uses some of it. The guiding principle is to find the smallest possible set of high-signal tokens.
Chroma’s Context Rot study tested 18 models and found performance grows less reliable as input length grows, even on simple tasks. Distractors made it worse, and more so as inputs got longer.
The Lost in the Middle paper found models do best when relevant facts sit at the start or end of the input, and worse when they sit in the middle.

Treat these as direction, not exact numbers for your model. Test your own. The rule holds: extra context is not free, and it can lower accuracy.

Context sources for an ops agent

Say you run an accounts payable exception agent. It reads an invoice that failed matching, finds out why, and proposes a fix. This example is hypothetical, but the shape fits most ops agents.

Its context can come from six places:

Source	Example	Changes how often
System instructions	Role, rules, output format	Rarely
Tool definitions	What the agent can call	Rarely
Task input	The failed invoice	Every run
Retrieved records	PO, vendor master, past exceptions	Every step
Policy documents	Approval limits, tolerance rules	Monthly
Working state	Notes, plan, results so far	Every step

Two of these are stable and four change. That split drives most of your design choices. Put the stable parts first and keep them identical between runs. Manus, which builds a general agent, calls the cache hit rate the single most important metric for a production agent. They report an input-to-output token ratio of about 100 to 1, and a 10x price gap between cached and uncached input on the model they used then. Check your own provider’s current pricing, but the lesson holds: a stable prefix saves money and time.

LangChain groups the work into four moves: write, select, compress, and isolate context. I’ll use those four words below.

Per-step context needs

The most useful exercise is a table with one row per step. For each step, list what the model needs, and what it does not.

Here it is for the AP exception agent:

Step	Needs	Leave out
1. Classify the exception	Invoice header and totals, short list of exception types	Line items, full policy text, vendor history
2. Gather facts	Invoice lines, matching PO, receipt data	Unrelated POs, old exceptions from other vendors
3. Decide the cause	Facts from step 2, tolerance rules that apply	Rules for other exception types
4. Propose the action	Cause, approval limits, allowed actions	Raw documents from step 2
5. Write the record	Final decision, short reason, IDs	Reasoning trail beyond a few lines

Look at the “leave out” column. Each step sees a different slice. That is the whole technique.

What context does each step need?

Use three questions for every item you plan to load:

Will the model use this to choose its next action? If no, cut it.
Can the agent fetch it later if it needs it? If yes, load a pointer now, not the content.
Is it the freshest version? Stale copies cause conflicts.

The second question is the big one. Anthropic describes keeping lightweight identifiers such as file paths, stored queries, or IDs, and loading the data into context at runtime through tools. A PO number costs a few tokens. The PO document costs hundreds. Load the document at the step that needs it.

Trimming and summarizing

You will need to shrink context. The fear is losing a fact the agent needs later. These rules help.

Cut tool output at the source. Many tools return far more than the agent needs. Make the tool return the fields the next step uses, not the whole record. My post on AI agent tool design covers how to shape responses.

Keep identifiers when you drop content. Manus describes compression that is restorable: drop a web page’s content but keep its URL. In ops, drop the invoice body but keep the invoice ID and the extracted fields. The agent can re-fetch if needed.

Move facts out of the chat and into notes. Anthropic calls this structured note-taking: the agent writes notes to storage outside the window and pulls them back later. For ops, this can be a small state record with fields like cause, evidence_ids, open_questions.

Summarize in phases, not by default. Anthropic’s multi-agent research team had agents summarize completed work phases and store essential information in external memory before moving on. Their lead agent also saved its plan to memory, since context past 200,000 tokens gets truncated. Summaries are lossy. Do them at clean boundaries, not mid-step.

Use a fresh context for heavy lookups. Sub-agents can work on a focused task in a clean window and return a short answer. Anthropic notes this in its context engineering post. Be careful: the same team found multi-agent systems use about 15 times more tokens than chat, against about 4 times for a single agent. Isolation moves tokens around.

Keep failures visible. Manus leaves failed actions and their errors in context, so the model learns not to repeat them. Trim the noise around an error. Keep the error.

A trim-without-losing-facts checklist

Each fact the final decision depends on has an ID or a source pointer.
Anything dropped can be re-fetched by ID.
The summary lists open questions, not just conclusions.
The summary is written by a step with a fixed template, not free text.
A test compares a run with trimmed context against a run with full context.

The last item is the one people skip. I cover it below.

Token budgets

A token budget is a limit you set for how much each part of the context may use. It is a design choice, not a model setting.

How do I set a token budget?

Do this in four steps.

Measure first. Log the token count of each context part for 50 real runs. Do not guess. Most teams find one part (usually tool output or history) is far bigger than they thought.
Set a working ceiling well under the window. I would not plan to fill the window. Models get less reliable as inputs grow, so pick a ceiling you have tested. Your own eval results should set it. I can’t give you a universal percentage.
Split the ceiling across parts. Fixed parts first, then per-step loads, then history, then headroom.
Add a rule for what happens at the limit. Trim, summarize, or hand off. Never let the framework truncate silently.

Here is a worksheet with made-up numbers. Say your tested working ceiling is 60,000 tokens for one run.

Part	Budget	Notes
System instructions	2,000	Stable, cached
Tool definitions	3,000	Stable, cached. Fewer tools means fewer tokens.
Policy excerpts	4,000	Only rules for this exception type
Retrieved records per step	8,000	Hard cap per tool call
Working notes	3,000	Structured, overwritten each step
History (summarized)	10,000	Older steps compacted
Headroom	30,000	Unused on purpose

The numbers are an illustration. What matters is that each part has an owner and a cap. If a tool call returns more than 8,000 tokens, your runtime code trims it and logs a warning. Tokens also drive cost. Anthropic found token usage alone explained 80% of the performance variance on one browsing benchmark, which shows tokens are both the quality lever and the cost lever. Track both together. For the cost side, see what an AI agent costs to run per month once you have your numbers.

Testing context changes

Treat every context change like a code change. Test it.

Build a fixed set of 30 to 50 real cases with known correct outcomes. My guide to agent evaluation covers how.
Run the baseline. Record accuracy, tokens per run, and cost per run.
Change one thing. Trim a tool output, add a summary, move a policy to retrieval.
Run again and compare. Accept the change only if accuracy holds and tokens or cost drop.
Check the hard cases separately. Averages hide the long, messy runs where trimming does the most harm.

Change one variable at a time, or you won’t know which change helped.

Common failure patterns

LangChain lists four failure modes that show up in practice:

Poisoning: a hallucinated fact enters the context and gets reused. Fix: write only verified facts to notes, and store the source ID with each.
Distraction: the context grows so large that the model leans on it and ignores its instructions. Fix: cap history and summarize at phase boundaries.
Confusion: extra, unneeded content shapes the answer. Fix: the “leave out” column from your step table.
Clash: parts of the context disagree. Fix: one source of truth per fact, and a freshness check on retrieved data.

Add three ops-specific traps I’d watch for:

Loading all policies “just in case.” Retrieve the rule that applies to this case.
Passing raw documents between steps. Pass extracted fields plus an ID.
Silent truncation. If your framework cuts the oldest messages when full, the plan and the original task are often the first to go. Pin them, or save them to notes.

For the wider design this sits inside, read the AI agent architecture guide. For storing facts across runs, see agent memory systems.

Bottom Line

Context is the part of agent quality you control most directly.

Build the per-step table. Load pointers, not documents. Cap every part with a budget you measured. Test each change against a fixed set of real cases.

Next step: log the token count of each context part for 50 runs. The biggest part is where you start.

FAQ

What is context engineering? Context engineering is choosing what information the model sees at each step of an agent run. That includes instructions, tool definitions, retrieved documents, message history, and notes. The goal is the smallest set of high-signal information that gets the right action.

How is context engineering different from prompt engineering? Prompt engineering is writing and organizing the instructions. Context engineering covers everything that ends up in the window, including tool results and history that change on every step. For agents that loop, the second one is usually the bigger lever.

What context does each step need? Only what the model uses to pick its next action. Classification needs summary fields. Fact-gathering needs the detailed records. The final write-up needs the decision and IDs. Build a table with one row per step and a “leave out” column.

How do I trim context without losing facts? Keep an ID or source pointer for every fact, and let the agent re-fetch details when needed. Write key findings to structured notes. Summarize at clean phase boundaries with a fixed template that includes open questions. Then test trimmed runs against full runs.

How do I set a token budget? Measure token use per part across 50 real runs. Pick a working ceiling you have tested, well under the window limit. Split it across instructions, tools, retrieval, notes, and history, and leave headroom. Decide what happens at the limit: trim, summarize, or hand off.

Does a bigger context window remove the need for this? No. Research from Chroma and others shows reliability can fall as inputs grow, even on simple tasks. A bigger window gives you room, but cost, latency, and accuracy still favor a tighter context.

Three Ways to Work With TMA

Need an agent built? We deploy production AI agents in your infrastructure. Working pilot. Real data. Measurable ROI. → Schedule Demo

Want to co-build a product? We’re not a dev agency. We’re co-builders. Shared cost. Shared upside. → Partner with Us

Want to join the Guild? Ship pilots, earn bounties, share profit. Community + equity + path to exit. → Become an AI Architect