Agentic loops, large context, and the discipline of optimization

This post builds that argument from first principles using one running example, a codebase refactor agent that migrates a project from an old client library to a new one. The example is hypothetical but realistic, and it is a good teacher because it makes context growth visible. Every time the agent reads a file, several thousand tokens land in the window. By the time it has read a dozen files, run the tests twice, and started editing, the window is heavy, and you are paying to reprocess all of it on every single step.

I will move from the shape of the loop, to the arithmetic of why it grows, to the three costs that growth imposes, and then to the five levers that bring it back under control. The code uses the Anthropic Python software development kit and Claude Sonnet 4.6, which is the practical default for a coding agent at the time of writing. Here is the whole system on one page. The rest of the article explains each box.

The loop, stated precisely

A single tool call is not an agent. A fixed chain of three prompts is not an agent either. What makes a loop agentic is that the model itself decides, at each step, whether to act again and what to do next, and the loop continues without further human input until a stopping condition holds. The Anthropic documentation calls the repetition of request and tool result the agent loop, and notes that the canonical shape is a short loop keyed on the stop reason returned by the model.

The contract is simple. You hand the model a set of tools, each with a name, a description, and an input schema. The model decides when to call one. It does not run anything itself. It emits a structured request, your code executes the operation, and you feed the result back as a tool result block. The model then decides whether it needs another tool or whether the task is done. Here is a complete loop, with the four tools our refactor agent uses.

import anthropic

client = anthropic.Anthropic()

TOOLS = [
    {
        "name": "read_file",
        "description": "Return the full contents of a file at the given path.",
        "input_schema": {
            "type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"],
        },
    },
    {
        "name": "write_file",
        "description": "Overwrite a file at the given path with new contents.",
        "input_schema": {
            "type": "object",
            "properties": {
                "path": {"type": "string"},
                "contents": {"type": "string"},
            },
            "required": ["path", "contents"],
        },
    },
    {
        "name": "run_tests",
        "description": "Run the test suite and return pass or fail output.",
        "input_schema": {"type": "object", "properties": {}},
    },
    {
        "name": "grep",
        "description": "Find a symbol across the repository without loading whole files.",
        "input_schema": {
            "type": "object",
            "properties": {"pattern": {"type": "string"}},
            "required": ["pattern"],
        },
    },
]

SYSTEM = "You are a careful refactoring agent. Migrate the project from the old client to the new one. Change one module at a time, run the tests after each change, and never edit a file you have not read."

def run_tool(name, args):
    # Dispatch to the real implementation and return a string result.
    ...

def agent(goal, max_steps=40):
    messages = [{"role": "user", "content": goal}]
    for step in range(max_steps):
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=4096,
            system=SYSTEM,
            tools=TOOLS,
            messages=messages,
        )
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason != "tool_use":
            return response  # the model is finished

        results = []
        for block in response.content:
            if block.type == "tool_use":
                output = run_tool(block.name, block.input)
                results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": output,
                })
        messages.append({"role": "user", "content": results})

    raise RuntimeError("step limit reached before the task finished")

Read the loop carefully, because the cost story is hiding in plain sight. On each pass we append the assistant message, then append a user message carrying the tool results, then call the model again with the full messages list. The list only ever grows. Nothing is removed. The model never remembers anything between calls, so the only way it can see what happened three steps ago is for you to send those three steps again. That resending of the full history is the mechanism, and it is the source of the cost problem.

Why context explodes

Let us put numbers on it. Call the stable prefix P, which is the system prompt, the tool schemas, and the repository file map that the agent needs in order to navigate. For our agent that is roughly 8,000 tokens, and it is identical on every turn. Each iteration then adds new material to the transcript: the assistant turn that contains a tool call, plus the tool result. A single read_file on a source file can easily return 2,000 tokens, and a run_tests result might be 1,500. Average the growth across a task and call it g, around 2,200 tokens added per step including the assistant tool call.

Now count what you are billed. On step k, the input you send is the prefix plus everything accumulated before this step, which is P plus (k minus 1) times g. Sum that across N steps and you get the total input processed across the whole task.

The first term is linear in N. The second term is the one that hurts, because it grows in proportion to N². This is the precise reason a naive agent gets expensive faster than people expect. You are not paying for the length of the final transcript. You are paying for the running sum of every intermediate transcript, and that running sum is quadratic in the number of steps.

Plug in the numbers for an eight step task with P equal to 8,000 and g equal to 2,200. The linear term contributes 64,000 tokens. The quadratic term contributes 2,200 times 28, which is 61,600 tokens. So the bill is for roughly 125,600 input tokens even though the final transcript is only about 25,000 tokens long. At the Claude Sonnet 4.6 input rate of three dollars per million tokens, that is about forty cents for one short task. Forty cents sounds trivial until you remember that a production agent runs this loop thousands of times a day, and that real tasks run far longer than eight steps. Figure 2 traces the cumulative cost as the task gets longer, and it shows the quadratic curve pulling away from a controlled one.

Two honest caveats keep this model from overstating the case. First, much of the prefix and even the transcript can be cached, which changes the price of those tokens without changing their count, and we will get to that shortly. Second, the average g is a simplification, since some steps add a tiny tool result and some add a large file. The shape of the conclusion does not change. Without intervention, a long agentic task pays a quadratic cost in steps, and the dominant driver is the transcript you keep resending.

The three costs of a heavy window

Large context is often discussed as a feature, and the one million token window on Claude Sonnet 4.6 is a real capability. The trouble is that filling it is not free along any axis. A heavy window imposes three separate costs, and a good agent attends to all three rather than only the first.

The direct cost

This is the quadratic effect above. Every token in the window is reprocessed on every step, so a window that drifts upward turns each step into a slightly more expensive version of the last one. This cost is easy to measure and easy to overlook, because a single step always looks cheap. The accumulation across many steps is what makes it significant.

The latency cost

Before the model can produce its first output token, it has to read and process the entire input. That prefill work grows with the size of the window, so the time to first token climbs as the transcript grows. A loop that felt responsive in its first three steps can feel sluggish by its fifteenth, not because the model got slower but because you are asking it to reread a much larger transcript before it can say anything. For an interactive agent this is the cost users actually feel.

The quality cost

This is the subtle one, and the one that surprises people. More context is not uniformly better. Research on long context retrieval found that models attend most reliably to information at the very start and the very end of the input, and that material buried in the middle is recalled far less reliably. The effect is strong enough that adding more documents can lower accuracy if the useful one ends up in the dead zone. Practitioners describe a related drift sometimes called context rot, where a long transcript accumulates stale instructions, abandoned approaches, and dead tool output that quietly compete with the part of the context that still matters. A focused window of the right 8,000 tokens often outperforms a full window of the wrong 200,000.

The optimization toolkit

There are five levers worth knowing well. None of them is universally correct. Each trades something away, and the skill is matching the lever to the situation. The architecture in Figure 1 uses all five together, but it is clearer to take them one at a time.

Lever one, prompt caching

Caching attacks the cost of the stable prefix directly. You mark the parts of the request that do not change between turns, and Anthropic stores the processed result. On a later request that begins with the same content, you pay a small fraction of the input price for the cached portion instead of the full price. The pricing structure is explicit. A write costs 1.25 times the base input price for a five minute lifetime, or 2.0 times for a one hour lifetime, and a subsequent read of cached content costs only 0.10 times the base input price, which is a ninety percent discount. Caching references the request in a fixed order, tools first, then system, then messages, up to and including the block you mark.

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=4096,
    system=[
        {
            "type": "text",
            "text": SYSTEM,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    tools=TOOLS,           # tool schemas sit in the cached prefix
    messages=messages,
)

That single marker turns the 8,000 token prefix from a full price item into a near free one after the first warm up call. You can push it further by marking the last block of the transcript on each turn as well, which lets the cache cover the growing history incrementally so that prior turns are read back cheaply rather than reprocessed at full price. Two operational details matter. The default cache lifetime is five minutes, and a step that takes longer than that to come back can let the cache expire, so long running agents either accept the occasional rewrite or choose the one hour lifetime. And matching is exact, which means a single changed byte early in the prefix invalidates everything after it, so a stable ordering of tools and system content is not a nicety, it is the thing that makes caching work at all.

Lever two, compaction

Caching lowers the price of tokens. It does not lower their count, and it does nothing for the quality cost. Compaction attacks the count. The idea is to replace bulky, finished context with a short summary of its outcome. In our refactor agent, once a module has been migrated and its tests pass, the full original contents of that file no longer need to sit in the window. What the agent needs going forward is one line: this module was migrated, its public exports are unchanged, its tests pass. Anthropic now exposes compaction and context editing as first class features for exactly this reason, and the pattern is easy to write by hand as well.

def compact(messages, keep_recent=6):
    """Replace older tool_result file dumps with one line summaries."""
    if len(messages) <= keep_recent:
        return messages
    head, tail = messages[:-keep_recent], messages[-keep_recent:]
    compacted = []
    for msg in head:
        if msg["role"] == "user" and isinstance(msg["content"], list):
            new_content = []
            for block in msg["content"]:
                if block.get("type") == "tool_result":
                    new_content.append({
                        "type": "tool_result",
                        "tool_use_id": block["tool_use_id"],
                        "content": summarize(block["content"]),
                    })
                else:
                    new_content.append(block)
            compacted.append({"role": "user", "content": new_content})
        else:
            compacted.append(msg)
    return compacted + tail

Compaction is lossy by design, and that is the tradeoff. If the summary drops a detail the agent later needs, it has to read the file again, which costs a round trip. The art is in the summary policy. Summaries should record decisions and outcomes, the things the agent will reason about later, and should discard raw material that can be fetched again on demand. A good rule is to summarize what is settled and keep verbatim what is still in play.

Lever three, pruning and truncation

Pruning is a blunter alternative to compaction. Rather than summarizing old content, you simply drop it or cap it. A sliding window keeps the last several turns in full and discards everything older. Truncation caps any single tool result at a fixed size, so one accidental ten thousand line log dump cannot flood the window. Pruning is cheap because it requires no extra model call, and it is the right choice when older context is genuinely irrelevant, for example when each step of a task is independent of the steps before it. It is the wrong choice when the agent needs long range memory, because a sliding window will silently forget a constraint stated at the very beginning. The honest move is to combine the two. Cap every tool result on the way in with truncation, and use compaction rather than pruning for anything that might still matter.

Lever four, subagent isolation

Sometimes the cleanest way to keep a window small is to not let it grow in the first place. A subagent is a fresh loop with its own empty transcript, handed one well scoped piece of work. It does its work in its own window, then returns only a short structured result to the parent. In the refactor agent, migrating a single module is a natural subagent. The parent does not need to see the forty messages it took to migrate that module. It needs one sentence about the outcome. The Anthropic guidance on building effective agents treats this orchestration of focused subagents as a core pattern rather than an exotic one.

def migrate_module(path):
    # A fresh window, isolated from the parent transcript.
    result = agent(
        goal=f"Migrate only the module at {path}. Return a one paragraph summary.",
        max_steps=20,
    )
    return summarize(result)   # only the summary returns to the parent loop

The tradeoff is coordination. The parent now has to decompose the task, dispatch subagents, and stitch results together, and a subagent that lacks a piece of shared context can make a locally sensible choice that is globally wrong. Isolation buys clean windows at the price of a harder orchestration problem. It pays off most when the subtasks are genuinely separable, which a module by module migration usually is.

Lever five, retrieval versus long context

The final lever is a decision rather than a mechanism. When the agent needs information from a large body of material, it has two options. It can load everything into the window and let the model find what it needs, or it can retrieve only the relevant slice and load just that. In our agent this is the difference between reading an entire file to find one function and using grep to locate the symbol, then reading only the span around it. The two approaches trade off cleanly, and the right answer depends on how much of the material the task actually touches.

Dimension	Load the full context	Retrieve the relevant slice
Cost	High and grows with corpus size on every step	Low, bounded by what you fetch
Recall of obscure detail	Strong, everything is present	Only as good as the retrieval step
Quality risk	Lost in the middle on very large inputs	A miss returns the wrong slice
Engineering effort	Minimal, just send more	You build and tune the retrieval step
Best when	The task touches most of the material	The task touches a small, findable part

A large window does not make retrieval obsolete, and retrieval does not make a large window pointless. They solve the problem at different scales. For a refactor that touches a handful of call sites scattered across a big repository, grep and a narrow read are both cheaper and more accurate than loading the repository. For a task that genuinely reasons over an entire document, loading it whole is the honest choice.

Assembling the optimized loop

Put the levers together and the loop gains a new component, a context governor that runs after each turn. The governor reads a token budget, decides whether the window has grown past a threshold, and applies compaction and truncation to bring it back down before the next call. Caching is configured once on the request, and subagents are invoked by the tool layer for separable work. The result is the architecture in Figure 1, expressed in code.

TOKEN_BUDGET = 30_000   # target working size, far below the 1M ceiling

def optimized_agent(goal, max_steps=40):
    messages = [{"role": "user", "content": goal}]
    for step in range(max_steps):
        response = client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=4096,
            system=[{"type": "text", "text": SYSTEM,
                     "cache_control": {"type": "ephemeral"}}],
            tools=TOOLS,
            messages=messages,
        )
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason != "tool_use":
            return response

        results = []
        for block in response.content:
            if block.type == "tool_use":
                output = truncate(run_tool(block.name, block.input), cap=2_000)
                results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": output,
                })
        messages.append({"role": "user", "content": results})

        # The context governor runs after every turn.
        if estimate_tokens(messages) > TOKEN_BUDGET:
            messages = compact(messages)

    raise RuntimeError("step limit reached before the task finished")

The difference between this loop and the first one is small in lines of code and large in behavior. Truncation caps each tool result on the way in. Caching makes the stable prefix nearly free after warm up. The governor keeps the transcript near its budget instead of letting it climb without limit. This is the optimized curve in Figure 2. The window stays small, so the per step cost stays roughly flat, and the quadratic blowup never gets started.

Failure modes and how the loop recovers

An optimized loop that never fails gracefully is not production ready. Three failure modes show up again and again, and each has a direct remedy that belongs in the loop itself.

The infinite loop

The model keeps acting but stops making progress, calling tools forever without converging on a result. The step limit in the loop is the backstop, but a better guard watches for the absence of progress, for example the same file being read repeatedly or the same edit being proposed and reverted. When progress stalls, the loop should stop and report rather than burn budget. The cheapest version of this is a simple counter on repeated identical tool calls.

The error spiral

A tool fails, the model tries the same failing call again, it fails again, and the transcript fills with identical error messages that teach the model nothing. The remedy is an error budget. Count consecutive failures of the same kind, and once the count crosses a threshold, change the situation rather than repeating it, by handing the model a different instruction or by stopping and surfacing the error to a human. Feeding the same failure back into the window is both expensive and useless.

def guarded_dispatch(block, recent_calls, error_counts):
    key = (block.name, json.dumps(block.input, sort_keys=True))
    if recent_calls.count(key) >= 3:
        return "Halted: this exact call was already attempted three times."
    try:
        return truncate(run_tool(block.name, block.input), cap=2_000)
    except Exception as exc:
        error_counts[block.name] += 1
        if error_counts[block.name] >= 3:
            return f"Tool {block.name} has failed repeatedly. Try a different approach."
        return f"Error from {block.name}: {exc}"

The mid task overflow

Despite every guard, a single task can outgrow even a generous budget, for instance a migration that turns out to span far more files than expected. The graceful path is not a crash. It is a deliberate handoff. The loop compacts aggressively, writes a structured summary of what has been done and what remains, and either continues in a fresh window seeded with that summary or returns it so a later run can resume. A task that can checkpoint its own state and resume is far more robust than one that holds everything in a single window and hopes it fits.

A decision checklist

The levers are most useful when you can reach for the right one quickly. The table below is the short version of everything above, organized by the symptom you are seeing rather than by the technique.

What you observe	Reach for	Because
A large prompt or tool set repeated every turn	Prompt caching	Cached reads cost a tenth of fresh input
The transcript climbs as the task runs long	Compaction	Settled context becomes a one line summary
One tool occasionally returns an enormous dump	Truncation	A single result cannot flood the window
The task splits into separable pieces	Subagents	Each piece runs in its own clean window
The agent needs a small part of a big corpus	Retrieval	Fetch the slice rather than the whole
Answers degrade as the window grows	Compaction and retrieval	A focused window beats a bloated one
Latency rises through a long session	Caching and a token budget	Smaller prefill means a faster first token

The thread running through all of it is the thesis we started with. An agentic loop grows its own context as a side effect of doing its work, and on a stateless endpoint that growth is paid for again on every step. The model receives most of the attention, but the loop around it is where the engineering effort pays off. Treat the window as a scarce resource that has to be actively governed, decide on purpose what stays and what goes, and the same loop that ran away from you becomes predictable in cost, steady in latency, and sharper in its answers.