Production architectures for AI agents, from single agents to supervisor graphs

By 2026, a clear majority of organizations report having agents in production, and the barrier they cite most often is no longer cost. It is reliability. An agent that answers correctly in a demonstration and then behaves unpredictably under real traffic is the rule rather than the exception, and the reason is almost always architectural. The demonstration treats the agent as a single function call. Production requires treating it as a long running, stateful, recoverable process that interacts with untrusted inputs and consequential tools. Those are very different things to build.

Start simpler than you think you need

Before reaching for an autonomous agent, it is worth asking whether you need one. Anthropic's guidance on building effective agents draws a useful line between workflows, where the steps are known in advance and the system follows a fixed path, and agents, where the model decides its own steps at runtime. Many tasks that teams solve with an open ended agent are better served by a simple workflow, which is cheaper, more predictable, and far easier to operate. The patterns worth knowing, in roughly increasing order of autonomy, are chaining steps in sequence, routing a request to the right handler, running independent steps in parallel, an orchestrator that delegates to workers, and an evaluator that critiques and improves another model's output. Reach for the simplest one that solves the problem.

The control flow problem

When you do need a genuine agent, the central engineering challenge is control flow. A real agent needs to loop while it works, branch on what it finds, pause and resume, recover after a failure, and stop to wait for a human before doing something irreversible. A simple linear pipeline cannot express any of that. This is the reason graph based orchestration has become the dominant approach for serious deployments. Representing the agent as a graph, where each node is a step and the edges decide what happens next, makes the state of the task an explicit, inspectable thing rather than something hidden inside a single long model call. Frameworks built on this idea provide persistence layers that save a snapshot of the state at every step, which is the foundation everything else rests on.

State and durability, the foundation layer

The persistence layer is what makes everything above it possible. When the state of a run is checkpointed at every step, three capabilities fall out almost for free. You can pause a run and resume it later, which is exactly what a human approval step needs. You can recover from a failure by resuming from the last good checkpoint rather than starting over. And you can replay a past run step by step to understand what happened, which is the difference between debugging an agent and merely guessing at it. In a framework like the graph based one that leads this space, compiling the graph with a checkpointer saves a snapshot at every step, organized into threads, and that single mechanism underpins human oversight, memory, recovery, and audit all at once.

Tools, through the gateway

An agent's tools are where it touches the real world, and therefore where it can do real damage. Everything I wrote in the gateway article applies here. Tool calls go through a single chokepoint that authenticates and scopes them, restricts them to an approved list, validates their inputs and outputs, and logs every invocation. In the deployment architecture, the gateway is simply the layer between orchestration and the outside world.

Human oversight as a first class step

For any action that is consequential and hard to reverse, the right design is to stop and ask a person. Because the state is checkpointed, this is clean rather than awkward. The graph reaches the approval node, the state is saved, a notification goes out, the run simply waits, and when the person approves it resumes exactly where it left off. This is the same dual control principle that governs money movement, expressed as a node in a graph.

Observability

A non deterministic system that takes actions in the world is not something you can run blind. You need a trace of every step, with the token count, the latency, and the cost attached, so that you can see where a run spent its time and its budget. You need evaluations of whole trajectories, not just final answers, so that you can tell whether the agent is reasoning well or simply arriving at the right answer by luck. The industry is standardizing this through the generative AI conventions of the OpenTelemetry project, and several mature platforms exist to capture and analyze the traces. The teams that succeed treat observability as something present from the first day, not bolted on after the first incident.

Design for the ways agents fail

Research into why multi agent systems fail points consistently at coordination, where agents duplicate work, miscommunicate, or drift from the shared goal. The architectural answers are the ones above. Clear task boundaries reduce duplication, checkpointed state makes failures recoverable, the gateway contains the damage a confused agent can do, human gates catch the consequential mistakes, and observability lets you find the rest. You cannot prevent every failure, but you can make failures visible, bounded, and recoverable.

The summary

Start with the simplest pattern that solves the task. When you need a real agent, build it as a state graph with checkpointing, route its tools through a gateway, make human approval a first class step for anything consequential, and instrument every layer from the beginning. That is the architecture that survives the journey from a demonstration that works once to a system you can actually operate.