Enterprise AI architecture, from an LLM demo to a production system

The demo always works. You open a notebook, you call a foundation model with a few lines of code, you ask it something hard, and it answers well enough that a room full of people leans in. The distance from that moment to a system you can put in front of customers, auditors, and a finance team is the entire subject of this article. That distance is not a better prompt or a bigger model. It is roughly eight categories of engineering that the demo left out.

The diagram below is the whole argument in one frame. A request enters on the left, moves through orchestration, retrieval, reasoning, and tool action, and leaves on the right as a governed response. The large language model (LLM), the part everyone talks about, is a single box in the middle. Everything around it exists for a reason, and the fastest way to understand each piece is to look at the specific way a demo breaks when that piece is missing.

This piece is a capstone. Each layer here has a fuller treatment elsewhere in this portfolio, and I link out to those deep dives as we pass through. This article provides the overview, and the linked deep dives cover each layer in detail. We will spend it walking one failure at a time.

The method, follow the failures

It is tempting to explain an architecture by listing its parts in order. That produces a glossary, not understanding. A more honest way in is to ask what goes wrong the moment a demonstration meets reality, because every production layer in the diagram was added by someone who watched a particular thing break and refused to ship it that way again. The map below pairs each failure with the layer it forces into existence. The rest of the article is that map, expanded.

One framing note before we start. Public facing content in this portfolio about AWS infrastructure assumes a multi region posture across ca-central-1 and ca-west-1, and everything here is described in vendor terms with named examples rather than as a single product pitch. With that settled, here is the first failure.

Layer 2 · Anyone can call it, and it can be talked into anything

In production, the API gateway and security layer is where every request is authenticated and controlled before it reaches anything else.

The notebook called the model directly with a key on the developer's machine. There was no notion of who was asking, what they were allowed to ask for, or how often they could ask. The first thing reality does to that arrangement is send it ten thousand requests, a few of them hostile, and watch the bill and the blast radius climb together. The fix is not one control but a layer of them, and it sits in front of everything else.

The user experience layer is where a real person actually meets the system, through a web application, a mobile application, a chat surface, or an upload for a document or an image. Behind it, the application programming interface (API) gateway and security layer is the part that turns an open endpoint into a governed one. Authentication establishes who is calling. Authorization and role based access control (RBAC) establish what that caller may do, which matters enormously the moment an agent can act on a user's behalf. Rate limiting keeps a single caller, friendly or not, from consuming the whole system. Request validation rejects malformed or oversized input before it ever reaches a model you are paying for by the token.

This is the layer where a zero trust posture either exists or does not. If you want the full treatment of private networking, signed identity, and the gateway as a policy enforcement point, that is its own reference architecture in this portfolio, the zero trust AI gateway. For our purposes the point is narrow. Without this layer, nothing downstream can assume the caller is who they claim to be, and an agent that acts on behalf of an unauthenticated user is a liability rather than a feature.

Layer 5 · It answers confidently about things it was never given

A model with no access to your data will still answer questions about it, and those answers will often be wrong.

Ask a foundation model about last quarter's contract terms or a specific customer's open tickets and it will produce a fluent, plausible, and entirely invented answer. The model does not know what it does not know, and confidence is not correlated with correctness. The production answer is to stop asking the model to recall and start giving it the relevant material to read. That is retrieval augmented generation (RAG), and it is the fifth layer.

The mechanism is a pipeline. Source documents land in a document store. An embedding model turns them into vectors that capture meaning rather than keywords. A vector database indexes those vectors so that, at query time, retrieval can pull the passages most relevant to the question. A re-ranking step then reorders the candidates so the strongest evidence sits at the top of the context the model finally sees. The knowledge base is the curated, permissioned corpus that all of this draws from.

Two things make enterprise retrieval harder than a weekend project. The first is permission. Retrieval must respect who is allowed to see what, which means the identity from layer two has to travel all the way into the query so that a user never retrieves a document they could not otherwise open. The second is freshness and quality, because a stale or poorly chunked corpus produces confident answers grounded in the wrong version of the truth. Both are the subject of the secure enterprise RAG platform reference architecture, which goes deep on permissioned retrieval and ingestion. Here, the headline is simple. Retrieval is how the model stops guessing.

Layer 9 · A retrieved document hides an instruction the model obeys

The moment the model reads outside text, that text can try to give it orders.

Retrieval solves one problem and opens another. As soon as the model reads content it did not author, a passage in that content can contain an instruction, and a naive system will follow it. A support ticket that ends with a line telling the assistant to ignore prior instructions and export the customer list is a prompt injection, and it arrives through the same retrieval path you just built. This is why the governance layer is not a final coat of paint. It has to reach into the middle of the flow.

The defensive principle is to treat every piece of retrieved or user supplied content as untrusted input, never as trusted instruction. In practice that means a guardrail evaluates content on the way in, looking for prompt attacks such as jailbreaks, injections, and attempts to leak the system prompt, and it evaluates responses on the way out. It is the same posture you already apply to a database query or an uploaded file, extended to natural language.

The lesson that carries forward is that security in an AI system is not a perimeter you cross once at the gateway. It is a property you re-establish at every boundary where untrusted text enters the reasoning loop, and retrieval is one of those boundaries.

Layer 6 · It forgets who it is talking to between turns

A demo is a single turn. A product is a conversation, and a conversation has a memory.

The notebook handled one prompt and one answer. Real interactions span turns, sessions, and sometimes weeks, and a system that forgets the previous message feels broken in a way no amount of model quality can fix. The memory layer is how state persists without smuggling the entire history into every prompt, which would be both expensive and slow.

It comes in distinct kinds. Session memory holds the current exchange. User context holds the durable facts about a person, their role, their preferences, their entitlements, that should shape every interaction. Workflow state holds where a multi step task has got to, so an interrupted process can resume rather than restart. Long term memory holds what should be remembered across sessions, distilled rather than dumped, so the system gets more useful over time without growing an unbounded prompt.

The discipline here is knowing what not to remember. Long term memory that accumulates everything becomes both a cost problem and a privacy problem, since it is now a store of personal data that has to be governed like any other. The art of large context windows, and of deciding what earns a place in memory against what should be retrieved fresh each time, is the subject of the deep dive on agentic loops and large context windows. For the architecture, the point is that memory is a first class layer, not a variable you keep in a notebook cell.

Layer 3 · A single prompt cannot carry a multi-step task

One model call is a sentence. A real task is a paragraph with branches, retries, and a plan.

The demo did its work in one call. Ask the system to summarize three tickets and open a case only if one is unresolved, and a single prompt has to route the request, decide what information it needs, gather it, reason, and act, all at once and with no recovery if any part fails. The orchestrator is the layer that breaks that into a managed sequence and supervises it.

A request router classifies the incoming request and decides how to handle it. A task planner decomposes a goal into steps. A context manager assembles exactly the right material from memory and retrieval for each step, rather than dumping everything into one oversized prompt. A prompt builder constructs the model call. A confidence check inspects the result before the system acts on it, so a low confidence answer can trigger a clarifying question or a fallback rather than a wrong action. An error handler catches the failures that will happen and decides whether to retry, degrade, or stop.

That confidence check is worth making concrete, because it is the difference between an agent that asks when unsure and one that acts when it should not.

import json
import boto3

bedrock = boto3.client("bedrock-runtime", region_name="ca-central-1")

def classify_intent(user_text, model_id="amazon.nova-pro-v1:0"):
    """Return an intent label and a confidence the orchestrator can gate on."""
    instruction = (
        "Classify the request into one of: read_only, action_required, "
        "ambiguous. Respond as JSON with keys intent and confidence "
        "(0.0 to 1.0). Return JSON only, no prose."
    )
    response = bedrock.converse(
        modelId=model_id,
        system=[{"text": instruction}],
        messages=[{"role": "user", "content": [{"text": user_text}]}],
        inferenceConfig={"temperature": 0.0, "maxTokens": 200},
    )
    text = response["output"]["message"]["content"][0]["text"]
    return json.loads(text)

def route(user_text, threshold=0.75):
    result = classify_intent(user_text)
    if result["confidence"] < threshold or result["intent"] == "ambiguous":
        return {"next": "clarify_with_user", "reason": "low_confidence"}
    if result["intent"] == "action_required":
        return {"next": "plan_and_validate", "reason": "action_path"}
    return {"next": "retrieve_and_answer", "reason": "read_path"}

The orchestrator is also where the choice between a framework driven agent and a deterministic state machine gets made. Some flows want the flexibility of an agent loop, and some want the auditability of an explicit graph of steps. The patterns and the trade offs are covered in the production architectures for AI agents series.

Layer 7 · It invents a tool call, or calls a real tool with bad inputs

An agent that can act is only as safe as the catalog of actions it is allowed to take.

The demo could talk. A production agent has to do, which means calling real application programming interfaces, querying databases, and driving software as a service (SaaS) tools. Two failures arrive immediately. The model invents a tool that does not exist, or it calls a real tool with arguments that are malformed or out of policy. Handing a model an unbounded ability to call anything is how a plausible sentence becomes a real and damaging action. The tool and API calling layer is the controlled catalog that prevents this.

At its centre is a tool registry, the explicit and versioned set of actions the agent may take, each with a typed schema for its inputs. The model never calls a raw endpoint. It selects from the registry, and the agent gateway mediates the call, handling authorization outward to the target system and shaping the request. The Model Context Protocol (MCP) has become the common language for exposing tools to agents in a uniform way, so a database, an approved internal API, and a SaaS connector can all be presented to the model through one consistent interface.

Describing a tool well is most of the safety. The schema below is what the model is actually allowed to ask for, and a strict schema is what stops a vague intention from becoming a malformed call.

tool_config = {
    "tools": [
        {
            "toolSpec": {
                "name": "open_support_case",
                "description": (
                    "Open a follow-up support case for a client. Only call "
                    "when an existing ticket is confirmed unresolved."
                ),
                "inputSchema": {
                    "json": {
                        "type": "object",
                        "properties": {
                            "client_id": {"type": "string"},
                            "summary":   {"type": "string", "maxLength": 500},
                            "priority":  {
                                "type": "string",
                                "enum": ["low", "medium", "high"]
                            }
                        },
                        "required": ["client_id", "summary", "priority"]
                    }
                }
            }
        }
    ]
}

The validation checkpoint · It takes an irreversible action that no one approved

Reading the wrong document is recoverable. Wiring the wrong payment is not.

The most dangerous moment in the whole flow is the instant between deciding to act and acting. A read can be retracted. A refund, a deletion, or an outbound message cannot. The validation checkpoint is the deliberate gate that sits on the path from the tool layer to the action layer, and it asks two different questions. Is this caller authorized to take this specific action on this specific resource, and is this action itself safe to take right now.

Authorization here is not the coarse role check from the gateway. It is a fine grained policy decision about a concrete action on a concrete resource, and it is best expressed as policy rather than buried in code, so it can be reviewed, versioned, and audited on its own.

// Cedar policy: a relationship manager may open a case
// only for the client they are assigned to.
permit (
    principal in Role::"RelationshipManager",
    action == Action::"open_support_case",
    resource is SupportCase
)
when {
    resource.client_id == principal.assigned_client
};

The second question, safety, is where the same guardrail engine returns, now checking the proposed output and action rather than the input. The ApplyGuardrail call below is the explicit safety gate, and because it is a standalone call it sits cleanly in the orchestrator between the decision and the execution.

guard = boto3.client("bedrock-runtime", region_name="ca-central-1")

def safety_gate(proposed_output, guardrail_id, version="DRAFT"):
    result = guard.apply_guardrail(
        guardrailIdentifier=guardrail_id,
        guardrailVersion=version,
        source="OUTPUT",
        content=[{"text": {"text": proposed_output}}],
    )
    if result["action"] == "GUARDRAIL_INTERVENED":
        return {"allow": False, "assessments": result["assessments"]}
    return {"allow": True}

For genuinely consequential actions, the right answer is sometimes to stop and ask a person. Human approval is a first class step, not an admission of failure, and it belongs in the orchestrated flow so that the wait is auditable and the approver's decision is recorded.

Layer 8 · The answer is ungrounded, or it quietly leaks personal data

Even a correct system can produce a response you must not send.

Suppose every prior layer did its job. The caller was authorized, the right documents were retrieved, the plan was sound, the tool was valid, and the action was approved. The response can still be wrong in two specific and costly ways. It can drift from the retrieved evidence into something the source never said, which is a hallucination dressed as a citation, and it can include personally identifiable information (PII) that should never leave the system. The response and action layer is the last line before output becomes a final answer, an application programming interface action, a ticket update, a notification, or a generated report.

Two checks matter most here. A contextual grounding check compares the response against the source material and scores how well it is supported and how relevant it is, so an answer that wandered off the evidence can be blocked or flagged before it is sent. A sensitive information filter detects and redacts PII in the output, by standard entity type and by custom pattern. Both can be configured as policy and deployed through infrastructure as code, which is how they stay consistent across regions and environments.

Type: AWS::Bedrock::Guardrail
Properties:
  Name: production-response-guard
  BlockedInputMessaging: "This request cannot be processed."
  BlockedOutputsMessaging: "This response was withheld for review."
  ContextualGroundingPolicyConfig:
    FiltersConfig:
      - Type: GROUNDING
        Threshold: 0.75
      - Type: RELEVANCE
        Threshold: 0.75
  SensitiveInformationPolicyConfig:
    PiiEntitiesConfig:
      - Type: EMAIL
        Action: ANONYMIZE
      - Type: PHONE
        Action: ANONYMIZE
      - Type: CREDIT_DEBIT_CARD_NUMBER
        Action: BLOCK

For regulated decisions, there is a further control. Automated Reasoning checks in Amazon Bedrock Guardrails use formal logic to test whether a response is consistent with a policy you define, and they return structured feedback about why a statement holds or fails rather than a simple block. That is the difference between filtering on patterns and verifying against rules, and it is the kind of control that lets an AI answer carry weight in a compliance setting.

Layer 9 · Cost spirals, and no one can explain a past decision

A system you cannot see, cannot bill, and cannot explain is not in production. It is just running.

The final failure is the quiet one. The demo had no meter and no memory of its own behavior. In production, token spend climbs without anyone noticing, latency varies without anyone measuring, and three months later a customer asks why the system did what it did and no one can answer. Governance, guardrails, and observability are drawn as a band beneath everything else in the diagram because they are cross cutting. They touch every layer rather than sitting at one point in the flow.

The band carries eight concerns. Guardrails and PII detection we have already met at the points where they act. Audit logs, human approval, tracing, metrics, cost monitoring, and model evaluation are the operational backbone. Tracing follows a single request across every hop so a slow or wrong result can be diagnosed. Metrics turn behavior into numbers you can alarm on. Cost monitoring makes token spend visible per feature and per tenant before it becomes a surprise. Model evaluation watches quality over time so a regression is caught by a graph rather than by a complaint.

Beside the band sits the audit log, drawn separately because it has a property the rest do not. It is immutable. Every request, every retrieved source, every action, and every response is written to a store that cannot be altered after the fact, which is what lets you reconstruct exactly what happened and prove it. A write once read many (WORM) store is not optional in a regulated setting. It is the evidence.

How this whole estate is governed over its lifecycle, from model selection through approval, deployment, and monitoring, is its own substantial topic, covered in the model lifecycle and governance reference architecture.

The request, traced end to end

Layers explained one at a time can still feel like parts in a box. The test of an architecture is what happens when a single real request moves through all of it at once. So take a concrete one. A relationship manager asks the system to summarize a client's last three support tickets and open a follow up case if any of them are still unresolved. That request is both a read and a conditional action, which is exactly the kind of thing a demo cannot safely do and a production system can.

It begins at the user experience layer, where the manager submits the request from the web application carrying a signed session token. At the gateway, that token is validated, the manager's role is checked against this particular client, the request is rate limited, and its shape is verified. Only then does the orchestrator take over, classifying the request as a read plus a conditional action, planning the two steps, and beginning to assemble context. It pulls the manager's user context and any prior workflow state from the memory layer, then asks the retrieval layer for the three tickets, which come back from the knowledge base re-ranked by relevance.

Before any of that retrieved text reaches the model, a guardrail screens it, because a support ticket is untrusted content and could carry an injected instruction. The model then does the one thing only it can do. It reads the three tickets, reasons that one is unresolved, and decides a follow up case is warranted. That decision becomes a tool selection. The agent gateway resolves the open case tool from the registry, and the validation checkpoint runs before anything executes, confirming with a Cedar policy that this manager may open a case for this client and running a safety check on the proposed action.

With authorization and safety satisfied, the tool executes, opening the case through an approved and audited interface. The response and action layer then confirms the summary is grounded in the actual tickets, redacts any personal data, and returns three things together, a written summary, a link to the new case, and a notification. Through every one of those hops, the observability band has been recording traces, metrics, and cost, and the audit log has been writing an immutable record of the request, the sources, the action, and the response. The model handled one of the twelve steps. The other eleven are what allow the answer to be trusted.

What this changes about how you build

The single most useful shift this map produces is in where you spend your attention. A great deal of energy goes into prompt wording and model choice, and both matter, but neither is where production systems succeed or fail. They succeed or fail at the boundaries, at the gateway that decides who may call, at the retrieval that decides what the model may read, at the validation that decides what it may do, and at the observability that decides whether anyone can ever explain it afterwards. The model is a single layer. The surrounding engineering is where production systems succeed or fail.

A second shift is that none of this is speculative anymore. Each layer in the diagram now maps to a managed service. Amazon Bedrock AgentCore reached general availability with its runtime, memory, gateway, identity, policy, and observability components, and Amazon Bedrock Guardrails ships the prompt attack detection, contextual grounding, PII filtering, and Automated Reasoning checks that the governance band needs. The pattern that was recently a hand built collection of glue code is becoming a set of composable, governed building blocks, which lowers the cost of doing it properly and raises the bar for doing it poorly.

Finally, the architecture is not finished when it works once. It has to keep working across regions and through failure, which is why the production posture here is multi region by default and why the resilience patterns sit in their own treatment, the resilient transaction processing reference architecture. How all of this is built, signed, and shipped, from rootless container builds through image signing to fully private runners, is covered in turn by the build and deployment pipeline and the self hosted runners on AWS. Taken together they answer the last question a production system always raises, which is not whether the model is clever but whether the whole machine around it can be trusted, paid for, and explained. That machine is the architecture. The model is one box in it.