Resilient transaction processing core

The difficult part of a transaction system is not the normal path but the failure cases: a caller that retries because it did not hear back, a step that succeeds while the next one fails, a queue that backs up under load, or a primary region that becomes unreachable during peak traffic. The design here treats every one of those as a normal event rather than an exception, because in a system that runs at scale they are normal. Two ideas carry most of the weight. Every operation is idempotent, so doing it twice is the same as doing it once, and every multi step transaction is a saga, so a failure partway through unwinds cleanly.

Requirements

The requirements the core has to satisfy, the quality attributes it is judged on, the constraints it must respect, and the assumptions it rests on. The quality attributes are deliberately the six pillars of the Well Architected Framework, because this is a workload where all six matter.

Functional requirements

Number	Requirement
1	Accept a high rate of transactions and never lose one once accepted.
2	Apply each transaction exactly once, even when a caller retries.
3	Coordinate multi step transactions so that a partial failure never leaves money half moved.
4	Keep a durable, ordered, and auditable record of every transaction.
5	Publish events so that downstream systems react without being coupled to the core.
6	Continue processing through the loss of an availability zone with no action.
7	Recover from the loss of a region within the recovery objective.

Quality attributes, the six pillars

Pillar	Intent
Operational excellence	Everything defined as code, with traceability of every transaction through the system.
Security	Authenticated entry, least privilege, encryption everywhere, inside the landing zone production segment.
Reliability	Tolerate a zone loss with no action, and a region loss within the recovery objective, with no double processing.
Performance efficiency	Absorb spikes through queues and scale the processors to demand.
Cost optimization	Pay for work done rather than for idle capacity, with a warm rather than a full standby.
Sustainability	Serverless and managed components scaled to actual load rather than provisioned for peak.

Constraints and assumptions

Number	Constraint
1	Data residency in Canada and the two approved regions.
2	The core sits inside the landing zone production segment and inherits its controls.
3	Every transaction must be traceable end to end and retained for audit.
4	Recovery point objective near zero, and recovery time objective in minutes.

Number	Assumption
1	Callers are authenticated services rather than anonymous clients.
2	Downstream consumers can react to events asynchronously.
3	The ledger is the system of record for anything of value.

Architecture decisions and rationale

The decisions that shaped the core, each with the alternative that was set aside and the cost that was accepted.

1. Event driven rather than a synchronous chain of calls

Decision. Accept a transaction quickly, place the work on a durable queue, and process it asynchronously, with components communicating through events.

Alternatives. A synchronous chain in which the caller waits while each downstream system is called in turn.

Rationale. A synchronous chain is only as available as its least available link and falls over under load, where a queue absorbs spikes, decouples the parts, and lets each scale and fail independently. This serves functional requirements 1 and 5 and the performance and reliability pillars.

Consequences. The system is eventually consistent for downstream effects, which is handled by emitting clear events and designing consumers to expect them.

2. Idempotency keys rather than hoping callers do not retry

Decision. Give every transaction an idempotency key recorded in DynamoDB, and make every operation safe to repeat under that key.

Alternatives. Assuming each request arrives exactly once.

Rationale. At scale, retries are inevitable, from clients, from networks, and from the platform itself, so the only safe assumption is that any request may arrive more than once, and the idempotency key makes a repeat harmless. This serves functional requirement 2 and the reliability pillar.

Consequences. Every operation must check and record the key, which is a small and consistent overhead.

3. The saga pattern rather than a distributed transaction

Decision. Coordinate multi step transactions with AWS Step Functions as an explicit saga, where each step has a compensating action.

Alternatives. A distributed transaction that tries to lock every system and commit them together.

Rationale. A lock held across independent services does not scale and creates new failure modes, where a saga makes the steps and their compensations explicit and lets a late failure unwind the earlier steps cleanly. This serves functional requirement 3.

Consequences. Each step needs a compensation written for it, which is the deliberate cost of doing this correctly.

4. An append only ledger as the system of record rather than mutable rows

Decision. Record every movement in an append only ledger in Amazon Aurora, and archive an immutable copy to object storage.

Alternatives. Updating balances in place and trusting the latest value.

Rationale. An append only record can always be replayed and audited, where a mutable balance loses the history that a regulated system has to be able to reconstruct. This serves functional requirement 4 and the operational excellence pillar.

Consequences. The ledger grows and is summarized for fast reads, which is a normal and well understood pattern.

5. Domain events for downstream rather than direct integration

Decision. Publish domain events to Amazon EventBridge for downstream systems to consume.

Alternatives. Calling each downstream system directly from the core.

Rationale. Direct calls couple the core to every consumer and make it fragile, where an event bus lets consumers come and go without the core knowing or caring. This serves functional requirement 5 and the operational excellence pillar.

Consequences. Consumers must handle events that may arrive more than once, which the idempotency discipline already assumes.

6. Multi region active and warm standby rather than a single region

Decision. Replicate idempotency keys through DynamoDB global tables and the ledger through Aurora Global Database, with a warm standby promoted on failover.

Alternatives. A single region, or a pair fully active in both regions.

Rationale. A single region cannot meet the recovery objective, and a fully active pair would have to resolve conflicting writes to the ledger, which is more risk than this objective needs, so the standby is warm and is promoted deliberately. This serves functional requirement 7 and the reliability and cost pillars.

Consequences. A failover is a rehearsed action that takes minutes, and because idempotency keys replicate, a request retried after a failover is still recognized and never applied twice.

The core

A transaction enters through an authenticated entry behind the web application firewall, is made idempotent immediately, and is placed on a durable queue. From there the processors pick it up, the saga coordinates the steps, the ledger records the movements, and an event is published for anyone downstream who needs to know. Each box in the picture can scale and fail on its own, which is the point of separating them.

The layers, end to end

It helps to walk the core one layer at a time, because each has a single job and a clear contract with the layers on either side of it. A transaction touches all of them in order on the way in, and the resilience of the whole comes from each layer being able to scale, retry, and fail on its own.

Layer	Responsibility	Built with
Edge and entry	Terminate the connection, filter hostile traffic, authenticate the caller, and apply a first rate limit before anything reaches the core.	Amazon CloudFront, AWS WAF, an authenticated API behind a private load balancer
Ingestion and idempotency	Validate the request, claim its idempotency key, and either accept it once or return the result of the accepted original. Nothing past this layer runs twice for the same key.	A thin ingestion function and a DynamoDB idempotency table with a conditional write
Durable buffer	Hold accepted work safely so that a spike or a slow downstream never drops a transaction, and so the processors scale independently of the arrival rate.	Amazon SQS, with a dead letter queue for anything that cannot be processed after retries
Processing and orchestration	Pull work from the buffer, run the saga that moves the money, and decide success, retry, or compensation for each step.	Amazon EKS with Fargate workers driving an AWS Step Functions saga
System of record	Append every movement to an ordered, immutable ledger and keep a summarized balance for fast reads. This layer is the truth.	Amazon Aurora for the ledger, with an immutable archive in Amazon S3 under Object Lock
Eventing	Publish a domain event for every committed transaction so that downstream systems react without being wired into the core.	Amazon EventBridge, with one bus and well defined event schemas
Observability	Trace any single transaction through every layer by its identifier, and alarm on what matters before a human notices.	AWS X-Ray traces, structured logs, and CloudWatch metrics and alarms

How the services talk to each other

The most important boundary in the core is the line between synchronous and asynchronous communication, and it is drawn deliberately. The caller talks to the core synchronously for exactly one thing, which is to have its transaction accepted and made idempotent. That call is fast and does almost no work, so it stays available under load. Everything after acceptance is asynchronous, because the moment work is on a durable queue it cannot be lost, and the parts that do the real work scale and fail without the caller ever waiting on them.

Inside the core, two communication styles do different jobs. A command, which is an instruction to do one specific thing such as debit this payer, travels on a queue to exactly one processor, so that it is handled once and can be retried safely if the processor dies mid step. An event, which is a statement that something has already happened such as this transaction settled, is published to the bus, where any number of consumers may read it, now or later, without the core knowing they exist. Commands are point to point and express an intended action, while events are broadcast and record something that has already happened.

Three rules keep that communication reliable. First, every message carries the transaction identifier and the idempotency key, so a redelivery, which a durable queue will eventually cause, is recognized and ignored rather than acted on twice. Second, failures retry with exponential backoff a bounded number of times, and anything that still fails lands in a dead letter queue for inspection rather than blocking the line behind it or being silently dropped. Third, where the order of operations matters, the queue is configured for strict ordering on the transaction identifier, so the steps of one transaction never overtake each other while unrelated transactions still run in parallel. The result is a system that absorbs a surge by letting the queue grow, drains it as the processors scale out, and never confuses backpressure with data loss.

The wider system, and the services the core depends on

The core does one thing well and delegates everything else, so in practice it sits at the centre of a small constellation of services rather than standing alone. Before a transaction posts, it consults a few decision services and treats their answers as part of the contract. After a transaction commits, it informs a wider set that react on their own time. Naming these and being explicit about which calls block matters, because every synchronous dependency is part of the acceptance latency budget and part of the failure surface.

Service	Responsibility	Interaction
Risk Decision	Scores the transaction for fraud and returns allow, deny, or step up.	Synchronous, hard deadline, fails closed
Screening	Checks the payer and payee against sanctions and watchlists.	Synchronous, hard deadline, fails closed
Limits	Enforces per account and per window value and velocity limits.	Synchronous, backed by an in memory store
Account Directory	Resolves account status, ownership, and eligibility.	Synchronous, cached
Notification	Tells the customer the outcome.	Asynchronous, on a domain event
Reconciliation and Settlement	Nets and settles to the scheme and reconciles the ledger against the scheme statement.	Asynchronous
Analytics sink	Lands every event for reporting and modeling.	Asynchronous
Scheme Connector	Carries the outbound instruction to the external network and ingests its responses.	Asynchronous, isolated

The four synchronous services share the acceptance budget, so the core calls them in parallel rather than in sequence, gives each its own short timeout, and treats a missed answer exactly as it treats a deny. In a payments system a risk service that does not answer in time is not a reason to take the risk, so the default on a timeout is to refuse the transaction cleanly with a reason the caller can act on. These services are usually owned by other teams and run in their own accounts inside the landing zone, reached over private connectivity, so the core depends on their published contracts and never on their internals.

The saga and idempotency

The saga is an explicit state machine. Funds are reserved, the payer is debited, the payee is credited, and the result is confirmed and emitted, and every step has a compensating action so that a failure late in the sequence reverses the earlier steps rather than leaving the transaction half done. Because each step is idempotent and keyed on the transaction identifier, a retry repeats the intent without repeating the effect.

Expressed as a state machine, each step catches its own failure and routes to the compensation for the steps that already succeeded.

{
  "Comment": "Transaction saga with compensations",
  "StartAt": "ReserveFunds",
  "States": {
    "ReserveFunds": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "reserve-funds", "Payload.$": "$" },
      "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "Fail" } ],
      "Next": "DebitPayer"
    },
    "DebitPayer": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "debit-payer", "Payload.$": "$" },
      "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "ReleaseHold" } ],
      "Next": "CreditPayee"
    },
    "CreditPayee": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "credit-payee", "Payload.$": "$" },
      "Catch": [ { "ErrorEquals": ["States.ALL"], "Next": "ReverseDebit" } ],
      "Next": "Confirm"
    },
    "Confirm":      { "Type": "Task", "Resource": "arn:aws:states:::events:putEvents", "End": true },
    "ReverseDebit": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Next": "ReleaseHold" },
    "ReleaseHold":  { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke", "Next": "Fail" },
    "Fail":         { "Type": "Fail" }
  }
}

Idempotency is enforced at the only place it can be, which is the first write. Before any work begins, the ingestion layer attempts to claim the transaction with a conditional write keyed on the idempotency key. If the claim succeeds, this is the first time the request has been seen and processing proceeds. If the claim fails because the key already exists, the request is a retry, and the core returns the stored result of the original rather than doing the work again. The whole guarantee rests on this one atomic operation.

# claim the transaction exactly once; a retry hits the condition and is rejected
table.put_item(
    Item={"pk": idempotency_key, "status": "ACCEPTED", "result": None,
          "ttl": now + RETENTION_SECONDS},
    ConditionExpression="attribute_not_exists(pk)",
)
# ConditionalCheckFailedException  means already seen:
#   read the stored record and return its result, do not reprocess

Across two regions

The core runs in two regions. Idempotency keys replicate through DynamoDB global tables, the ledger replicates through Aurora Global Database with a recovery point objective near zero, and Amazon Route 53 with the Application Recovery Controller directs traffic and supports a controlled failover. The combination of replicated keys and a replicated ledger is what makes the failover safe. A request that was in flight when the primary region failed can be retried against the standby and will be recognized as already seen, so it is never applied a second time.

How the data replicates, and why nothing is lost

Zero data loss is a property of the data layer, so it is worth being precise about how each piece of state crosses the region boundary and what guarantee that gives. There are three kinds of state, and each replicates in the way that suits it.

The idempotency keys live in a DynamoDB global table, which replicates every write to the second region within a second or two, in both directions. This is the piece that makes a failover safe rather than dangerous. A transaction accepted in the primary region just before it failed has already had its key replicated, so when the caller retries against the standby, the conditional claim fails exactly as it would have in the primary, and the transaction is recognized as already seen rather than applied a second time.

The ledger, which is the system of record, lives in an Amazon Aurora Global Database. Aurora replicates at the storage layer, typically with under a second of lag and a recovery point objective measured in single digit seconds, and a healthy secondary can be promoted to a full read and write primary in about a minute. Writes go to one region at a time, which avoids the risk of two regions disagreeing about a balance, and the standby is kept warm and current rather than cold. For the strongest durability the immutable archive of the ledger is written to Amazon S3 under Object Lock and replicated across regions, so even the catastrophic loss of a database leaves an independent, tamper proof copy of every movement.

Traffic is steered by Amazon Route 53 with the Application Recovery Controller, whose routing controls turn a failover into a deliberate, rehearsed switch rather than a guess made under pressure. A regional failover then follows a short, practiced runbook.

1. Confirm the primary is genuinely unhealthy and stop new work from entering it.
2. Promote the Aurora secondary in the standby region to primary.
3. Flip the Route 53 routing control so the entry points resolve to the standby.
4. Let the standby processors drain the replicated queue. The idempotency keys are already present, so any in flight retries settle exactly once.
5. Confirm the ledger and the idempotency table agree, then bring the original region back up as the new standby.

Because the keys and the ledger both crossed the boundary before any failure, the two outcomes that matter most are both closed off. A transaction in flight is never lost, because it was either not yet accepted, in which case the caller retries cleanly, or already accepted and replicated, in which case the standby recognizes it. And a transaction is never applied twice, because the replicated idempotency key rejects the duplicate on whichever region serves the retry. That is what zero data loss means here in practice, stated as two failure modes that cannot occur rather than as a headline figure.

The deployment architecture

Everything above describes behavior. This is where it physically lives. Each region is a single Virtual Private Cloud spread across three availability zones and divided into four subnet tiers, attached to the rest of the organization through AWS Cloud WAN, and able to reach the external payment scheme only from an isolated connector. The second region is the same definition with the region as a parameter, not a hand built copy.

The subnet tiers separate concerns so that a fault or a compromise in one does not become a fault in another.

Tier	What runs there	Reachability
Ingress	The Ingress Gateway and an internal load balancer.	Reachable only from the organization network over AWS Cloud WAN, never from the public internet.
Application	The core domain services on Amazon EKS with Fargate, behind a mesh that mutually authenticates every service to service call.	East to west traffic only, inside the mesh.
Data	The ledger, the idempotency store, the balance cache, the signing cluster, and the interface endpoints for the regional services.	No route to the public internet at all.
Egress and inspection	The isolated Scheme Connector and centralized outbound inspection.	The only tier permitted to reach outside the Virtual Private Cloud, and only to the scheme.

Each named component has a clear home and a clear runtime.

Component	Tier	Runs on
Ingress Gateway	Ingress	Amazon EKS with Fargate, behind an internal load balancer
Saga Orchestrator	Application	AWS Step Functions, driven by Amazon EKS workers
Posting Engine	Application	Amazon EKS with Fargate
Event Publisher	Application	Amazon EKS with Fargate, writing to Amazon EventBridge
Work Queue	Application	Amazon SQS, reached through an interface endpoint
Idempotency Store	Data	Amazon DynamoDB global table
Ledger	Data	Amazon Aurora Global Database
Balance Cache	Data	Amazon ElastiCache, fronting the Aurora read replicas
Signing	Data	AWS CloudHSM
Scheme Connector	Egress and inspection	Amazon EKS with Fargate, AWS PrivateLink to the scheme with a Direct Connect fallback

The tiers say where things sit. The next view says how they talk, component by component, and what travels on each link.

One transaction, end to end

1. An authenticated caller reaches the Ingress Gateway over AWS Cloud WAN. The gateway validates the request and asks the Idempotency Store to claim it with a conditional write. A duplicate stops here and returns the stored result of the original.
2. The gateway calls the Risk Decision, Screening, Limits, and Account Directory services in parallel, each with a short deadline. Any deny, and any timeout, ends the transaction with a clear reason. This is the only synchronous fan out, and it is what the acceptance latency is spent on.
3. On a clean decision the gateway places the work on the Work Queue and returns accepted to the caller. The acceptance path is complete, well inside the objective of a 99th percentile under 250 milliseconds.
4. A worker in the application tier starts the Saga Orchestrator, which reserves funds, debits the payer, and credits the payee through the Posting Engine, each step idempotent and each with a compensation.
5. The Posting Engine appends every movement to the Ledger and updates the Balance Cache for fast reads. The ledger is the system of record.
6. The Event Publisher emits a settled event. The Notification, Reconciliation, and Analytics services react on their own time, and the Scheme Connector carries the instruction outward over the isolated path, signing it inside the hardware security module.

Scale, delivery, and key custody

The worker fleet scales on the depth of the Work Queue rather than on a fixed size, which lets the core hold a sustained rate of about 2,000 transactions per second and absorb peaks near 8,000 by letting the queue grow and draining it as workers scale out. Capacity follows demand rather than being provisioned for a peak that rarely arrives.

The whole core is defined in AWS Cloud Development Kit stacks and delivered through a pipeline that runs the tests, deploys to the standby region first, runs a synthetic transaction from end to end, and only then promotes the primary, with a blue green cutover so that a bad release is reversed in seconds rather than recovered from over hours. The core sits in the production segment of the landing zone described elsewhere in these architectures and inherits its account isolation, its centralized inspection, and its guardrails.

Operations that move money are signed inside AWS CloudHSM, so the keys that authorize a movement never leave the hardware security module. Every store is encrypted at rest, every call between services is mutually authenticated inside the mesh, and the components that touch payment data sit inside a clearly drawn compliance boundary, which keeps the scope of the relevant standard small and auditable. Every component runs with the least privilege it needs and nothing more.

The full build sheet

The view below is deliberately literal. It names every network boundary, the address ranges, the security groups, the subnets, the compute namespaces, the data stores, and the managed services, so that someone could take it and stand the environment up. One region is shown. The second region is identical for production, with the address range moved from 10.20.0.0/16 to 10.21.0.0/16. The decision services named earlier sit in their own accounts and are reached over AWS Cloud WAN, so they are not redrawn here.

A few components on the sheet did not need naming in the narrative above. Amazon ECR holds the signed container images for every service. Amazon MSK carries the high volume event stream that the analytics and reconciliation consumers read, while Amazon EventBridge routes the lower volume domain events. Amazon ECS runs the scheduled reconciliation and netting job, which suits a batch runtime better than the always on services on Amazon EKS.

The six pillars, made concrete

Because this workload is judged on all six pillars of the Well Architected Framework, it is worth stating exactly how each one is met rather than claiming it in passing.

• Operational excellence. The whole core is defined in infrastructure as code and delivered through the standby first, blue green pipeline, so a change is rehearsed in the second region before it touches the first. A single transaction identifier threads through the Ingress Gateway, the Work Queue, the saga, the Ledger, and the events, so any one transaction can be traced from end to end, and the same identifier keys the metrics and alarms that page a human before a customer notices.
• Security. Entry is authenticated and rate limited at the Ingress Gateway, every call between services inside the mesh is mutually authenticated, and the data subnets have no route to the public internet. The keys that authorize a movement never leave AWS CloudHSM, every store is encrypted at rest, and the core inherits the account isolation and centralized inspection of the landing zone production segment. The only way out is the isolated Scheme Connector over AWS PrivateLink.
• Reliability. Every component runs across three availability zones, so the loss of one is handled with no action. Idempotency makes every retry safe, the saga prevents a half moved transaction, and the rehearsed regional failover, backed by replicated idempotency keys and a replicated ledger, recovers from the loss of a whole region within minutes and without double processing.
• Performance efficiency. The synchronous path does almost nothing, so acceptance stays inside the 99th percentile target of 250 milliseconds even under load. The Work Queue absorbs a surge while the worker fleet scales on its depth, and the Balance Cache keeps hot reads off the ledger, which together hold a sustained rate of about 2,000 transactions per second and absorb peaks near 8,000.
• Cost optimization. Compute runs on Amazon EKS with Fargate and AWS Lambda and follows demand rather than a provisioned peak, so the bill tracks real work. The second region is kept warm rather than fully scaled, which meets the recovery objective at a fraction of the cost of a pair that is fully active in both regions.
• Sustainability. The same elasticity is the sustainability story. Capacity scaled to actual load draws less power than capacity provisioned for a peak that rarely arrives, and the managed and serverless components share their underlying hardware across many tenants rather than sitting idle.

This core is the kind of backbone that the other reference architectures publish their events to and rely on for anything that moves value.