Model lifecycle and governance on SageMaker

In a regulated setting, accuracy is rarely the main question. What matters is whether you can show how the model was built, what it was tested for, who approved it and on what evidence, what it is allowed to be used for, and how you would know if it started to behave differently. A pipeline that produces a good model but cannot answer those questions has not finished the job. This design makes the answers a by-product of how the work is done, rather than something assembled after the fact for an audit.

Requirements

The requirements the platform has to satisfy, the quality attributes it is judged on, the constraints it must respect, and the assumptions it rests on.

Functional requirements

Number	Requirement
1	Build and train models through a repeatable pipeline rather than by hand.
2	Evaluate every candidate for accuracy, for bias, and for explainability before it can advance.
3	Register every model as a versioned package with its documentation attached.
4	Require a named reviewer to approve a version before it can be deployed.
5	Deploy only approved versions, and record which version is in production.
6	Monitor every deployed model for drift and alert on deviation.
7	Keep one central and auditable record of every model and its approvals.

Quality attributes

Number	Attribute	Intent
1	Auditability	Reconstruct the full history of a model, who approved it and on what evidence, retained for at least one year.
2	Separation of duties	The people who build models cannot deploy them to production unreviewed.
3	Reproducibility	Rebuild a model and its evaluation from source.
4	Responsibility	Bias and explainability are assessed and recorded for every model.
5	Reliability	Serve inference across two regions and tolerate a regional failure.
6	Operability	Detect drift and act on it before it harms outcomes.

Constraints and assumptions

Number	Constraint
1	Data and models remain within Canada and the two approved regions.
2	The platform sits inside the landing zone and uses its accounts, keys, and logging.
3	A model must carry a documented intended use and a risk rating before production.
4	Managed SageMaker services are preferred over self managed training and serving.

Number	Assumption
1	Training data is governed and access controlled within the landing zone.
2	Reviewers and approvers are defined roles with the authority to sign off.
3	Models are served as endpoints rather than embedded inside applications.

Architecture decisions and rationale

The decisions that shaped the platform, each with the alternative that was set aside and the cost that was accepted.

1. The model as a governed artifact in a registry rather than a file

Decision. Treat every model as a versioned package in the SageMaker Model Registry, which is the single source of truth for what exists and what is approved.

Alternatives. Storing model files in object storage and tracking their status in a spreadsheet or a wiki.

Rationale. A registry gives every model a version, an approval status, attached documentation, and a lineage, which is what makes governance possible at all. This serves functional requirements 3 and 7 and the auditability attribute.

Consequences. Teams must register models rather than copy files, which is the intended discipline.

2. Three accounts for build, governance, and production rather than one

Decision. Separate the development account where models are built, the governance account that holds the registry and the evidence, and the production account that serves them.

Alternatives. Doing all of it in one account with permissions to separate the activities.

Rationale. An account boundary is the strongest separation of duties available, so a data scientist building a model has no path to deploy it to production, and the central record of approvals lives in an account that neither the builders nor the operators can quietly change. This serves the separation of duties attribute and functional requirement 5.

Consequences. Models, cards, and evidence are shared across accounts through Resource Access Manager, which is a small amount of additional setup.

3. Approval as a hard gate rather than a guideline

Decision. Make the deployment pipeline refuse to deploy any version whose approval status is not approved.

Alternatives. Asking teams to follow an approval process by convention.

Rationale. A gate enforced by the pipeline cannot be skipped under deadline pressure, where a convention always can. This serves functional requirement 4.

Consequences. An unapproved model cannot reach production even in an emergency, which is the intended behavior.

4. Model Cards attached to the version rather than separate documentation

Decision. Capture the intended use, the risk rating, the training details, and the evaluation results in a SageMaker Model Card linked to the exact model version.

Alternatives. Maintaining documentation in a separate system that can fall out of step with the model.

Rationale. Documentation attached to the version cannot drift away from the thing it describes, and it travels with the model through approval and deployment. This serves constraint 3 and the auditability attribute.

Consequences. A version is not complete until its card is filled in, which is enforced as part of registration.

5. Bias and explainability in the pipeline rather than as an afterthought

Decision. Run SageMaker Clarify as a step in the pipeline, so that every candidate is assessed for bias and for feature attribution before it can be registered as a passing version.

Alternatives. Assessing fairness only when someone asks, usually after deployment.

Rationale. Responsibility checks that run automatically on every model are consistent and create evidence, where manual checks are infrequent and create gaps. This serves the responsibility attribute and functional requirement 2.

Consequences. The pipeline takes longer because it always evaluates, which is accepted for the evidence it produces.

6. Continuous drift monitoring rather than periodic review

Decision. Run SageMaker Model Monitor on every endpoint, watching data quality, model quality, bias drift, and feature attribution drift, with alerts on deviation.

Alternatives. Reviewing model performance on a schedule by hand.

Rationale. A model degrades continuously as the world changes, so the monitoring has to be continuous to catch it before it harms outcomes. This serves functional requirement 6 and the operability attribute.

Consequences. Monitoring jobs and a baseline must be maintained for each model, which the Model Dashboard tracks and flags when missing.

7. The same approved model in two regions rather than one

Decision. Replicate the approved model package and its evidence to the second region and serve from a warm standby endpoint there.

Alternatives. Serving from a single region.

Rationale. A failover must not change which version is in production or weaken the audit trail, so the same approved package and the same evidence exist in both regions. This serves the reliability attribute.

Consequences. A standby endpoint carries a running cost, kept low by scaling it up only on failover.

The lifecycle as a pipeline

The work moves in one direction through a pipeline, and two of its steps are gates rather than tasks. A model is built and trained, evaluated for accuracy and fairness, registered as a version with its card, approved by a named reviewer, deployed only if approved, and then monitored. Registration and approval are the gates. A version that has not been approved cannot be deployed, and the documentation that justifies the approval is attached to that exact version for the life of the model.

The registry entry carries the approval status and points at the evidence, so that the gate is a simple and unambiguous check. A representative entry shows a version waiting for approval, with its intended use and risk rating recorded and its bias and explainability reports stored in the governance account.

{
  "ModelPackageGroupName": "payments-risk-scorer",
  "ModelPackageVersion": 7,
  "ModelApprovalStatus": "PendingManualApproval",
  "CustomerMetadataProperties": {
    "intended-use": "Score the risk of a single transaction in real time",
    "risk-rating":  "High",
    "owner":        "fraud-platform"
  },
  "ModelMetrics": {
    "ModelQuality":   { "areaUnderCurve": 0.94 },
    "Bias":           { "reportPath": "s3://lz-ml-governance-evidence/clarify/v7/bias.json" },
    "Explainability": { "reportPath": "s3://lz-ml-governance-evidence/clarify/v7/shap.json" }
  }
}

Separation across accounts

The lifecycle is spread across three accounts so that no single team holds the whole chain. Models are built in the development account, the registry and the model cards and the evidence live in the governance account, which is the source of truth, and only approved versions are deployed into the production account. The registry and the cards are shared from the governance account using Resource Access Manager, so the governance account can see and audit everything while remaining the only place where approval status is set.

Serving in two regions

The approved model serves from both regions. The model artifact and the governance evidence are replicated to the second region, an endpoint runs warm there, and Amazon Route 53 directs inference to the healthy region. Model Monitor runs in each region against the same baseline, so a failover changes neither the version in production nor the strength of the monitoring.

The result is a platform where the model that serves a real decision is always a known, approved, documented, and monitored version, in either region, and where the evidence for all of that is a by product of how the work is done rather than a burden added at the end.