akanjilal.dev
Back to reference architectures
Reference architectureGitHub Actions and Amazon Web Servicesca-central-1 and ca-west-1

Build and deployment pipeline on GitHub Actions

The journey from a line of code on a laptop to a running service in production crosses a lot of ground, and in a regulated environment every step of it has to be checked, recorded, and reversible. This reference architecture describes that journey as a single pipeline on GitHub Actions. It runs from a local commit through static and dynamic analysis, dependency and image scanning, signing and provenance, behind hard quality gates, to a deployment that reaches Amazon Elastic Kubernetes Service through ArgoCD and Amazon Elastic Container Service through a blue and green release, in two Canadian regions, with no long lived credentials held anywhere.

A pipeline is where intentions about security and quality are either enforced on every change or left to chance. It is easy to agree that code should be scanned, that dependencies should be checked, that images should be signed, and that nothing should reach production without review. The pipeline determines whether those checks run on every change rather than only when someone remembers to run them. The design here makes them happen automatically, makes the checks block rather than merely warn, and produces, as a side effect of how the work is done, a complete record of what was built, what it was scanned for, who approved it, and where it went. The tools named are examples. The shape of the pipeline matters more than any one vendor, and any of the named tools can be swapped for an equivalent without changing the design.

Requirements

The requirements the pipeline has to satisfy, the quality attributes it is judged on, the constraints it must respect, and the assumptions it rests on.

Functional requirements

NumberRequirement
1Run security and quality checks automatically on every change before it can merge.
2Produce one immutable, signed artifact and promote it unchanged through every environment.
3Block any change that fails a gate from progressing.
4Authenticate to Amazon Web Services without storing long lived credentials.
5Deploy to Amazon Elastic Kubernetes Service and to Amazon Elastic Container Service.
6Promote the same artifact through development, test, staging, and production with approvals.
7Deploy to two regions and roll back quickly on failure.
8Keep an auditable record of what was built, scanned, signed, approved, and deployed.

Quality attributes

NumberAttributeIntent
1SecurityEvery change scanned, only signed and compliant artifacts deployed, and no standing credentials.
2AuditabilityReconstruct the full provenance of anything in production, retained for audit.
3ReliabilitySafe rollouts and fast rollback, across two regions.
4SpeedFast feedback to developers, with independent checks running in parallel.
5RepeatabilityThe same pipeline and the same artifact every time.
6Least privilegeEach stage and each environment holds only the access it needs.

Constraints and assumptions

NumberConstraint
1Deployments target the two approved regions inside the landing zone.
2Runners are GitHub hosted, so the cluster and private resources are not reachable from a runner.
3Only signed images that pass every gate may run in production.
4Source, artifacts, and deployment evidence are retained for audit.
NumberAssumption
1Application code and infrastructure as code live in GitHub repositories.
2The landing zone provides the registries, the clusters, the roles, and the networking.
3Workloads are packaged as container images.

Architecture decisions and rationale

The decisions that shaped the pipeline, each with the alternative that was set aside and the cost that was accepted.

1. Security checks inside the pipeline rather than a review at the end

Decision. Run static analysis, dependency analysis, secret scanning, infrastructure as code scanning, image scanning, and dynamic analysis as stages of the pipeline on every change.

Alternatives. A separate security review performed late, by a different team, after the code is written.

Rationale. A check that runs on every change is consistent and cheap to fix, where a late review finds problems when they are expensive to undo and creates a bottleneck. This serves functional requirement 1 and the security attribute.

Consequences. The pipeline takes longer because it always checks, which is reduced by running the independent checks in parallel.

2. Gates that block rather than warnings that are ignored

Decision. Make a failed check fail the build, so a change that does not meet the bar cannot merge or deploy.

Alternatives. Reporting findings as advisory notes that a developer may choose to act on.

Rationale. A warning that does not block is a warning that is eventually ignored under deadline pressure, where a gate that blocks turns the policy into something that actually holds. This serves functional requirement 3.

Consequences. A noisy or wrong check stops work, so the thresholds are tuned and exceptions are explicit and time limited rather than silent.

3. OpenID Connect federation rather than long lived access keys

Decision. Authenticate from GitHub to Amazon Web Services with OpenID Connect, where the workflow exchanges a short lived token for temporary credentials by assuming a role.

Alternatives. Storing an access key and a secret key in the repository as secrets.

Rationale. A stored key is a standing credential that nobody rotates and that leaks badly, where a federated token is short lived, scoped by the role's trust policy, and never stored. This serves functional requirement 4 and the least privilege attribute.

Consequences. Each environment needs a role with a carefully written trust policy, which is the intended point of control.

4. One immutable, signed artifact promoted by digest rather than rebuilt per environment

Decision. Build the image once, sign it, and promote that exact image by its content digest through every environment.

Alternatives. Rebuilding the image for each environment, or promoting by a mutable tag such as latest.

Rationale. The thing tested in staging must be the thing that runs in production, bit for bit, which is only true if the same digest is promoted, where a rebuild or a moving tag can quietly change what runs. This serves functional requirement 2 and the repeatability attribute.

Consequences. Promotion moves a reference to an artifact rather than running a build, which is simpler and faster as well as safer.

5. Amazon Elastic Container Registry as the single artifact source

Decision. Push every image to Amazon Elastic Container Registry with tag immutability and enhanced scanning, and pull only from there.

Alternatives. Pulling images from several registries, including public ones, at deployment time.

Rationale. A single private registry with immutable tags and continuous scanning by Amazon Inspector is the one place to know exactly what an image is and whether it has developed a known vulnerability since it was built. This serves functional requirement 8 and the security attribute.

Consequences. Base images must be mirrored into the registry rather than pulled from the internet, which is the intended control.

6. Pull based GitOps with ArgoCD for Kubernetes rather than pushing into the cluster

Decision. For Amazon Elastic Kubernetes Service, have the pipeline commit the desired state to a Git repository, and let ArgoCD running inside each cluster pull and apply it.

Alternatives. Having the pipeline reach into the cluster and apply changes directly.

Rationale. Because runners are GitHub hosted, a push model would require opening a path from the runner into the private cluster, where a pull model keeps the cluster unreachable and limits the runner to pushing an image and committing a file, which is the smallest possible blast radius. This serves functional requirement 5 within constraint 2.

Consequences. The cluster state lives in Git, which is a benefit, since the desired state is now reviewable, auditable, and the same in both regions.

7. Blue and green for Elastic Container Service rather than in place replacement

Decision. For Amazon Elastic Container Service, deploy with AWS CodeDeploy in a blue and green pattern, shifting traffic only once the new version is healthy.

Alternatives. Replacing the running tasks in place with the new version.

Rationale. Keeping the old version serving until the new one is proven, then shifting traffic, makes a bad release a quick and automatic rollback rather than an outage. This serves functional requirement 7 and the reliability attribute.

Consequences. Both versions run briefly during a release, a small and temporary use of extra capacity.

8. The one artifact in two regions, admitted only if signed

Decision. Deploy the same artifact to both regions, and enforce at the cluster that only images carrying a valid signature may run, using an admission policy.

Alternatives. Deploying to one region, or trusting the pipeline alone to have signed the image.

Rationale. A second region meets the recovery requirement, and an admission check inside the cluster is a second, independent enforcement of the signing rule, so an image that somehow arrived unsigned still cannot run. This serves functional requirement 7 and the security attribute.

Consequences. The signing identity and the policy must be maintained in both regions, which is part of the cluster baseline.

The engineering standards

A pipeline only enforces what the team has agreed to. The standards below are the agreed engineering rules, and the pipeline enforces every one of them on every change rather than leaving them to memory. The right hand column is where each standard stops being a document and becomes something that blocks.

AreaThe standardHow the pipeline holds it
Source controlTrunk based development, short lived branches, a protected main branch, and signed commits.Branch protection rules, a linear history, and a signed commit check on every pull request.
Code reviewAt least one approving review, no self approval, and every comment resolved before merge.Required reviews and a code owners file, enforced as merge conditions.
Infrastructure as codeAll infrastructure defined in code, with no change made by hand in the console.The pipeline runs the plan on a pull request and applies only from main, with drift detection.
TestingUnit and integration tests above a coverage threshold, with contract tests on every interface.Tests are a blocking gate, and coverage below the threshold fails the build.
Static analysisLinting, formatting, type checks, and static application security testing on every change.Each runs as a parallel gate, and a new finding fails the build rather than warning.
Dependencies and supply chainReviewed and pinned dependencies, a software bill of materials, and signed images with build provenance.Software composition analysis is a gate, and the build attaches the bill of materials and the provenance to the signed image.
SecretsNo secret in source, and only short lived federated credentials at deploy time.Secret scanning with push protection, and OpenID Connect federation in place of any stored key.
Container imagesA minimal base image, run as a non root user, with no high or critical vulnerability.Image scanning by Amazon Inspector is a gate, and a cluster admission policy refuses anything unsigned or unscanned.
Artifacts and versioningSemantic versioning, and one immutable artifact promoted unchanged by its digest.The image is built once and promoted by digest, with tag immutability in Amazon Elastic Container Registry.
ObservabilityStructured logs, traces, metrics, and health endpoints ship with every service.A readiness check confirms the required signals before a deployment is promoted.
Release and rollbackProgressive delivery, automated rollback on a failed health check, and a recorded change.Blue and green or GitOps with health gates, an automatic rollback, and a change record written for audit.
OwnershipEvery service has a named owner and a runbook.A code owners file and a required service manifest, checked on every change.

The regulated baseline

A standard is only as strong as the number attached to it. The settings below are the concrete values this pipeline is configured to in a highly regulated environment such as a payments platform under the Payment Card Industry Data Security Standard. They are deliberately strict, and an exception to any of them is explicit, approved, and time limited rather than silent.

ControlRequired value in a regulated environment
Protected main branchNo direct push, no force push, a linear history, and every required check green before a merge.
Signed commitsRequired and verified on every commit.
Pull request review2 approvals, at least one from a code owner. Self approval refused. Approvals dismissed on a new commit.
Test coverage≥ 80% overall and ≥ 90% on money handling code, and never below the main branch.
Static analysis0 new high or critical findings. Type checking in strict mode.
Dependency scanning0 known high or critical vulnerabilities in shipped dependencies, each pinned by hash.
Software bill of materialsGenerated in the CycloneDX format on every build and stored with the artifact.
Build provenanceAttested to supply chain level 3 and verified before a deploy.
Secret scanningPush protection on. Any detected secret blocks the push.
Cloud credentialsNo long lived keys. Federated through OpenID Connect, session lifetime ≤ 1 hour, a separate role per environment.
Image signingKeyless signature required, verified by a cluster admission policy in both regions.
Image vulnerabilities0 high or critical at deploy time. Base image rebuilt at least every 7 days. Runs as a non root user with a read only root filesystem.
RegistryTag immutability on, continuous scanning by Amazon Inspector.
Artifact promotionThe same image digest from development through to production, never rebuilt.
Environments and approvalDevelopment, test, staging, and production. A manual approval by a second authorized person gates the move to production.
Regions2, ca-central-1 and ca-west-1.
Release strategyBlue and green, or a canary at 10%, 50%, then 100%, with automatic rollback on a failed health check or an error budget breach.
Rollback time≤ 5 minutes, with the previous version kept ready.
EncryptionTransport security version 1.2 or higher in transit, customer managed keys at rest.
Audit retentionPipeline logs, scan reports, approvals, the bill of materials, and provenance kept for 7 years on storage under object lock.
Change recordEvery production deploy linked to a change ticket and traceable from commit to running service.
AccessLeast privilege on every role, with multi factor authentication on all human access.

The developer journey

The journey begins before the pipeline does. On the developer's own machine, fast checks run on every commit, formatting and linting, the unit tests, a quick secret scan, and a signed commit, so that the most common mistakes never even reach the repository. The change then arrives as a pull request, which cannot be merged directly and which triggers the pipeline. From there the work runs on a GitHub hosted runner through a sequence of stages, each of which can fail the build, ending in a signed image pushed to the registry and a deployment to either platform.

The full pipeline from local pre-commit checks through pull request, ten continuous integration stages, a quality gate, and two deployment paths
The end to end pipeline. Local checks, then ten stages on the runner, then a hard quality gate, then deployment to Kubernetes through GitOps or to the container service through blue and green.

Expressed as a workflow, the stages are ordinary steps, and the important detail is that each scanning step is configured to fail the job when it finds something above the agreed severity rather than to pass and report.

permissions:
  id-token: write          # request the OpenID Connect token
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Static analysis
        run: sonar-scanner                       # fails on a failed quality gate
      - name: Software composition analysis
        run: snyk test --severity-threshold=high
      - name: Secret scan
        run: gitleaks detect --no-banner
      - name: Infrastructure as code scan
        run: checkov --directory infra
      - name: Build the image
        run: docker buildx build --tag "$IMAGE_DIGEST_REF" --push .
      - name: Image vulnerability scan
        run: trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMAGE_DIGEST_REF"
      - name: Sign and attest
        run: cosign sign --yes "$IMAGE_DIGEST_REF"

The quality gates

The gates are central to the design, so it is worth setting them out plainly. Each control inspects something specific, and each has a condition that decides whether a change may proceed. The bar is deliberately strict for anything that reaches production, and the conditions are part of the pipeline configuration rather than a document that describes good intentions.

A table of nine controls, what each inspects, example tools, and the gate condition that blocks a change
The controls and their gates. A finding that does not meet the bar fails the build rather than raising a note that is later ignored.

Keyless trust and promotion

The pipeline never holds a long lived key. When a job needs to act in Amazon Web Services, it asks GitHub for a short lived OpenID Connect token whose claims describe the repository, the branch, and the environment, and it presents that token to assume a role. The role's trust policy pins the claim, so only that repository, on that environment, can assume it, and the credentials returned are temporary and scoped to what the environment needs.

The keyless trust flow from GitHub to AWS, and the promotion of one immutable artifact through four environments with gates
Keyless trust and promotion. No stored keys, and one immutable artifact promoted by digest through environments, each with its own protection rules and its own scoped role.

The trust is expressed in the role's policy. The condition on the subject claim is what limits which workflow, on which environment, may assume the role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:akanjilal-dev/payments-app:environment:production"
        }
      }
    }
  ]
}

The same immutable image is promoted by its digest through development, test, staging, and production. Each environment is a GitHub environment with its own protection rules, so production requires two approvers and a wait timer, and each environment has its own role, so a deployment to one environment cannot touch another.

Deploying to Kubernetes with GitOps

For Amazon Elastic Kubernetes Service the deployment is pull based. The pipeline pushes the signed image to the registry and commits the new image digest into a configuration repository that holds the desired state of the clusters. ArgoCD, running inside each cluster, watches that repository, notices the change, pulls it, and reconciles the cluster to match. The runner is never given a way into the cluster, which is exactly what makes a GitHub hosted runner acceptable for a private, regulated cluster.

GitOps with ArgoCD pulling desired state from Git into two clusters, with Kyverno admission and Argo Rollouts canary
Pull based GitOps. The runner only pushes an image and commits a file. ArgoCD pulls the desired state from inside each cluster, and the same state converges in both regions.

Inside the cluster, a policy refuses to schedule any image that does not carry a valid signature from the pipeline's identity, which is the second, independent enforcement of the signing rule. Argo Rollouts then moves the new version in as a canary and rolls back automatically if it is unhealthy.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds: [ Pod ]
      verifyImages:
        - imageReferences:
            - "111122223333.dkr.ecr.ca-central-1.amazonaws.com/*"
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/akanjilal-work/payments-app"

Deploying to the container service with blue and green

For Amazon Elastic Container Service the deployment is push based but safe. A deploy job assumes its scoped role, registers a new task definition that points at the same image digest, and hands the release to AWS CodeDeploy, which stands up the new version alongside the old one behind the load balancer. The current version keeps serving traffic until the new one passes its health checks, at which point traffic shifts. If an alarm fires during the shift, the release rolls back automatically and the old version simply keeps serving.

Blue and green deployment on the container service in two regions, with CodeDeploy shifting traffic and rolling back on an alarm
Blue and green on the container service. The old version stays in place until the new one is proven, and a failed shift rolls back on its own.

Both paths deploy the same signed artifact to both regions, so a release is consistent across the estate, and a regional problem is handled by the landing zone's regional failover rather than by the pipeline. The result is a pipeline that turns a local commit into a production change through a sequence of checks that cannot be skipped, that holds no standing credentials, that deploys in a way that is quick to undo, and that leaves behind a complete and honest record of how the change got there.

References

  1. GitHub, GitHub Actions documentation.https://docs.github.com/en/actions
  2. GitHub, configuring OpenID Connect in Amazon Web Services.https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services
  3. GitHub, managing environments for deployment.https://docs.github.com/en/actions/managing-workflow-runs-and-deployments/managing-deployments/managing-environments-for-deployment
  4. Amazon Web Services, the configure AWS credentials action for GitHub Actions.https://github.com/aws-actions/configure-aws-credentials
  5. Amazon Web Services, Amazon Elastic Container Registry.https://docs.aws.amazon.com/AmazonECR/latest/userguide/what-is-ecr.html
  6. Amazon Web Services, Amazon Elastic Container Registry image scanning.https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-scanning.html
  7. Argo Project, Argo CD.https://argo-cd.readthedocs.io/en/stable/
  8. Argo Project, Argo Rollouts.https://argoproj.github.io/argo-rollouts/
  9. Amazon Web Services, blue and green deployment for Amazon Elastic Container Service.https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-bluegreen.html
  10. Kyverno, verifying container image signatures.https://kyverno.io/docs/writing-policies/verify-images/
  11. Sigstore, Cosign documentation.https://docs.sigstore.dev/
  12. Supply chain Levels for Software Artifacts.https://slsa.dev/
  13. Open Worldwide Application Security Project, Top 10 Continuous Integration and Continuous Delivery Security Risks.https://owasp.org/www-project-top-10-ci-cd-security-risks/