AWS Production Architecture
I designed this containerized, multi-AZ AWS environment in Singapore around five goals: keep compute private, put one encrypted entry point in front, move slow work off the request path, give the database its own boundary, and make operations visible.
This page walks through the decisions behind each layer, how a request moves through it, what happens when a piece fails, and what each choice costs.
Multi-AZ ECS design with private compute and an isolated data layer.
- Role
- Architecture design
- Ownership
- Design owner
- Core stack
- ECS on EC2 · ALB · RDS Proxy · SQS · Route 53
A containerized workload needed to survive the loss of a host or an AZ without exposing application servers to the internet.
Architecture diagram
The full topology, layer by layer.
Regional managed services sit outside the VPC; subnet-bound capacity sits inside it. Filter by layer or select a service to see why it is there.
Drag service cards to arrange them. Pan or zoom the canvas, select a service to see its role, or select an arrow to isolate that flow.
What the infrastructure needed to solve
Private runtime
Protect application workloads from direct public ingress while retaining controlled egress.
Failure domains
Spread compute and the data layer across two Availability Zones.
Reproducible releases
Run versioned container artifacts instead of building source on production hosts.
Independent work
Separate synchronous HTTP handling from background processing.
Persistent data
Isolate relational state from replaceable container and EC2 lifecycles.
Operability
Provide monitoring, audit, administration, backup and recovery capabilities.
Seven boundaries that shape runtime behavior
Centralized public ingress
- Problem → decision
- Public callers need one stable, encrypted entry point. Route 53 resolves the hostname; ACM supplies the certificate; a public ALB receives HTTPS and selects healthy ECS targets.
- Runtime behavior
- DNS completes before application traffic. The browser connects to the ALB—not through Route 53.
- Trade-off
- Listener, certificate, subnet mapping, health check and target configuration become critical shared dependencies.
Private application compute
- Problem → decision
- Containers should not accept arbitrary inbound internet traffic. Place ECS EC2 capacity in private application subnets and make the ALB the intended ingress boundary.
- Runtime behavior
- ALB → application target is the only inbound path; NAT gives tasks outbound-only internet access.
- Trade-off
- Private placement still requires least-privilege security groups, IAM, patching and egress control.
Multi-AZ compute
- Problem → decision
- One Availability Zone must not be the only compute failure domain. Run ECS capacity in AZ-1a and AZ-1b behind one ALB.
- Runtime behavior
- If one AZ fails, the ALB keeps routing to healthy targets in the other AZ.
- Trade-off
- Multi-AZ topology increases resilience and cost, but does not promise zero downtime.
ECS on EC2
- Problem → decision
- Container orchestration is needed with control over the underlying hosts. Use an ECS cluster on EC2 capacity, with three container instances per AZ running the same workloads.
- Runtime behavior
- ECS places tasks onto available container instances; task demand and host capacity must be reconciled separately.
- Trade-off
- More host control and potentially efficient steady load, in exchange for AMI lifecycle, patching and capacity scaling. Fargate would remove host operations but change cost/control trade-offs.
Immutable container artifacts
- Problem → decision
- Production releases should be traceable and repeatable. Build a Docker image before runtime, store it in a registry, and deploy the selected image to ECS.
- Runtime behavior
- CI builds an image, pushes it to ECR, and a new task definition revision rolls it out.
- Trade-off
- Tag mutability, retention, vulnerability scanning and rollback conventions must be governed.
Asynchronous workloads
- Problem → decision
- Background work should not extend or destabilize synchronous HTTP requests. Put SQS between the API and background workers.
- Runtime behavior
- Messages buffer bursts; workers retry failed jobs and scale on queue depth.
- Trade-off
- At-least-once delivery concerns, idempotency, visibility timeout, DLQ and poison-message handling must be designed.
Database isolation
- Problem → decision
- Persistent state and database connections need a boundary independent of compute churn. Place RDS in dedicated database subnets with RDS Proxy in front of the primary.
- Runtime behavior
- ECS tasks connect through RDS Proxy, which pools connections to the primary; a second instance runs in AZ-1b.
- Trade-off
- RDS Proxy smooths connection spikes during deploys and scale-outs, at the cost of an extra hop and another managed service to pay for.
DNS is not the request path.
The client first resolves a name, then opens an HTTPS connection to the ALB. From there the task talks to the database, the queue and object storage.
Drag service cards to arrange them. Pan or zoom the canvas, select a service to see its role, or select an arrow to isolate that flow.
Least privilege should follow the traffic chain.
Each layer only accepts traffic from the security group directly in front of it. Only 443 is open to the internet.
Drag service cards to arrange them. Pan or zoom the canvas, select a service to see its role, or select an arrow to isolate that flow.
How the architecture can scale
HTTP load
The ALB spreads traffic across healthy ECS targets. Request load drives the service task count.
Compute capacity
EC2 capacity has to grow before new tasks can be placed, so host capacity scales as its own layer ahead of the tasks.
Async work
Queue depth drives the number of workers, independently of the API.
Database
Connections, reads, writes, storage and failover scale independently of compute. RDS Proxy absorbs connection spikes while the instance size stays the same.
What happens when something fails?
01Container failure
What happensECS replaces the stopped task and the ALB stops routing to it until it passes health checks.
What it relies onNeeds spare host capacity for the replacement task to be placed.
02EC2 container instance failure
What happensECS reschedules its tasks onto the remaining container instances.
What it relies onEach AZ keeps headroom so losing one host does not block rescheduling.
03Availability Zone failure
What happensThe ALB routes all traffic to the surviving AZ; NAT and compute there keep running independently.
What it relies onCapacity in one AZ has to absorb the full load, so tasks are spread evenly across both.
04Consumer failure
What happensUnacknowledged messages reappear after the visibility timeout and are retried.
What it relies onConsumers must be idempotent; messages that keep failing go to a dead-letter queue.
05Database failure
What happensThe second instance in AZ-1b keeps a copy of the data outside the failed zone.
What it relies onRDS Proxy keeps client connections stable while the database endpoint changes.
06Public target health failure
What happensThe ALB takes the unhealthy target out of rotation.
What it relies onHealth-check grace periods are long enough for the app to start before it is judged.
How releases reach production
CI builds the image once and pushes it to ECR with an immutable tag. A new task-definition revision points at that tag, and ECS rolls it out behind the ALB health checks. Rolling back means pointing the service at the previous revision.
Availability, backup and recovery are different promises
CloudWatch
“What is happening?”Metrics, logs and alarms for ECS, the ALB and RDS.
CloudTrail
“Who changed what?”Records every AWS API call so infrastructure changes are traceable.
Systems Manager
Managed administrationSession access and patching for hosts, without opening inbound SSH.
AWS Backup
Recoverable copiesScheduled backups of RDS and S3, kept separate from the HA setup.
Resilience and control are not free
Two NAT Gateways
Availability ↑ AZ-local egress avoids one shared NAT failure domain.
Cost ↑ Hourly and data-processing costs apply to each gateway.
ECS on EC2
Control ↑ Host shape, AMI and reserved capacity stay explicit.
Operations ↑ Patching, draining and scaling the hosts remain yours.
Bastion host
Explicit path Familiar jump-host administration.
Attack surface Another host and credential path to secure.
RDS Proxy
Connection control Pools client connections during task churn.
Managed cost Adds a hop, price and engine-specific behavior.