Standardizing Deployment and Separating Environments: From Connectivity Checks to a module + envs Structure
By the end of the previous chapters, the Tasuku infrastructure was fully assembled, from the VPC through the ECS services and RDS. However, the verification we did along the way was confined to the scope each chapter was responsible for. In this chapter, we first nail down an end-to-end connectivity check that spans the entire infrastructure, along with a standard procedure for ECS deployments. We then pay off the foreshadowing from Chapter 2, where we started including an environment identifier in name_prefix, and refactor main/ — which has been a single-environment, flat structure — into a modules/ + envs/dev,prod structure that can handle multiple environments.
11.1. E2E checklist: verifying full connectivity
So far, each chapter has only verified the part it added — Chapter 10 checked from the ALB through to the ECS services, Chapter 9 checked the connection to RDS, and so on. Here, we build a checklist that verifies the entire path from the internet through to RDS, all at once.
| # | Check | Method | Expected Result |
|---|---|---|---|
| 1 | DNS resolution (apex) | dig tasuku.example | Returns an Alias record pointing to the ALB's DNS name |
| 2 | DNS resolution (api) | dig api.tasuku.example | Same as above |
| 3 | TLS certificate validation | curl -v https://tasuku.example | Certificate chain validation passes and the response is received without warnings |
| 4 | ALB target health (web) | aws elbv2 describe-target-health --target-group-arn <web_target_group_arn> | TargetHealth.State is healthy for all registered targets |
| 5 | ALB target health (api) | Same as above (api_target_group_arn) | Same as above |
| 6 | ECS service deployment status | aws ecs describe-services --cluster <cluster> --services <web_service_name> <api_service_name> | deployments[].rolloutState is COMPLETED |
| 7 | HTTP response (web) | curl https://tasuku.example | 200, response from the web container |
| 8 | HTTP response (api) | curl https://api.tasuku.example | 200, response from the api container. Also confirm it's returned from a different container than web |
| 9 | Connectivity to RDS | The same psql-check run-task as Chapter 9 Section 9.9 | SELECT version() responds normally |
| 10 | Outbound internet connectivity from ECS tasks | The task itself reaching the RUNNING state | Indirect evidence that ECR pull and CloudWatch Logs writes are succeeding via the NAT Gateway |
This checklist simply bundles together the verification from previous chapters from a path-based perspective — it doesn't introduce any new verification technique. It uses only existing output (Terraform outputs), and maintains the same verification standard as through Chapter 10 (presented as the procedure to follow if you actually try this in a real AWS environment; the author doesn't guarantee the outcome of the apply).
11.2. Standard deployment procedure
None of the chapters so far have touched on the actual procedure for updating container images. Let's settle on a standard procedure for continuously deploying Tasuku in real operation.
By default, an ECR repository's image_tag_mutability is MUTABLE.1 You can repeatedly push new images to the same tag (such as latest), which is convenient, but it means you lose the ability to trace which build is currently deployed from the tag alone. In this series, the standard is to use the CI pipeline's commit SHA as the image tag. You push with a unique tag like web:a1b2c3d, update the ECS task definition's image to that tag, and then reflect the new task definition with aws ecs update-service.
If you just want to re-pull the same tagged image without changing the service definition itself, use force_new_deployment.2 It can trigger a new deployment even without any change to the task definition or service configuration. Since an ECS task definition itself is immutable once created, and a new revision is created every time you change the container image or environment variables,2 a normal deployment flows as: create a new task-definition revision → update the service.
The deployment circuit breaker we enabled in Chapter 10 Section 10.7 continues to work unchanged with this standard deployment procedure as well. If the switch to a new task definition is judged to have failed, it automatically rolls back to the last healthy state.
11.3. Why separate environments now?
Since Chapter 2, we've included an environment identifier in the resource name prefix, in the form name_prefix = "${var.project}-${var.env}". Through Chapter 9, var.env was effectively always "dev", and there was never a situation where we actually used multiple environments side by side. Here, we finally pay off that foreshadowing.
As a way to handle multiple environments (dev/staging/prod, and so on), Terraform has a feature called CLI workspaces. It's a mechanism for switching between states while keeping the same configuration and the same backend.3 However, the practical consensus is that long-lived environments should be separated using directory separation, not CLI workspaces.3 Environment separation via workspaces makes drift between environments harder to see within a single shared configuration, and it's also a poor fit when you need different access controls per environment (such as IAM policies or backend permissions). In this series too, we separate environments into distinct directories, envs/dev/ and envs/prod/.
11.4. Designing module boundaries
Up to now, main/ has been a flat structure where we added a .tf file for each chapter. We now carve this code — which has grown to 14 files and around 1,600 lines — into reusable units.
If you naively split things up as "one resource file = one module," you run into a circular reference. The one cycle we can actually confirm in this codebase is between the ALB and Route 53/ACM. Because the port-443 listener references the ACM certificate's ARN, and Route 53's Alias record references the ALB's DNS name, splitting out a loadbalancer module and a dns module separately would leave them referencing each other.4 In this series, we resolve this cycle by combining the ALB, Route 53, and ACM into a single edge module.
The final split is these five modules.
| Module | Contents |
|---|---|
network | VPC, 6 subnets, IGW, NAT Gateway, route tables, tightening the default SG |
storage | Attachment bucket, S3 Gateway endpoint |
edge | ALB, target groups, listeners, ALB log bucket, Route 53 zone, ACM certificate |
compute | ECR, ECS cluster, execution/task roles, task definitions, services, Auto Scaling |
database | DB subnet group, parameter group, RDS instance |
The three security groups we've built since Chapter 4 — for the ALB, the ECS tasks, and RDS — are deliberately left out of every module. Because the ALB, ECS, and RDS form a structure where all three reference each other (the ALB's egress references the ECS task SG, the ECS task SG's egress references the RDS SG, and so on), confining them inside a module would bring back the same circular-reference problem. We write the three SGs and their ingress/egress rules directly in the caller, envs/dev,prod, and pass the resulting SG IDs into each module as input variables. For the same reason, Chapter 3's terraform_executors (an account-level concern that belongs to the Terraform executor itself) and the budget alert also stay on the caller side, un-modularized.
The dependency direction between modules is one-way. With network as the foundation, storage, edge, and database all depend on it in parallel, and finally compute receives all three of their outputs — the attachment bucket's ARN, the target groups' ARN, and RDS's connection information.
11.5. envs/dev, prod and the tfvars strategy
We narrow down what changes per environment to the following four items.
| Item | dev | prod | Illustrates |
|---|---|---|---|
RDS's multi_az | false | true | Cost |
RDS's deletion_protection | false | true | Safety |
Auto Scaling's min_capacity/max_capacity | 1 / 2 | 2 / 4 | Scale |
Budget alert's budget_limit_usd | 30 | 100 | Cost |
The initial value of the ECS service's desired_count is tied directly to this min_capacity. In Chapter 10 Section 10.8, the rationale given for setting min_capacity to 2 was an availability requirement: "for a service whose specified subnets span multiple Availability Zones, ECS tries to place new tasks evenly across AZs, so with a minimum of 1, the one constantly running task could end up skewed into a single AZ." Choosing min_capacity = 1 for the dev environment means deliberately stepping back from this availability requirement. It's a trade-off that prioritizes cost in the development environment, accepting that things may end up skewed into a single AZ. The prod environment keeps min_capacity = 2, so it continues to satisfy this requirement.
Other hardcoded values, such as the container's cpu/memory, the ALB's ssl_policy, and the CloudWatch Logs retention period, are declared as variables, but we carry them over as shared default values common to dev and prod. We narrow it down to four items in order to represent the three decision axes — cost, safety, and scale — while keeping the volume of tfvars down.
For state management, we share the single tfstate bucket we created in Chapter 5 Section 5.4, and separate environments by key (the path within the bucket) — envs/dev/terraform.tfstate and envs/prod/terraform.tfstate, for example. A design where each env gets its own bucket is conceivable, but this series doesn't adopt it. The bucket argument of aws_s3_bucket is a Forces new resource attribute (changing the name causes it to be deleted and recreated).5 If you re-apply in the same bootstrap/ directory changing only a variable, like -var="env=prod", it will try to destroy the existing dev bucket before creating the new one. A shared bucket with key separation is less prone to this kind of accident, and it's also the configuration generally recommended for state management across multiple environments.6
domain_name needs careful attention. Because envs/dev/ and envs/prod/ each create their own aws_route53_zone, setting the same domain name in both can leave you with two hosted zones for the same domain name in existence. Since var.domain_name, going back to Chapter 7, has no default value, this accident can happen simply by "copy-pasting tfvars carelessly and using the same one for both." Because name server delegation at the domain registrar can only point to one place, the hosted zone that isn't delegated will never finish ACM certificate DNS validation. In this series' sample, we distinguish them by assigning a subdomain (dev.tasuku.example) to envs/dev/ and the apex domain (tasuku.example) to envs/prod/.
11.6. Zero-downtime migration as an option: the moved block
As a way to move from the existing main/ to the new modules/ + envs/ structure, Terraform provides a mechanism called the moved block. If you write from (the source address) and to (the destination address), Terraform looks for the existing object that corresponds to the source address and reassigns it to the new address without triggering a destroy and create.7 It also supports reassignment across module boundaries — the so-called shim pattern for turning a root-level resource into a module.8
However, this mechanism assumes a refactor within the same Terraform configuration and the same state. Our migration here moves from one root module, main/, to a different root module, envs/dev/ (a different state), so the premise doesn't hold. It's a basic principle of Terraform's module resolution that passing values between modules creates an implicit dependency.9 However, a move that crosses state boundaries itself can't be completed by this mechanism alone. If you actually want to bring your existing state into the new structure, it becomes a two-step process: move the state file itself with something like terraform state mv, and then reconcile the modules within the new structure using moved blocks.
In this series, as explained in the next section, 11.8, a fresh apply is the primary procedure. We introduce the moved block only as an alternative path for readers who have already loaded real data into RDS and want to migrate without downtime; we don't perform hands-on verification of the actual state operations.
11.7. Terraform implementation
We newly create modules/ and envs/dev,prod/ under terraform-verify/refactored/. We leave the existing bootstrap/ and main/ code structure untouched (except for the temporary configuration changes and the destroy involved in tearing down main/, described in Section 11.8). This is where we finally wire up, for the first time, the design from Chapter 5 Section 5.5, which said we would only present backend "s3" as a sample and keep the verification project's main/ running on local state.
As a representative example, we show the network module. variables.tf is shown in full, while main.tf and outputs.tf are excerpts of the main parts. For the other four modules (storage, edge, compute, database), we only describe the key design points that follow, in prose.
# modules/network/variables.tf
variable "name_prefix" {
description = "Prefix for all resource names (the caller passes in a value that already combines the project name and env name as-is)"
type = string
}
variable "vpc_cidr" {
description = "IPv4 CIDR block for the entire VPC"
type = string
}
# modules/network/main.tf (excerpt, VPC/subnet definition part)
locals {
subnets = {
"public-az1" = { tier = "public", az_key = "az1", cidr = "10.0.0.0/24", public_ip_launch = true }
"public-az2" = { tier = "public", az_key = "az2", cidr = "10.0.1.0/24", public_ip_launch = true }
"private-app-az1" = { tier = "private-app", az_key = "az1", cidr = "10.0.10.0/24", public_ip_launch = false }
"private-app-az2" = { tier = "private-app", az_key = "az2", cidr = "10.0.11.0/24", public_ip_launch = false }
"private-data-az1" = { tier = "private-data", az_key = "az1", cidr = "10.0.20.0/24", public_ip_launch = false }
"private-data-az2" = { tier = "private-data", az_key = "az2", cidr = "10.0.21.0/24", public_ip_launch = false }
}
# ... (same logic as vpc.tf in Chapter 4, only name_prefix is replaced with var.name_prefix)
}
resource "aws_vpc" "main" {
cidr_block = var.vpc_cidr
enable_dns_support = true
enable_dns_hostnames = true
tags = {
Name = "${var.name_prefix}-vpc"
}
}
# aws_subnet.this / aws_internet_gateway.main / aws_route_table.* /
# aws_nat_gateway.this / aws_default_security_group.main have the same content as vpc.tf in Chapter 4
# modules/network/outputs.tf (excerpt)
output "vpc_id" {
description = "The VPC's ID"
value = aws_vpc.main.id
}
output "private_app_route_table_ids" {
description = "List of private-app route table IDs (referenced when associating the S3 Gateway endpoint)"
value = [for v in aws_route_table.private_app : v.id]
}
# public_subnet_ids / private_app_subnet_ids / private_data_subnet_ids /
# nat_gateway_public_ips are also output
The other four modules (storage, edge, compute, database) also have resource definitions that are nearly identical to the corresponding files in the existing main/. The only difference is that same-file references like local.name_prefix or aws_vpc.main.id are replaced with the input variables var.name_prefix and var.vpc_id. The edge module has an extra output for a specific reason. ECS services have a problem where "creating a service while its target group isn't yet associated with a listener causes an error" (already covered in Chapter 10 Section 10.9); to avoid this, we also output the listener's and listener rule's ARNs.
The caller's envs/dev/main.tf defines the security groups directly, and then calls the five modules in this order.
module "network" {
source = "../../modules/network"
name_prefix = local.name_prefix
vpc_cidr = var.vpc_cidr
}
resource "aws_security_group" "alb" {
vpc_id = module.network.vpc_id
# ... ingress/egress rules have the same content as alb.tf in Chapters 6 and 10
}
# ecs_tasks and rds security groups are also defined here in the same way
module "compute" {
source = "../../modules/compute"
name_prefix = local.name_prefix
ecs_tasks_security_group_id = aws_security_group.ecs_tasks.id
web_target_group_arn = module.edge.web_target_group_arn
alb_listener_https_arn = module.edge.alb_listener_https_arn
rds_address = module.database.rds_address
autoscaling_min_capacity = var.autoscaling_min_capacity
autoscaling_max_capacity = var.autoscaling_max_capacity
# ... other inputs omitted
}
Inside the compute module, aws_ecs_service.web has a depends_on for the same reason as in Chapter 10, but the target isn't a resource within the same module — it's the input variable var.alb_listener_https_arn.
resource "aws_ecs_service" "web" {
# ... task_definition / network_configuration / load_balancer are the same as in Chapter 10
depends_on = [var.alb_listener_https_arn]
}
We confirmed with terraform graph whether a depends_on that crosses modules through a variable actually works. The generated graph does in fact include the dependency edge module.compute.aws_ecs_service.web -> module.edge.aws_lb_listener.https,10 which confirms that the intended ordering is guaranteed at the graph level.
envs/prod/ has the same main.tf and outputs.tf as envs/dev/; the only differences are the default values in variables.tf (the four items from Section 11.5) and the backend key in versions.tf.
11.8. Switching environments: tearing down the old main/ → standing up envs/dev → verification
When thinking through the migration procedure to the new structure, there's a constraint we can't overlook. main/'s name_prefix is "${var.project}-${var.env}", and the default value of var.env is "dev". Since envs/dev/ also uses the same env=dev, both end up with the same name_prefix, tasuku-dev. Resources such as the S3 bucket name, the RDS instance identifier, and the ECR repository name all derive their unique names from this name_prefix. In other words, if we do a fresh apply into envs/dev/ while leaving the old main/ in place, the apply will fail due to duplicate creation of identically named resources.
For this reason, the order of operations is as follows.
- Completely destroy the old
main/. Since RDS hasdeletion_protection = true, first change it todeletion_protection = falseand re-apply, then runterraform destroy. The S3 attachment bucket and the ALB log bucket don't haveforce_destroyset, so they can't be deleted while objects remain inside them. Either empty the buckets, or temporarily setforce_destroy = trueand re-apply. Likewise for the ECR repository — if images remain, either delete them or temporarily setforce_delete = true. - After the destroy completes, revert the temporarily loosened settings (
deletion_protection,force_destroy,force_delete) to their original values, and commit that. This is because the rollback procedure below assumes we can re-apply this code as-is. If the loosened code were left in place, then in the event of a rollback re-apply, the environment would end up rebuilt with weaker-than-intended settings — no deletion protection, force-delete enabled. - Do a fresh apply into
envs/dev/. This is the usualterraform init→terraform applyprocedure. - Verify with the E2E checklist from Section 11.1.
Because resources with globally/account-unique names would collide, the old and new structures can't run in parallel. If a rollback is needed, we handle it at the code level rather than the resource level. Since the main/ code remains as-is in a directory separate from refactored/ (with the loosened settings restored per step 2), if a problem occurs on the envs/dev/ side, we can re-apply the original structure using the main/ code.
Readers who have already loaded real data into RDS or elsewhere can't use this procedure. Consider the zero-downtime migration with the moved block covered in Section 11.6.
We ran terraform fmt -check -recursive and terraform validate against both the five modules/ and envs/dev,prod, confirming syntax and provider schema consistency. However, as in Chapter 10 Section 10.10, what this confirms is limited in scope. Whether the input variables passed between modules are semantically correct — for example, whether an SG ID has been passed to the wrong module — isn't something validate detects.
Summary
- We built an E2E checklist that verifies the entire path from the internet through to RDS — a perspective that checks connectivity across the whole path, not just the scope each individual chapter was responsible for
- As the standard ECS deployment procedure, we sorted out when to use commit-SHA tagging versus
force_new_deployment - We paid off the foreshadowing, going back to Chapter 2, of introducing an environment identifier into
name_prefix, and chose directory separation (envs/dev,prod) over CLI workspaces - We split the code into five modules:
network,storage,edge,compute, anddatabase. To avoid a circular reference between ALB and Route 53/ACM, we merged them into theedgemodule, and left the three security groups un-modularized, on the caller side - We narrowed the items whose values change between dev and prod down to four: RDS's
multi_azanddeletion_protection, Auto Scaling'smin/max_capacity, and the budget alert - For state management, we used a single tfstate bucket with key-based separation, and confirmed that
domain_namemust always differ between dev and prod - We confirmed that the
movedblock assumes a refactor within the same state, and can't be used for a migration between separate root modules - Because the old
main/and the newenvs/dev/collide on resource names, they can't run in parallel, so we designed the migration to follow the order "fully destroy the old main/, then do a fresh apply of envs/dev"
Next steps
- Chapter 12: Mapping the Billing Structure and a Safe Ending: the billing structure map, cost-reduction patterns, safe teardown, and next steps
- Chapter 10: ECS Part 2: Publishing the web/api services and Auto Scaling
- Series table of contents: the structure of all 12 chapters and how to work through them