ECS Part 2: Publishing the web/api Services and Auto Scaling
With RDS added in the previous chapter, Tasuku's Terraform code now has the networking, DNS, compute foundation, and database all in place. However, the ECS work in Chapter 8 stops at confirming that things start up with a one-off run-task using the hello-world image; it isn't yet in a state where multiple tasks stay running continuously, the way a real web application would be. In this chapter, we build out two ECS services, web and api, and publish them through the ALB.
10.1. From task definition to service
The run-task command used in Chapter 8 is an operation that starts a task exactly once. Once the task finishes, that's the end of it. An ECS service, by contrast, is a mechanism that keeps a specified number of tasks running continuously.1 If a task terminates abnormally, the service starts a new one to replace it, keeping the specified count running at all times. Services are needed not for tasks like hello-world, the kind that "start up, print a log, and finish," but for web applications and API servers that need to keep listening for HTTP requests.
Services have settings called deploymentMaximumPercent and deploymentMinimumHealthyPercent that control how tasks are swapped out.2 With the default deployment type (rolling update), when switching to a new task definition, these two values set the upper and lower bounds, as percentages, on the number of running tasks allowed during the deployment. The standard scheduler's defaults are 200% and 100% respectively,2 so for example, a service with a desired task count of 2 would temporarily scale up to as many as 4 tasks while always keeping at least 2 tasks running during the swap. This chapter uses the default values as they are.
10.2. Two services: web and api
In section 1.3 of Chapter 1, we established the premise that the Tasuku app consists of two container images: "web," a browser-facing screen listening on port 3000, and "api," a JSON API listening on port 8080. In that same section, we noted that the image names and port numbers would become Terraform variables. Here, we implement exactly that.
The images are var.web_image and var.api_image, and, for the same reason as var.domain_name in Chapter 7, we don't set default values for them. That's because the Tasuku app doesn't actually exist, and the premise is that readers will substitute their own images. The ports are var.web_port (default 3000) and var.api_port (default 8080); since these values were already settled in section 1.3, we give them default values.
The task definitions share the execution role and task role (both already created in Chapter 8) between web and api. They can be shared because both access the same ECR repository, the same log group, the same RDS instance, and the same S3 bucket. The only things set up individually are the port number and the container name.
10.3. Target groups and health checks
For the ALB to forward requests to ECS tasks, a target group is required. The tasks in Chapter 8 run in awsvpc network mode, which ties them directly to an Elastic Network Interface rather than an EC2 instance. Because of this, the target group's target_type must be ip rather than the default instance.3 As long as we're using the Fargate launch type, there's no room for doubt in this choice.
We use /healthz as the health check path. In section 1.3 of Chapter 1, we established, as a premise the Tasuku app satisfies, that it has "an endpoint that returns 200 for GET /healthz." We create a dedicated target group for each of web and api, with both sharing /healthz as the health check path.
10.4. Host-based routing and security group connectivity
Ever since Chapter 6, the port 443 listener has only returned a fixed string via fixed-response. In this chapter, we finally switch it over to forward to actual target groups. As in the overview diagram in section 1.4 of Chapter 1, the destination is determined by hostname: requests to tasuku.example go to web, and requests to api.tasuku.example go to api.
We set web as the default forwarding destination and override only api with a listener rule. The listener rule's host_header condition supports wildcards and case-insensitive matching.4 This time, an exact match on api.${var.domain_name} alone is enough. The single certificate with a wildcard SAN issued in Chapter 7 already covers this, so there's no need to issue an additional one for the api subdomain. We add an Alias record for api.${var.domain_name} in Route 53, pointing to the same ALB.
Switching the routing alone doesn't yet let actual traffic through. In Chapter 6, we deliberately left the ALB security group's egress empty, because at that point no target group existed yet, so the ALB had no need to reach a forwarding destination. Now that we have two target groups, we add egress rules to the ALB's SG for the web and api ports respectively. On the ECS task SG side, in addition to the port 3000 ingress rule for web created in Chapter 8, we add a port 8080 ingress rule for api. With this, the path from the ALB to the ECS tasks is now connected at the security group level as well.
10.5. Secrets Manager injection
In section 9.9 of Chapter 9, we retrieved the password from Secrets Manager for verification purposes, manually extracting it with jq and passing it to the container. We can't have web and api, which run continuously as services, do this manual work every single time. We use the ECS task definition's secrets parameter to automatically inject the password when the container starts.
For secrets' valueFrom, appending :<key name>:: to the end of the secret's full ARN lets you extract just a single key from a JSON-formatted secret.5 Since we know the secret from Chapter 9 contains a password key, we bind the value <secret_arn>:password:: to the environment variable name DATABASE_PASSWORD. The hostname, port number, database name, and username aren't sensitive, so we pass them directly from RDS's respective attributes via the regular environment.
Resolving secrets requires the secretsmanager:GetSecretValue permission on the execution role. The custom policy for the execution role that we created in Chapter 8 wasn't designed with this use case in mind, and what's more, it can't be edited. As established by the constraint in section 8.5 of Chapter 8, attaching an AWS managed policy isn't an option, and the Terraform executor isn't granted iam:CreatePolicyVersion on the existing policy either. Just as we did in Chapter 9, we add this as a separate, new policy.
The permissions required depend on whether the KMS key encrypting this secret is a customer-managed key or an AWS managed key. Since the aws_db_instance in Chapter 9 doesn't specify master_user_secret_kms_key_id, an AWS managed key (Secrets Manager's default key) is used.6 The kms:Decrypt permission is only needed when the secret is encrypted with a customer-managed key,7 so it's not needed for this configuration, which uses the default AWS managed key. The new policy on the execution role grants only secretsmanager:GetSecretValue.
10.6. S3 permissions for the task role
In the preview table in section 3.4 of Chapter 3, we established that adding permissions to the task role would happen in this chapter. As of Chapter 8, the task role had only a trust policy and no attached policy, so the container had no permission to access the S3 attachments bucket.
The required S3 actions are s3:GetObject, s3:PutObject, and s3:DeleteObject, corresponding to retrieving, adding, and deleting objects.8 These target the object ARN (<bucket ARN>/*). If listing the contents of the bucket is needed, we also add s3:ListBucket, but note that this targets the bucket ARN itself, so the shape of the Resource differs.8 We attach a new policy with these permissions to the shared task role (as noted in section 10.2, the task role is shared between web and api, so this permission is granted to both tasks).
10.7. Deployment strategy: rolling + circuit breaker
When switching to a new task definition, if the post-switch tasks keep failing to start, the service can end up stuck indefinitely with old and new tasks mixed together. Enabling the deployment circuit breaker automatically rolls the service back to the last healthy state once a deployment is judged to have failed.9 We enable both enable and rollback, configuring it to automatically roll back a failed deployment.
We also set health_check_grace_period_seconds. This value is a grace period, right after a task starts, during which load balancer health check failures are ignored. The default is 0, meaning no grace period.10 If the container takes several seconds to start up, the task could be judged unhealthy before the first health check result even comes in, potentially causing a cycle of starting and stopping. We set a 60-second grace period to ride out this unstable period right after startup.
10.8. Auto Scaling
In Chapter 8, we hardcoded the task's CPU and memory as fixed values of 256 and 512 MiB. In this chapter, we automatically scale the number of running tasks up and down based on load — adjusting the horizontal count rather than the vertical size.
Application Auto Scaling's target tracking is a mechanism that automatically adjusts the task count so that a specified metric stays close to a target value. ECSServiceAverageCPUUtilization is one of the predefined metrics available for ECS services.11 We set the target value to 70%: tasks are added when this is exceeded, and removed when it falls below. Based on the scale assumptions from section 1.2 of Chapter 1 (a few dozen tenants, on the order of a few hundred concurrent users), we scale within a range of a minimum of 2 and a maximum of 4. The minimum is set to 2 because, for a service whose subnets span multiple Availability Zones, ECS tries to place new tasks evenly across those Availability Zones.12 With a minimum of 1, the one continuously running task could end up confined to a single Availability Zone.
Once Auto Scaling starts adjusting the task count, the desired_count value managed by Terraform and the value Auto Scaling actually sets will drift apart. Left as is, every subsequent terraform plan would detect Auto Scaling's changes as a diff. Specifying desired_count in the ignore_changes of the aws_ecs_service's lifecycle block makes Terraform ignore external changes to this value, including those from Auto Scaling.13
10.9. Terraform implementation
We write what we've covered so far into a new file, ecs-service.tf, following this series' established policy of "gathering a chapter's main resources into a single file." Changes to existing resources — target groups, listeners, Route 53 records, and security group rules — go into alb.tf, ecs.tf, and route53-acm.tf respectively. Only the new resources — new policies for the execution role and task role, plus the task definitions, services, and Auto Scaling — go into this ecs-service.tf. First, the two new policies: we add the permissions decided in sections 10.5 and 10.6, each as a separate policy.
resource "aws_iam_policy" "ecs_secrets_read" {
name = "${local.name_prefix}-ecs-secrets-read-policy"
path = "/${var.project}/"
description = "Permission to read the Secrets Manager secret containing the RDS master password. kms:Decrypt is not included, since the default AWS managed KMS key is used"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "ReadRdsMasterSecret"
Effect = "Allow"
Action = ["secretsmanager:GetSecretValue"]
Resource = aws_db_instance.main.master_user_secret[0].secret_arn
},
]
})
}
resource "aws_iam_role_policy_attachment" "ecs_secrets_read" {
role = aws_iam_role.ecs_task_execution.name
policy_arn = aws_iam_policy.ecs_secrets_read.arn
}
resource "aws_iam_policy" "ecs_task_s3" {
name = "${local.name_prefix}-ecs-task-s3-policy"
path = "/${var.project}/"
description = "Read, write, delete, and list permissions for the attachments S3 bucket"
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "AttachmentsObjectAccess"
Effect = "Allow"
Action = [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
]
Resource = "${aws_s3_bucket.attachments.arn}/*"
},
{
Sid = "AttachmentsListBucket"
Effect = "Allow"
Action = ["s3:ListBucket"]
Resource = aws_s3_bucket.attachments.arn
},
]
})
}
resource "aws_iam_role_policy_attachment" "ecs_task_s3" {
role = aws_iam_role.ecs_task.name
policy_arn = aws_iam_policy.ecs_task_s3.arn
}
Here are the target groups and host-based routing from section 10.4. We switch the port 443 listener's default over to forward, and route only api through a listener rule.
resource "aws_lb_target_group" "web" {
name = "${local.name_prefix}-web-tg"
port = var.web_port
protocol = "HTTP"
vpc_id = aws_vpc.main.id
target_type = "ip"
health_check {
path = "/healthz"
}
tags = {
Name = "${local.name_prefix}-web-tg"
}
}
resource "aws_lb_target_group" "api" {
name = "${local.name_prefix}-api-tg"
port = var.api_port
protocol = "HTTP"
vpc_id = aws_vpc.main.id
target_type = "ip"
health_check {
path = "/healthz"
}
tags = {
Name = "${local.name_prefix}-api-tg"
}
}
We rewrite the default_action of aws_lb_listener.https (already created in Chapter 7) from fixed-response to a forward to web.
resource "aws_lb_listener" "https" {
load_balancer_arn = aws_lb.main.arn
port = "443"
protocol = "HTTPS"
ssl_policy = "ELBSecurityPolicy-TLS13-1-2-Res-PQ-2025-09"
certificate_arn = aws_acm_certificate_validation.main.certificate_arn
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.web.arn
}
}
resource "aws_lb_listener_rule" "api" {
listener_arn = aws_lb_listener.https.arn
priority = 100
action {
type = "forward"
target_group_arn = aws_lb_target_group.api.arn
}
condition {
host_header {
values = ["api.${var.domain_name}"]
}
}
}
We add egress rules to the ALB's SG and an api-facing ingress rule to the ECS task SG (the web-facing ingress rule was already created in Chapter 8).
resource "aws_vpc_security_group_egress_rule" "alb_to_web" {
security_group_id = aws_security_group.alb.id
referenced_security_group_id = aws_security_group.ecs_tasks.id
from_port = var.web_port
to_port = var.web_port
ip_protocol = "tcp"
description = "Forward to the ECS task SG (web service)"
}
resource "aws_vpc_security_group_egress_rule" "alb_to_api" {
security_group_id = aws_security_group.alb.id
referenced_security_group_id = aws_security_group.ecs_tasks.id
from_port = var.api_port
to_port = var.api_port
ip_protocol = "tcp"
description = "Forward to the ECS task SG (api service)"
}
resource "aws_vpc_security_group_ingress_rule" "ecs_tasks_from_alb_api" {
security_group_id = aws_security_group.ecs_tasks.id
referenced_security_group_id = aws_security_group.alb.id
from_port = var.api_port
to_port = var.api_port
ip_protocol = "tcp"
description = "Allow only from the ALB (used for turning api into a service and host-based routing in Chapter 10)"
}
We add an Alias record for api.${var.domain_name} in Route 53.
resource "aws_route53_record" "api" {
zone_id = aws_route53_zone.main.zone_id
name = "api.${var.domain_name}"
type = "A"
alias {
name = aws_lb.main.dns_name
zone_id = aws_lb.main.zone_id
evaluate_target_health = true
}
}
Here's the task definition. We gather web's DB-connection environment variables and secrets into locals so they can be shared with api. For the username, rather than assembling a string, we reference aws_db_instance.main.username directly, so that things keep working silently even if the value on the RDS side changes in the future.
locals {
db_environment = [
{ name = "DATABASE_HOST", value = aws_db_instance.main.address },
{ name = "DATABASE_PORT", value = tostring(aws_db_instance.main.port) },
{ name = "DATABASE_NAME", value = aws_db_instance.main.db_name },
{ name = "DATABASE_USER", value = aws_db_instance.main.username },
]
db_secrets = [
{ name = "DATABASE_PASSWORD", valueFrom = "${aws_db_instance.main.master_user_secret[0].secret_arn}:password::" },
]
}
resource "aws_ecs_task_definition" "web" {
family = "${local.name_prefix}-web"
requires_compatibilities = ["FARGATE"]
network_mode = "awsvpc"
cpu = 256
memory = 512
execution_role_arn = aws_iam_role.ecs_task_execution.arn
task_role_arn = aws_iam_role.ecs_task.arn
runtime_platform {
operating_system_family = "LINUX"
cpu_architecture = "X86_64"
}
container_definitions = jsonencode([
{
name = "web"
image = var.web_image
essential = true
portMappings = [
{
containerPort = var.web_port
protocol = "tcp"
}
]
# WHOAMI_PORT_NUMBER makes the verification placeholder (traefik/whoami) listen on
# containerPort. Tasuku's actual app image doesn't reference this variable, so it's ignored
environment = concat(local.db_environment, [
{ name = "WHOAMI_PORT_NUMBER", value = tostring(var.web_port) },
])
secrets = local.db_secrets
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.app.name
"awslogs-region" = var.region
"awslogs-stream-prefix" = "web"
}
}
}
])
tags = {
Name = "${local.name_prefix}-web"
}
}
The api task definition is the same as web's, except for the port number and the S3 bucket name environment variable. We show only the differences.
resource "aws_ecs_task_definition" "api" {
family = "${local.name_prefix}-api"
# requires_compatibilities / network_mode / cpu / memory / execution_role_arn /
# task_role_arn / runtime_platform are the same as web
container_definitions = jsonencode([
{
name = "api"
image = var.api_image
essential = true
portMappings = [
{
containerPort = var.api_port
protocol = "tcp"
}
]
# WHOAMI_PORT_NUMBER makes the verification placeholder (traefik/whoami) listen on
# containerPort. Tasuku's actual app image doesn't reference this variable, so it's ignored
environment = concat(local.db_environment, [
{ name = "ATTACHMENTS_BUCKET", value = aws_s3_bucket.attachments.id },
{ name = "WHOAMI_PORT_NUMBER", value = tostring(var.api_port) },
])
secrets = local.db_secrets
logConfiguration = {
logDriver = "awslogs"
options = {
"awslogs-group" = aws_cloudwatch_log_group.app.name
"awslogs-region" = var.region
"awslogs-stream-prefix" = "api"
}
}
}
])
tags = {
Name = "${local.name_prefix}-api"
}
}
Here's the service itself. Creating a service while the target group isn't yet associated with a listener or listener rule can sometimes cause an error due to timing on the AWS API side.14 We make the dependency on the listener and listener rule explicit with depends_on to guarantee this ordering.
resource "aws_ecs_service" "web" {
name = "${local.name_prefix}-web"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.web.arn
desired_count = 2
launch_type = "FARGATE"
network_configuration {
subnets = [for k, v in aws_subnet.this : v.id if local.subnets[k].tier == "private-app"]
security_groups = [aws_security_group.ecs_tasks.id]
assign_public_ip = false
}
load_balancer {
target_group_arn = aws_lb_target_group.web.arn
container_name = "web"
container_port = var.web_port
}
health_check_grace_period_seconds = 60
deployment_circuit_breaker {
enable = true
rollback = true
}
lifecycle {
ignore_changes = [desired_count]
}
tags = {
Name = "${local.name_prefix}-web"
}
depends_on = [aws_lb_listener.https]
}
resource "aws_ecs_service" "api" {
name = "${local.name_prefix}-api"
cluster = aws_ecs_cluster.main.id
task_definition = aws_ecs_task_definition.api.arn
desired_count = 2
launch_type = "FARGATE"
network_configuration {
subnets = [for k, v in aws_subnet.this : v.id if local.subnets[k].tier == "private-app"]
security_groups = [aws_security_group.ecs_tasks.id]
assign_public_ip = false
}
load_balancer {
target_group_arn = aws_lb_target_group.api.arn
container_name = "api"
container_port = var.api_port
}
health_check_grace_period_seconds = 60
deployment_circuit_breaker {
enable = true
rollback = true
}
lifecycle {
ignore_changes = [desired_count]
}
tags = {
Name = "${local.name_prefix}-api"
}
depends_on = [aws_lb_listener_rule.api]
}
Finally, Auto Scaling. We repeat the same configuration twice, once for web and once for api.
resource "aws_appautoscaling_target" "web" {
service_namespace = "ecs"
resource_id = "service/${aws_ecs_cluster.main.name}/${aws_ecs_service.web.name}"
scalable_dimension = "ecs:service:DesiredCount"
min_capacity = 2
max_capacity = 4
}
resource "aws_appautoscaling_policy" "web" {
name = "${local.name_prefix}-web-cpu-tracking"
policy_type = "TargetTrackingScaling"
resource_id = aws_appautoscaling_target.web.resource_id
scalable_dimension = aws_appautoscaling_target.web.scalable_dimension
service_namespace = aws_appautoscaling_target.web.service_namespace
target_tracking_scaling_policy_configuration {
predefined_metric_specification {
predefined_metric_type = "ECSServiceAverageCPUUtilization"
}
target_value = 70
}
}
# aws_appautoscaling_target.api / aws_appautoscaling_policy.api have the same configuration too
We add the target group ARNs and service names used for verification to outputs.tf.
output "web_target_group_arn" {
description = "ARN of the target group for the web service (used to check health check status during verification, etc.)"
value = aws_lb_target_group.web.arn
}
output "api_target_group_arn" {
description = "ARN of the target group for the api service (used to check health check status during verification, etc.)"
value = aws_lb_target_group.api.arn
}
output "web_service_name" {
description = "web service name (referenced via aws ecs describe-services, etc.)"
value = aws_ecs_service.web.name
}
output "api_service_name" {
description = "api service name (referenced via aws ecs describe-services, etc.)"
value = aws_ecs_service.api.name
}
10.10. Verification
We run terraform fmt -check -recursive -diff and terraform validate to confirm the syntax and provider schema are consistent. That said, it's important to note that what validate guarantees in this chapter is narrower than in previous chapters. Since container_definitions is passed as a JSON string, validate won't catch mistakes in secrets key names or portMappings values. Such errors only surface once you attempt to register the task definition itself (RegisterTaskDefinition) or start a task. The correctness of this chapter's Terraform code is guaranteed not by passing validate, but by cross-checking against the primary sources presented in sections 10.5 and 10.9.
Here are the steps if you actually want to apply and verify this. First, since we couldn't find a widely used public image that guarantees a 200 response on /healthz, we use a demo image called traefik/whoami in place of web and api. As with the hello-world image in Chapter 8, we re-host it on ECR and specify that tag for var.web_image and var.api_image. This image has handlers for fixed paths like /health, but it doesn't provide a distinct /healthz path.15 However, requests to unregistered paths are handled by Go's standard router falling back to the handler registered on the root path /,16 so /healthz ends up returning 200 as well. Note that this isn't a guarantee of a dedicated response for /healthz — it's merely borrowing the fallback behavior for unregistered paths.
The listening port is also aligned with the 3000 and 8080 specified for containerPort, by passing the WHOAMI_PORT_NUMBER environment variable in the task definitions from section 10.9. This image's default listening port is 80.17
docker pull traefik/whoami
docker tag traefik/whoami <account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/tasuku-dev-app:web
docker tag traefik/whoami <account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/tasuku-dev-app:api
docker push <account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/tasuku-dev-app:web
docker push <account-id>.dkr.ecr.ap-northeast-1.amazonaws.com/tasuku-dev-app:api
Once the apply finishes, use aws ecs describe-services to confirm that deployments[].rolloutState for web_service_name and api_service_name is COMPLETED. Next, run aws elbv2 describe-target-health against web_target_group_arn and api_target_group_arn respectively, and confirm that TargetHealth.State is healthy for all registered targets. At this point, the tasks have started as a service and have also been judged healthy by the ALB.
Finally, we verify things over HTTP directly. Access tasuku.example and api.tasuku.example each over HTTPS, and confirm that a 200 is returned and that the response content, as expected, comes back from a different container for web versus api. Whether host-based routing is actually working only becomes clear once you compare these two responses.
What this procedure confirms is, strictly speaking, only that the tasks start up and that HTTP traffic passes through the ALB. Because traefik/whoami doesn't reference app-specific environment variables or secrets like DATABASE_PASSWORD, the most we can confirm for the Secrets Manager injection in section 10.5 is that "the IAM permissions and ARN reference are valid, and they don't block the task from starting." Unlike the psql-check in section 9.9 of Chapter 9, we don't go so far as to confirm that you can actually connect to the database with the injected values.
The ECS services in this chapter are held to the same verification standard as before. We don't guarantee the results of actually applying this to AWS and running through this procedure.
Summary
- An ECS service is a mechanism that keeps a specified number of tasks running continuously, a different role from the one-off startup of
run-task - As previewed in section 1.3 of Chapter 1, we implemented the two services web (port 3000) and api (port 8080). Both the images and ports are Terraform variables, and, for the same reason as
var.domain_name, we didn't set default values for the images - The target group's
target_typemust beip, because Fargate tasks inawsvpcnetwork mode are tied to an ENI rather than an EC2 instance - We switched the port 443 listener from fixed-response to forward, routing only the
api.subdomain to the api target group based on the host header. The wildcard SAN certificate from Chapter 7 covers this without any changes - We added the ALB security group's egress rules, which we had deliberately left empty in Chapter 6, completing the connectivity path from the ALB to the ECS tasks
- We switched the RDS password, already integrated with Secrets Manager, to automatic injection via the ECS task definition's
secrets, replacing Chapter 9's manualjqextraction with something suited to a service. Since the KMS key is the default AWS managed key,kms:Decryptisn't needed on the execution role - As previewed in Chapter 3, we added access permissions for the S3 attachments bucket to the task role as a new policy
- We enabled automatic rollback of failed deployments with the deployment circuit breaker, and set a health check grace period right after startup with
health_check_grace_period_seconds - We configured Auto Scaling to automatically adjust the task count within a range of 2 to 4 based on CPU utilization. If Chapter 8's fixed cpu/memory was the vertical size, this is the horizontal count
- To keep the
desired_countmanaged by Terraform from conflicting with Auto Scaling's changes, we excluded it vialifecycle.ignore_changes
Next steps
- Chapter 11: Standardizing Deployment and Separating Environments: End-to-end connectivity checks, a standard ECS deployment procedure, and a refactor into a module + envs structure
- Chapter 9: RDS: Design decisions for managed PostgreSQL and Secrets Manager integration
- Series table of contents: The structure of all 12 chapters and how to read through them