Reworking an $823 AWS Bill in Two Days

Cloud cost optimization isn’t simply a matter of “switching to smaller servers.” It starts with understanding the actual workload, then using billing data and monitoring to identify the true sources of cost, and finally verifying through rolling deployments, health checks, and cold-start tests that the optimized system remains reliable.

Background: Fixed Costs in a Production Environment

This is a SaaS system that has gone live and supports real business operations. It maintains both Production and Staging AWS environments and includes a Dashboard, Gateway API, OpenAPI, SuperAdmin, asynchronous Workers, and infrastructure such as Aurora, Valkey, ALB, NAT Gateway, and CloudWatch.

Although the system’s overall load and resource utilization are relatively low, the UnblendedCost recorded by AWS Cost Explorer for the full billing cycle before optimization reached $822.96.

The goal was to bring monthly infrastructure costs down to roughly $100–$200 while preserving the ability to scale quickly as the business grows.

These two goals naturally conflict. A fully provisioned high-availability architecture creates significant fixed costs regardless of actual traffic. On the other hand, consolidating every service onto a single inexpensive server might reduce the bill further, but would introduce a single point of failure and increase future migration costs.

We therefore established three constraints before making any changes:

• Production changes must not cause prolonged downtime;
• The path to future scaling must remain intact;
• No resource should be removed based solely on intuition.

Look at the Bill First, Not the Servers

We began by breaking down the pre-optimization billing cycle by AWS service:

• Amazon ECS: $276.59
• RDS: $174.79
• EC2 – Other: $142.32
• Elastic Load Balancing: $77.82
• CloudWatch: $57.10
• VPC: $41.84

Further analysis revealed that within the EC2 – Other category, hourly fees and data processing charges for just two NAT Gateways accounted for $135.72.

ALB LCU traffic charges were minimal, with most of the cost coming from the fixed hourly fees of running multiple load balancers. CloudWatch costs, meanwhile, were almost entirely attributable to metric monitoring.

This showed that the high bill was not caused by a large volume of requests. Instead, it came from simultaneously paying for multiple always-on compute resources, ingress and egress infrastructure, database instances, and enhanced monitoring.

The optimization priority therefore became clear: address hourly fixed costs first, then optimize smaller usage-based charges.

First, Determine Which Capabilities Are Non-Negotiable

One of the easiest mistakes in cloud cost optimization is to rank resources purely by price without first defining the system’s operational boundaries.

Production already supports real business activity, so the database, cache, object storage, secrets management, and a stable public ingress layer all needed to remain. What could be adjusted were replica counts, task sizes, and redundant infrastructure.

Staging serves a different purpose. It exists for testing and release validation and does not need to be available 24/7. That means startup latency can be traded for substantially lower always-on costs.

We also chose not to migrate every service onto a single low-cost instance. Such an approach could reduce the bill further, but it would concentrate the application, ingress, and deployment process into a single failure domain and disrupt the existing deployment workflow.

By keeping ECS, ALB, and Aurora, future scaling can still be handled by increasing task counts, task sizes, or database capacity without redesigning the entire architecture.

Every optimization therefore had to answer three questions:

1. Does the current workload actually require this capacity?
2. What capability do we lose if we remove it?
3. Can we restore that capability quickly when the business grows?

Those questions matter more than asking whether another ten dollars can be removed from the bill.

Consolidate ALBs While Keeping a Mature Ingress Layer

An Application Load Balancer can be thought of as the traffic coordinator at the front of the system.

Initially, both Production and Staging had multiple ALBs serving different subsystems. At low traffic levels, the fixed hourly cost of each ALB was more significant than the actual traffic-related charges.

We ultimately retained only one shared ALB per environment. Using Host Header and Path Rules, requests for the Dashboard, Gateway, OpenAPI, and SuperAdmin are forwarded to their respective Target Groups.

This allows multiple domains and independent services to continue operating separately while sharing the same ingress layer.

During the migration, we first created the new forwarding rules and Target Groups and confirmed that the targets were healthy. Only then did we update DNS.

After verifying all domains, certificates, and application entry points, we removed the old ALBs and their associated resources.

We also evaluated Cloudflare Tunnel, but it would have introduced new account permissions, ingress dependencies, and troubleshooting workflows. For this system, retaining AWS-native ALB infrastructure was the more predictable choice.

The most important failure mode during this type of migration is a 503. Validation therefore cannot stop at confirming that the ALB itself is marked as Active.

We also verified that:

• Listener Rules routed to the correct Target Groups;
• Health check paths returned successful responses;
• DNS had propagated to the new ingress layer.

The old ALBs remained available until migration validation was complete, preserving a clear rollback path if anything went wrong.

Use Monitoring Data to Guide Fargate Downsizing

The Production API and Web services were originally configured conservatively.

Monitoring data from the 24 hours before downsizing showed:

• API CPU averaged approximately 2.60%, with a peak of 5.86%;
• Web CPU averaged approximately 2.36%, with a peak of 5.60%;
• Peak memory utilization remained below 4%.

We gradually reduced both the API and Web services to 0.25 vCPU / 0.5 GB. OpenAPI and SuperAdmin were already running at the minimum task size and were left unchanged.

Each change used an ECS rolling deployment: new tasks were started first, ALB health checks were allowed to pass, and only then were the old tasks drained.

Monitoring during the 24 hours after downsizing showed:

• API CPU averaged approximately 4.84%, with a peak of 12.88%;
• Web CPU averaged approximately 5.34%, with a peak of 17.28%;
• Peak memory utilization for both remained below 7%.

Production ultimately retained four online tasks at the minimum specification, with estimated monthly Fargate costs of approximately $36.

Idle Workers were not deleted. Instead, they were configured with desired=0, allowing them to be restored quickly when needed.

Reducing the task specifications did not change the container images, network interfaces, or deployment model.

If traffic increases in the future, the first scaling option is simply to increase the desired count. If individual requests begin requiring substantially more CPU or memory, a larger Task Definition can then be deployed.

Keeping that scaling path intact was one of the prerequisites for reducing always-on capacity.

There is, however, a clear trade-off. A single task does not provide high availability in the traditional multi-replica sense. ECS will automatically replace a failed task, but the service may experience a brief interruption during recovery.

As traffic, SLA requirements, or business criticality increase, restoring multiple replicas for critical services should be one of the first infrastructure changes.

The Reader Is Valuable, but Is It Worth Keeping Always On?

The Production Aurora cluster originally had one Writer and one Reader.

The Writer handles writes and transactions and can also serve reads. The Reader uses the same Aurora distributed cluster storage but runs as a separate database instance for read-only queries.

It can reduce read pressure on the Writer and can also be promoted if the Writer fails, reducing recovery time.

After examining application connections, database sessions, and actual request paths, we confirmed that the Reader was receiving real production reads. It was not an idle resource.

The actual question was:

Does the current read volume and failover benefit justify the cost of keeping another database instance running continuously?

Based on the overall workload and connection volume, the Reader was offloading only a limited amount of traffic, and the Writer still had sufficient capacity to absorb those reads.

Before deleting the Reader, we safely redirected read connections back to the Writer, performed a rolling application restart, and validated the read path.

We then queried PostgreSQL’s pg_stat_activity and confirmed that only the internal rdsadmin connection remained on the Reader. Once all business traffic had drained, the Reader was removed.

Production now retains a single Serverless v2 Writer with a minimum capacity of 0.5 ACU.

Based on the current pricing model, removing the Reader is expected to save approximately $43.20–$44.64 per month.

The trade-off is losing a database instance that can immediately take over during a Writer failure. Aurora storage remains distributed across Availability Zones, but recovery from a Writer failure may take longer without an existing Reader available for promotion.

If read pressure or recovery requirements increase in the future, a Reader can be recreated and read/write separation restored.

This database change also required careful Terraform state management.

The original configuration used count to manage the two database instances. Simply changing the value from 2 to 1 could cause Terraform to interpret array index changes as a request to replace the wrong instance.

We therefore migrated to stable for_each identifiers and used a moved block to preserve the Writer’s resource identity.

Only after confirming that the Terraform Plan would delete the intended Reader—and nothing else—did we apply the change.

Make Staging Truly On-Demand

Staging was originally configured to start and stop automatically on weekdays.

However, the test environment is not used every day, so scheduled startups still generated unnecessary compute costs.

We ultimately disabled automatic startup, set all four Staging ECS services to a default desired count of 0, and configured Aurora with a minimum capacity of 0 ACU, automatically pausing after 15 minutes of inactivity.

We also created a unified script supporting:

• start
• status
• stop

This allows the entire Staging environment to be managed through a single command.

After completing the script, we ran a full recovery drill covering startup, health checks, endpoint validation, and shutdown. All four public application endpoints passed validation.

An on-demand environment is only reliable if it has been proven capable of starting successfully from zero.

For routine testing, we now run:

staging-on-demand.sh start

After the services and database have resumed, testing can begin. When testing is complete:

staging-on-demand.sh stop

This is more reliable than manually modifying individual services in the AWS Console and makes it less likely that someone will forget to shut Staging down after testing.

Before Removing NAT, Prove the System Does Not Depend on It

A NAT Gateway typically provides private subnets with outbound access to the public internet. AWS charges both an hourly fee and a data processing fee for traffic passing through it.

During the billing cycle before optimization, the two NAT Gateways generated approximately $77.38 in hourly charges and $58.34 in data processing fees.

We migrated ECS Fargate tasks to public subnets and assigned them public IP addresses, while keeping inbound access tightly restricted.

The task security groups do not allow inbound connections from public CIDR ranges. Application ports can only be accessed from the ALB Security Group.

The database and Valkey remain in private networking.

We also added free S3 Gateway Endpoints to the private route tables in both VPCs, reducing private-network dependency on NAT.

Running a task in a public subnet does not mean its application ports are directly exposed to the internet.

The public IP provides an outbound path for tasks to reach services such as ECR and Secrets Manager. Inbound access is still governed by the security group.

During final verification, task security groups contained no 0.0.0.0/0 or public IPv6 inbound rules, and application ports were reachable only from the ALB Security Group.

Before deleting the NAT Gateways, we verified:

• Container image pulls;
• Secrets access;
• CloudWatch Logs;
• Database connectivity;
• Valkey connectivity;
• ALB health checks;
• Production application domains.

Staging successfully performed a cold start from zero without NAT, and Production successfully rolled out a complete new set of tasks.

Only after confirming that the application no longer depended on NAT did we remove both NAT Gateways and their dedicated Elastic IP addresses.

After accounting for the additional public IPv4 charges for Fargate tasks, the expected reduction in fixed costs is approximately $60–$70 per month.

If traffic remains similar to the previous billing cycle, the associated NAT data processing charges should also largely disappear.

This design is appropriate for the system’s current security requirements. If future requirements include fixed egress IP addresses, strict private-network compliance, or more sophisticated network controls, NAT Gateway or PrivateLink should be reconsidered.

Address Other Fixed Costs

Container Insights has been disabled for both Production ECS clusters, while standard ECS metrics and application logs remain available.

CloudWatch cost $57.10 during the pre-optimization billing cycle. However, the actual reduction from disabling Container Insights can only be confirmed after a complete billing cycle, so we are not treating the estimated reduction as realized savings yet.

The Bastion instance is now started only when needed and releases its public IP when stopped, with estimated savings of approximately $7 per month.

The inactive Worker is expected to save approximately $8–$10 per month.

The two Valkey Serverless instances remain in place. They are already near the minimum storage billing level of approximately 0.1 GB, while their combined ECPU charges are less than $0.01.

Reducing this cost further would require deleting the services entirely, which does not align with current operational requirements.

Change Only One Cost Point at a Time

The entire optimization followed the same validation workflow:

Review current state → Modify one cost point → Roll out the change → Check health → Request the real domain → Review logs and metrics → Remove legacy resources.

If an unexpected replacement or deletion appeared in a Terraform Plan, execution stopped immediately until the resource identity or lifecycle configuration was corrected.

For example, after the Bastion instance was stopped, a state discrepancy caused Terraform to plan an unnecessary recreation of the instance. After correcting the lifecycle configuration and running the Plan again, the changes returned to the expected scope.

Once optimization was complete, Terraform Plan was executed separately for Production and Staging using the correct workspaces and variable files.

Both ultimately returned:

No changes

Final Cost: Optimized Monthly Cost Estimate

The final infrastructure review showed:

• One shared ALB remains in each environment;
• There are no remaining NAT Gateways;
• Production runs four minimum-size online tasks;
• One Worker remains configured but stopped;
• Production retains one Aurora Writer;
• All Staging ECS services are stopped by default;
• Staging Aurora can automatically pause when idle.

When Staging is rarely activated and builds are infrequent, the estimated monthly cost is approximately $180–$200.

During normal low-load operation, the expected range is approximately $190–$220 per month.

If builds are frequent or Staging is used extensively, monthly costs may rise to approximately $215–$245.

Compared with the full billing cycle before optimization, a typical low-load month is expected to cost approximately 73%–77% less.

The month in which the optimization was performed includes both pre-optimization and post-optimization resource charges, and Cost Explorer data may also be delayed. The actual impact should therefore be evaluated against the next complete billing cycle.

Summary

The core of this optimization was not to reduce every configuration to the absolute minimum. It was to realign infrastructure costs with the system’s current workload while preserving the ability to scale later.

Ingress can be shared. Compute capacity can be reduced based on monitoring data. Test environments can run on demand. Whether an Aurora Reader should remain online depends on both read pressure and recovery objectives. Network egress infrastructure should only be removed after its dependencies have been proven unnecessary.

Five principles are particularly reusable:

1. Review the bill first, then the monitoring data;
2. Prioritize fixed costs that are billed hourly;
3. Migrate and validate first, then remove legacy resources;
4. Use rolling deployments in Production;
5. Maintain a clear scaling and recovery path for every reduction in capacity.

Moving from approximately $823 per month to an estimated $180–$220 represents a potential reduction of roughly three-quarters.

More importantly, we now understand why each major cost exists—and which capabilities should be restored first as the business grows.

Data sources and methodology: Costs for the full billing cycle before optimization were sourced from AWS Cost Explorer using UnblendedCost. Resource states, task specifications, and security group configurations were verified through read-only AWS API calls after the optimization was completed. Performance metrics were based on five-minute CloudWatch sampling intervals. Monthly projections use publicly available On-Demand pricing and approximately 730 hours per month. Estimates exclude potential taxes, AWS Support fees, discounts, credits, and unexpected traffic spikes. Project names, domains, account identifiers, and resource identifiers have been anonymized. These estimates do not constitute an official AWS quote.