$72,706 spent on AWS in December - A full breakdown of Kit's AWS bill

generalengineeringaws
$72,706 spent on AWS in December - A full breakdown of Kit's AWS bill
Kris Hamoud
Kris Hamoud is an Infrastructure Engineer who enjoys building simple and scalable solutions • Kris's website

Overview

We spent $72,706.88 on AWS in December. This is up 7% from November and is 4.4% of MRR in December. In December, we increased our spend on RDS, data transfer, and purchased an AWS Savings Plan.

High-level breakdown:

  1. EC2-Instances - $21,817.60 (-18%)
  2. Relational Database Service - $21,491.51 (+10%)
  3. EC2-Other - $7,321.81 (+25%)
  4. S3 - $7,159.65 (0%)
  5. Support - $4,383.84 (+18%)
  6. Savings Plan For Compute Usage - $4,454.40 (+100%)
  7. EC2-ELB - $2,112.00 (+23%)

EC2-Instances - $21,817.60 (-18%)

We purchased a Savings Plan For Compute Usage a few days into December. This will allow us more freedom and flexibility with how we scale our compute resources. We increased the size of our Elastic Stack, which caused an increase in our USE2-HeavyUsage:i3.2xlarge costs, but we’ve gotten to a place in January where we can maintain the costs of our Elastic Stack with less work than we were investing into it in the previous months. Overall our EC2 costs should either decrease or stay flat for the immediate future.

Service breakdown

  1. USE2-HeavyUsage:i3.2xlarge - $8,870.71 (+14%)
    • These are our reserved Cassandra and Elasticsearch clusters.
    • We use Cassandra to store massive amounts of data.
    • We use Elasticsearch to search through massive amounts of data and to store our logs.
    • The increase comes from additional reserved instances we purchased for our Elastic Stack.
    • We stopped adding these instance types to our Elastic Stack in favor of the storage optimized i3en.2xlarge instances that come with 5TB total SSD storage.
  2. HeavyUsage:i3.2xlarge - $2,865.89 (+3%)
    • These are reserved Cassandra instances.
    • We use them for our secondary Cassandra cluster.
    • The price is higher because December is longer than November.
  3. USE2-BoxUsage:c5.2xlarge - $1,708.94 (-55%)
    • These are on-demand instances.
    • The decrease comes from our Savings Plan For Compute Usage USE2-BoxUsage:c5.2xlarge
  4. USE2-HeavyUsage:c5.2xlarge - $1,928.45 (+3%)
    • These are reserved instances.
    • We use these for our web servers.
    • We’ll pay this much until Q3 2020.
    • The price is higher because December is longer than November.
  5. USE2-DataTransfer-Out-Bytes - $1,626.82 (-4%)
    • This is the cost of our services to communicate with the internet.
    • We saw some billing wins here because we migrated away from our old logging provider.
    • Christmas had less data transfer than Cyber Monday or Black Friday, which had an impact on how much we spent. USE2-DataTransfer-Out-Bytes
  6. USE2-BoxUsage:c5.xlarge - $835.25 (-42%)
    • These are on-demand instances.
    • We use these instances for compute-heavy services such as email sending and event tracking. USE2-BoxUsage:c5.xlarge
  7. USE2-BoxUsage:t3.medium - $529.25 (-60%)
    • These are on-demand instances.
    • We use these instances for everything from email tracking to Elasticsearch indexing.
    • Despite their high usage we move around a lot of instance types in December which had an impact on the bill.
    • The decrease also comes from our Savings Plan For Compute Usage USE2-BoxUsage:t3.medium
  8. USE2-BoxUsage:t3.xlarge - $416.71 (-68%)
    • These are on-demand instances.
    • We use these instances for a variety of jobs that rely on burstable CPU.
    • The decrease also comes from our Savings Plan For Compute Usage USE2-BoxUsage:t3.xlarge
  9. USE2-BoxUsage:t3.large - $320.30 (-65%)
    • These are on-demand instances.
    • We use these for a variety of different jobs.
    • The decrease also comes from our Savings Plan For Compute Usage USE2-BoxUsage:t3.large

Relational Database Service - $21,491.51 (+10%)

We added another database in December to allow our analytics and engineering teams to do some data migrations and reporting. The extra DB was necessary to keep our main application healthy. Unfortunately, we started running backups on that database, which is unnecessary because it is only a snapshot of our production database and will be terminated in the future anyway. These backups cost us nearly $700.

Service breakdown

  1. USE2-HeavyUsage:db.r5.12xl - $4,949.68 (+3%)
    • This instance is reserved.
    • This is our master MySQL database.
    • We’ll continue to pay this much until Q3 2020.
  2. RDS:ChargedBackupUsage - $4,425.70 (+26%)
    • These are our disaster recovery backups.
    • We take additional backups and send them to a different region in case of emergencies.
    • The increase comes from the extra backups being taken by the temporary database we’re running. We turned off the unnecessary backups in January. RDS:ChargedBackupUsage
  3. USE2-InstanceUsage:db.r4.8xlarge - $2,856.96 (+3%)
    • This is an on-demand instance.
    • This replica is being kept around because we will need it to maintain a healthy application for the foreseeable future.
    • The price is higher because December is longer than November.
  4. USE2-RDS:ChargedBackupUsage - $2,251.27 (-11%)
    • These are our normal backups.
    • The decrease is because we started running backups on the temporary database. Those backups aren’t useful, though, because they’re only accurate up to the date the database was created. It’s unfortunate, but we’ve fixed the problem. USE2-RDS:ChargedBackupUsage
  5. USE2-RDS:Multi-AZ-GP2-Storage - $2,119.22 (+16%)
    • These are daily charges.
    • The cost increased because we added storage in November in preparation for Black Friday and Cyber Monday and December was the first full month feeling those increases.
  6. USE2-RDS:GP2-Storage - $1,811.02 (+37%)
    • This is the cost of our storage.
    • It increased because we added storage to prepare for Black Friday and Cyber Monday.
    • It also increased because we launched the temporary database. USE2-RDS:GP2-Storage
  7. USE2-HeavyUsage:db.r4.8xlarge - $1,649.89 (+3%)
    • This instance is reserved.
    • This is our MySQL replica.
    • We’ll continue to pay this much until Q3 2020.
    • The price is higher because December is longer than November.

EC2-Other - $7,321.81 (+25%)

The majority of this bill is our inter-region data transfer. We shipped something on December 9th that increased the number of logs we write. This change effectively doubled the cost of our data transfer bill.

Service breakdown

  1. USE2-DataTransfer-Regional-Bytes - $3,262.07 (+64%)
    • This is the cost of replicating data in our data stores across AWS regions.
    • The cost increased here because of a logging change we shipped on December 9th. USE2-DataTransfer-Regional-Bytes
  2. USE2-NatGateway-Bytes - $1,677.95 (+7%)
    • We use a NAT gateway for our services to communicate with the internet.
    • We can expect this cost to drop in the future by a small amount because we fully migrated off of our old logging provider.
  3. USE2-EBS:VolumeUsage.gp2 - $1,367.89 (0%)
    • This is the cost of having gp2 disks connected to our instances.
    • The number of these was flat for all of December.

S3 - $7,159.65 (0%)

We continue to see wins here from our data transfer costs as more and more content is served from behind our Cloudflare account. We remained flat because the amount of stuff we’re storing in S3 is increasing. I’m thrilled with the way this cost is headed. We can adjust our timed storage needs as necessary, but the data transfer portion of this bill used to be out of our control. Now it’s under our control, and it’s headed in the right direction.

Service breakdown

  1. DataTransfer-Out-Bytes - $2,030.44 (-4%)
    • We continue to see billing wins here as the number of objects being served straight from S3 decreases in favor of our Cloudflare CDN.
    • The slope continues downward and I’m not sure how far it will go, but I’m excited to find out. DataTransfer-Out-Bytes
  2. USE2-TimedStorage-ByteHrs - $1,717.98 (+6%)
    • This is the steady growth of our backups coming from our Cassandra and Elasticsearch clusters.
    • This will continue to grow as the amount of data we store grows.
    • We can probably find wins here by changing our storage class to infrequent access.
  3. USE2-DataTransfer-Out-Bytes - $1,495.72 (-4%)
    • We served more data from our Cloudflare CDN here than I was anticipating.
    • Aside from a couple of days at the end of the month, our daily data transfer was lower than most of November.
  4. TimedStorage-ByteHrs - $1,131.64 (0%)
    • This increase comes from our secondary Cassandra cluster.
    • It will continue to grow as the amount of data we store grows.

Support - $4,383.84 (+18%)

Service breakdown

  1. 7% of monthly AWS usage from $10K-$80K - $3,342.18 (+11%)
    • This is the cost of only our production account.
  2. 10% of monthly AWS usage for the first $0-$10K - $1,809.27 (+32%)
    • This is the cost of our production account and billing account.
    • We could save money by turning off support for our billing account.

Savings Plan For Compute Usage - $4,454.40 (New)

This is the biggest change we made in December. We purchased a Savings Plan For Compute Usage and committed to spending $9.60/hour for the next year on AWS instances. We saved $1,685.00 in December from purchasing this. Savings Plan For Compute Usage

EC2-ELB - $2,112.00 (+23%)

We use load balancers for many different parts of our application. The cost has been increasing steadily over the past few months as we add more autoscaling groups to our infrastructure, handle more requests to our Free Landing Pages, and increase our log volume. We can find wins here in the future as we migrate to Kubernetes and can rely on the internal networking of Kubernetes rather than AWS resources.

Service breakdown

  1. USE2-DataTransfer-Out-Bytes - $985.29 (+6%)
    • This is the cost of data going out to the internet from behind our load balancers.
    • This increase comes from the fact we had more data out on Cyber Monday than we did on Black Friday.
  2. USE2-LoadBalancerUsage - $559.51 (+27%)
    • This cost comes from the number of connections to our load balancers and their overall usage.
    • This cost is steadily increasing as we handle more traffic to our landing pages. USE2-LoadBalancerUsage
  3. USE2-LCUUsage - $539.21 (+69%)
    • This increase comes with the EC2 data transfer increase we saw on December 9th.
    • Not only are we paying more for the logs being sent around the VPC, but we’re also paying for that same amount of data to be handled by the load balancer sitting in front of our logstash instances. USE2-LCUUsage

Conclusion

We’re saving money on on-demand instances from our Savings Plan For Compute Usage, but our Elastic Stack cost has increased from an EC2 storage standpoint as well as a data transfer standpoint. Our current Elastic Stack is about 26TB. Storing that much data is expensive and transferring it around our infrastructure is also expensive.

The addition of a temporary database added about $1000 to our bill in 15 days, and we can expect it to add about $2000 to our bill in January as it will exist for the full month rather than the last 15 days.

Our load balancer costs have increased, but we have an exit strategy for this in Q1 of 2020. There will likely be hidden fees in the migration, but at a minimum, it should be cheaper than what we are currently paying for load balancers.

We have a lot of work cut out for us in 2020 to decrease our AWS costs, but making sure we identify each line item every month has been really helpful in finding and fixing any bill that is getting out of control.