$72,732.36 spent on AWS in February - A full breakdown of Kit's AWS bill

generalengineeringaws
$72,732.36 spent on AWS in February - A full breakdown of Kit's AWS bill
Kris Hamoud
Kris Hamoud is an Infrastructure Engineer who enjoys building simple and scalable solutions • Kris's website

Overview

We spent $72,732.36 on AWS in February. This is down 5% from January and is 4.3% of MRR in February. Despite being a leap year, February is still the shortest month of the year, and that gets reflected in this AWS bill. We saw a decrease across most line items, and that is attributed to it being a short month.

High-level breakdown:

  1. Relational Database Service - $19,295.95 (-10%)
  2. EC2-Instances - $19,264.28 (0%)
  3. EC2-Other - $10,275.86 (-1%)
  4. S3 - $7,005.97 (-6%)
  5. Savings Plans for Compute usage - $6,681.60 (-6%)
  6. Support - $4,912.42 (-4%)
  7. EC2-ELB - $2,195.73 (-7%)

Relational Database Service - $19,295.95 (-10%)

February was a pretty boring month from an RDS perspective. We felt a full month of savings from the temporary database that we deleted in late January. We did some minor maintenance by adding a terabyte of storage to our master MySQL database and two replicas.

  1. USE2-HeavyUsage:db.r5.12xl - $4,630.35 (-6%)
    • This instance is reserved.
    • This is our master MySQL database.
    • It’s cheaper because February is shorter than January.
    • We’ll continue to pay this much until Q3 2020.
  2. RDS:ChargedBackupUsage - $3,896.11 (-15%)
    • These are our disaster recovery backups.
    • We take additional backups and send them to a different region in case of emergencies.
    • The decrease comes from removing a database and fixing a configuration mistake we made in January. RDS:ChargedBackupUsage
  3. USE2-InstanceUsage:db.r4.8xlarge - $2,672.64 (-6%)
    • This is an on-demand instance.
    • This replica is being kept around to maintain a healthy application.
    • The price is lower because February is shorter than January.
  4. USE2-RDS:Multi-AZ-GP2-Storage - $2,119.22 (0%)
    • These are daily charges.
    • We added storage to our database to accommodate our growth which offset the length of the month.
  5. USE2-RDS:ChargedBackupUsage - $1,665.00 (-2%)
    • These are our normal backups.
    • The decrease is because February is shorter than January.
  6. USE2-HeavyUsage:db.r4.8xlarge - $1,543.45 (-6%)
    • This instance is reserved.
    • This is our MySQL replica.
    • We’ll continue to pay this much until Q3 2020.
    • The price is lower because February is shorter than January.
  7. USE2-RDS:GP2-Storage - $1,499.60 (-25%)
    • This is the cost of our storage.
    • The decrease comes from deleting the temporary database we were running in January.
    • It’s also cheaper because February is shorter than January.

EC2-Instances - $19,264.28 (0%)

We expanded our Elastic Stack again and added an i3en.2xlarge to handle an increase in logs. We also needed to double the number of on-demand c5.2xlarge instances as we migrate our infrastructure to Kubernetes. We’ll see some billing wins on EC2 costs as we settle into our new Kubernetes infrastructure.

  1. USE2-HeavyUsage:i3.2xlarge - $8,340.86 (-6%)
    • These are our reserved Cassandra and Elasticsearch clusters.
    • We use Cassandra to store massive amounts of data.
    • We use Elasticsearch to search through massive amounts of data and to store our logs.
    • It’s cheaper because February is shorter than January.
  2. HeavyUsage:i3.2xlarge - $2,680.99 (-6%)
    • These are reserved Cassandra instances.
    • We use them for our secondary Cassandra cluster.
    • The price is lower because February is shorter than January.
  3. USE2-BoxUsage:c5.2xlarge - $2,481.19 (+84%)
    • These are on-demand instances.
    • We needed to scale out these instances for our email sending service and web applications as we migrate to Kubernetes. USE2-BoxUsage:c5.2xlarge
  4. USE2-HeavyUsage:c5.2xlarge - $1,804.03 (-6%)
    • These are reserved instances.
    • We use these for our web servers.
    • We’ll pay this much until Q3 2020.
    • The price is lower because February is shorter than January.
  5. USE2-DataTransfer-Out-Bytes - $1,384.29 (-15%)
    • This is the cost of our services to communicate with the internet.
    • Our daily usage decreased slightly in February. USE2-BoxUsage:c5.2xlarge
  6. USE2-HeavyUsage:i3en.2xlarge - $1,316.90 (+6%)
    • These are reserved instances.
    • We use these for our Elastic Stack.
    • They are more efficient for storing data than the i3.2xlarge, which is why we moved our Elastic Stack data to i3en.2xlarge instances.

EC2-Other - $10,275.86 (-1%)

Data transfer is something we’ve been fighting for a long time. We’re getting better at preventing data from moving too much, but sometimes it can be hard to predict where the data leaks will occur. When we were storing our Logs in i3.2xlarge instances, we didn’t realize how much data there would be. We added more i3.2xlarge instances, but the costs started to add up. We switched to the i3en.2xlarge because they are cheaper per gig of storage. We had reserved the i3.2xlarge instances, and we had no way to get rid of them, so we ran both i3.2xlarge and i3en.2xlarge together. Running both caused a problem where the Elasticsearch shards would always be rebalancing. The shard rebalancing resulted in huge amounts of data transferred around our VPC.

  1. USE2-DataTransfer-Regional-Bytes - $6,743.19 (+6%)
    • This is the cost of replicating data in our data stores across AWS availability zones…
    • The cost increased throughout February due to the imbalance in our Elasticsearch nodes.
    • The Elastic Stack was always rebalancing 100GB shards through February.
    • We can expect this to decrease in the future because our Elasticsearch nodes are all the same now.
  2. USE2-EBS:VolumeUsage.gp2 - $1,394.79 (-5%)
    • This is the cost of having gp2 disks connected to our instances.
    • The number of these was flat for most of February but February is shorter than January, so the bill is cheaper.
  3. USE2-NatGateway-Bytes - $1,081.83 (-21%)
    • We use a NAT gateway for our services to communicate with the internet.
    • The decrease comes from taking steps towards rebalancing our Elastic Stack. USE2-NatGateway-Bytes

S3 - $7,005.97 (-6%)

It was a pretty exciting month for S3. For the first time ever we spent more on our backups than we did on data-transfer-out. We’re down 73% from last year for our DataTransfer-Out-Bytes bill. We’re also down 70% from our peak laster year for USE2-DataTransfer-Out-Bytes.

  1. USE2-TimedStorage-ByteHrs - $2,049.69 (+11%)
    • This is the growth of our backups coming from our Cassandra and Elasticsearch clusters.
    • This will continue to grow as the amount of data we store grows.
    • We can find wins here by changing our storage class to infrequent access.
  2. DataTransfer-Out-Bytes - $1,532.81 (-25%)
    • We continue to see billing wins here as the number of objects served straight from S3 decreases in favor of our Cloudflare CDN. USE2-NatGateway-Bytes
  3. USE2-DataTransfer-Out-Bytes - $1,521.29 (-8%)
    • We served more data from our Cloudflare CDN, which reduced our data transfer.
    • February is shorter than January, which accounts for most of the savings here.
  4. TimedStorage-ByteHrs - $1,091.73 (-1%)
    • This is our backups we keep in a separate region.
    • February is shorter than January, but the amount of data we backed up increased, which made this bill flat.

Savings Plans for Compute usage - $6,681.60 (-6%)

We saved $2,529.00 in February from purchasing this. Savings Plan For Compute Usage

Support - $4,912.42 (-4%)

  1. 7% of monthly AWS usage from $10K-$80K - $2,953.61 (-7%)
    • This is the cost of only our production account.
  2. 10% of monthly AWS usage for the first $0-$10K - $1,958.81 (+11%)
    • This is the cost of our production account and billing account.
    • We could save money by turning off support for our billing account.

EC2-ELB - $2,195.73 (-7%)

We’re working on decreasing our reliance on AWS load balancers. As more of our services migrate to Kubernetes, the number of AWS load balancers that we need will decrease. A decrease in load balancers decreases our costs here.

  1. USE2-DataTransfer-Out-Bytes - $914.62 (-14%)
    • This is the cost of data going out to the internet from behind our load balancers.
    • The decrease comes from smarter routing of our data from behind the load balancer.
  2. USE2-LCUUsage - $657.40 (+5%)
    • This cost comes from the number of connections to our load balancers and their overall usage.
    • This cost is steadily increasing as we handle more traffic to our landing pages.
  3. USE2-LoadBalancerUsage - $553.64 (-12%)
    • This increase comes with the EC2 data transfer increase we saw beginning in December.
    • We’ll see wins here in the future as we migrate more of our workload to Kubernetes.

Conclusion

There is a lot to get excited for while looking at this bill. We know that we’ll need to scale out our EC2 and RDS usage as our company grows. Controlling the growth of our compute costs is something we’ll have to do forever, but we must keep track of our data transfer costs. Looking at where we were a year ago and where we are now makes me excited because it shows how far we’ve come at controlling our data costs.

Here are the last 12 months of data transfer out from S3. S3 Data Transfer - One Year