Kit spent over $1 million on AWS in 2021
Overview
We spent $1,065,754.35 on AWS in 2021. This is up 12.5% from 2020 and was 3.9% of net revenue for the year. We increased our spending on EC2-Instances by 96% from January to December because we scaled out our Cassandra and Elasticsearch clusters. We also made investments in our data pipeline and began migrating customers from FanBridge to Kit. Despite the 96% increase on EC2-Instances, our EC2-Other spend only increased by 49.5% because we made our regional data transfer more efficient. Additionally, we reduced our S3 costs by 33% by improving our CDN usage and optimizing our backup process.
High-level breakdown:
- EC2-Instances - $348,266.05
- Relational Database Service - $242,178.24
- EC2-Other - $147,318.75
- Savings Plan For Compute Usage - $97,732.48
- S3 - $74,010.44
- Support - $65,565.37
- EC2-ELB - $37,714.76
- Redshift - $21,421.36
- Everything Else - $32,388.12
Total Monthly Spend
Our total monthly bill increased by 44% from January to December. The leading causes of the increase came from spending more on EC2-Instances and Redshift. We also had a considerable rise in Relational Database Service spend for November and December due to mandatory maintenance we had to perform on our primary MySQL server. I’ll dig into that more later.
Our September bill was 36% cheaper than August and 38% cheaper than October. The decrease came from AWS credits that we qualified for by working with our AWS account rep.
EC2-Instances - $348,266.05
Our EC2 spending increased by 96% from January to December. The biggest increase (18.3%) came between November ($38,636.85) and December ($45,730.12). There are a few reasons for such a large increase.
First, these are the busiest months because of Black Friday and Cyber Monday. We sent 85.1 million emails on Black Friday, and on Cyber Monday, we sent 93.9 million emails. We increased capacity across our infrastructure to ensure a smooth holiday season for both our users and our on-call engineers and support.
Second, Kit acquired FanBridge in April 2021 and ramped up our migration process during the second half of the year.
Thirdly, we increased the capacity of our multi-region Cassandra cluster and our Elasticsearch clusters.
Lastly, our largest Savings Plan For Compute Usage reservation expired in December, and we didn’t replace it until 2022. Forgetting to replace that reservation was an oversight on our part.
Relational Database Service - $242,178.24
Our RDS bill is pretty shocking to see. If you remove December, our average monthly spend was $19,155.60, which is where it’s been since August 2019.
What happened in December?
AWS gave us notice in late November that we needed to do a mandatory OS update for security reasons. We had until the end of January 2022 to do it ourselves, or they would do it automatically. Keeping things secure is extremely important, but so is ensuring uptime, data integrity, and stability. Our MySQL database is 9TB+, and the default maintenance option given by RDS is too unpredictable to be reliable. Instead, we opted to create a brand new MySQL server with the OS update already installed as a read-replica. We could then promote the read-replica to primary at our leisure with minimal downtime. Because RDS lazy loads data into read-replicas, we had to launch it at the beginning of December and give it time to catch up to the primary. We effectively doubled our RDS cost by keeping two primary-sized RDS instances running for the entire month of December.
Ultimately, it was worth it because we performed the maintenance on December 28 at 1 am EST, and the entire process took 30 minutes and went off without a hitch.
EC2-Other - $147,318.75
Most of our EC2-Other comes from data transfer. As I mentioned before, Q4 is a busier time for our infrastructure than the rest of the year. Combining that with the FanBridge migration means we naturally spent more at the end of the year than we did at the beginning of the year.
We do use the AWS-managed NAT gateway. It is expensive, but it’s also reliable, and it only came out to 3.5% ($36,829.50) of what we spent on AWS in 2021. For an infrastructure team of two, the reliability and peace of mind are worth the cost.
Savings Plan For Compute Usage - $97,732.48
We saved $35,384 in 2021 from purchasing our savings plans.
The chart is trending down because we purchase multiple savings plans, and they expire throughout the year. The way they trend down correlates with our EC2-Instances spend trending up. The worst offender is December. Don’t forget to renew your savings plans, or it will cost you.
Also, we didn’t pay anything in September because of the credits we received.
S3 - $74,010.44
At the end of 2020, we made improvements to our backup process that reduced our S3 bill by 47%. The gradual increase afterward comes from the expected growth of our product usage. We use S3 for email attachments, landing pages, commerce, backups, and many other parts of the application. Paying more for S3 is a positive sign that more people are using more of our products.
Support - $65,565.37
The cost of support changes with usage. Because our AWS cost increased month-over-month, we spent more on support as we spent more on everything else. The exception here was September when we got AWS credits.
EC2-ELB - $37,714.76
Our ELB costs increased by 12.5% from January to December. Given that Kit grew MRR by 12.8% in 2021, it’s possible that the number of new accounts contributing to the MRR growth also contributed to the additional connections and bandwidth that increased our load balancer costs. We primarily use ELB for our web application. As other AWS bill posts mentioned, we expect this cost to grow as more people use our service.
Redshift - $21,421.36
Our analytics team has been using Redshift for years, but 2021 was when we invested a lot more into our data and analytics pipeline. Improving our data flows meant we started ingesting more data and moved to more powerful hardware. We migrated off our old Redshift cluster in July and August. We ran two Redshift clusters simultaneously during that migration, which is why August has such a big spike.
Everything Else - $32,388.12
The one hump you’ll notice between March and May came from introducing the Managed Streaming for Apache Kafka service that AWS offers. We used Kafka for a project but ultimately scrapped it when we decided to go in a different direction. Had we decided to continue with that project, it would have been about $1,100 per month for the managed Kafka cluster. Everything else we spent on AWS was mostly flat.
Conclusion
Kit is a bootstrapped company. We have no outside funding, so keeping our cloud costs under control is vital. We made a few mistakes throughout the year but fortunately, they weren’t anything we couldn’t correct. We do our best to be conscious of how much we’re spending while avoiding being cheap. Finding efficiencies wherever we can while being willing to spend money where it makes sense is a continuous balancing act that we work on every day.