Kit spent over $1.4 million on AWS in 2022
Overview
We spent $1,471,169.38 on AWS in 2022. This is up 38% from 2021 and was 4.7% of net revenue for the year. We increased spending on every major AWS service that we use. We spend the most on EC2-Instances and Relational Database Service. Those two services account for 55% of our annual spending. If we include EC2-Other and Savings Plan For Compute Usage, which are unavoidable accompanying expenses to EC2-Instances, those four services account for 76.2% ($1,120,380) of our annual spending. Most of our EC2 spend goes toward our Elasticsearch and Cassandra clusters. We use RDS for our primary MySQL database and smaller databases that power our small services. You’ll notice we’re spending significantly more on Redshift. As I mentioned last year, we’re investing heavily in improving our data and analytics pipeline, and having a better Redshift cluster was our first significant change.
Below is the high-level breakdown of our spending. I’ll go over each service and detail how we use it.
High-level breakdown:
- EC2-Instances - $480,740.30 +38% YoY
- Relational Database Service - $328,634.92 +35.7% YoY
- EC2-Other - $195,019.31 +32.4% YoY
- Savings Plan For Compute Usage - $115,985.76 +18.7% YoY
- Support - $90,990.09 +38.8% YoY
- S3 - $88,725.09 +19.9% YoY
- EC2-ELB - $43,272.45 +14.7% YoY
- Redshift - $43,165.27 +101.5% YoY
- Everything Else - $84,636.18 +161.3% YoY
Total Monthly Spend
Our total monthly bill increased by 12.8% from January to December. The leading cause of the increase comes from spending more on many smaller services. Despite spending more on every service year-over-year, the individual services’ monthly costs remained flat throughout 2022. You’ll also notice that February is significantly lower than the rest of the months.
There are two reasons:
- February is the shortest month
- We got the second tranche of our AWS credits from our account rep
You’ll see the effects of the credits throughout the breakdown.
EC2-Instances - $480,740.30 +38% YoY
Our EC2 spending increased by +1.9% from January to December. You’ll notice again the effects of the AWS credits we received in February. You’ll also see that our EC2 spending decreased from January to May. This year we worked to reduce the number of on-demand instances we use. As of December 2022, only 1.3% of our EC2 spend is on-demand. Instance reservations, savings plans, or spot instances cover the remaining 98.7%. This improvement is how we reduced EC2 spending while increasing the number of emails we send every month.
However, EC2 costs increased again in June and continued to do so until October because we had to scale out our Elasticsearch and Cassandra clusters multiple times this year.
We use the i3 and i4 instance families to power our Elasticsearch and Cassandra clusters. We don’t use them for anything else, so it’s easy to get a basic idea of how much we spend on those data stores compared to everything else in our fleet. One of our goals for 2023 is to find ways to optimize how much data we store in these databases to help us curb their growth rates. Storage optimization and the use of cold storage is one of the highest leverage things we can do to ensure a healthy and predictable AWS bill.
Relational Database Service - $328,634.92 +35.7% YoY
Kit has a love/hate relationship with RDS. In 2019 we spent over $45,000/month on one MySQL database. After years of effort, we’ve gotten ourselves to a stable and predictable RDS spend. RDS is an excellent service for small teams like Kit because it reduces operational complexity in most cases. However, you can see that we have a few months where the cost of RDS increased substantially. That happens every time we have to do database maintenance. We did database maintenance twice in 2022. Once in January for an OS upgrade and the other in September when we upgraded MySQL from 5.7 to 8.0. When performing these maintenance windows, we have two choices. We can either upgrade the instance in place or create a new database instance with all of the necessary upgrades as a read-replica and promote it when the time is right.
Upgrading a database of our size in place via the native RDS methods would cause hours of downtime that is entirely out of our control. Would the upgrade take one hour? Six hours? 15 minutes? It’s impossible to know. Because of the uncertainty and lack of control, we’ve decided that any time we need to do maintenance on our MySQL database that it’s better to pay the extra money to create a new read-replica with all the upgrades and do the promotion process ourselves. It costs more money but results in stress-free upgrades requiring less than one hour of downtime. We can communicate with our users when we’re doing it and are confident about when we’ll be up and running again. Looking at this graph, we average a little more than $24,000/month if we don’t include the months we did maintenance. If we include our maintenance months, the average goes up to about $27,000/month. We view the extra $3,000/month as “peace of mind insurance.”
EC2-Other - $195,019.31 +32.4% YoY
Despite spending more money every month on EC2-Other than we did last year, the monthly cost was still mostly flat. We continue to spend the majority of our EC2-Other on regional data transfer. That’s the cost of having a multi-AZ architecture. As we continue to grow, there is no reason to believe this number will ever go down.
We continue to use the AWS-managed NAT gateway. As I said last year, it’s expensive, but it’s also reliable, and it’s easy to use. We’re still only a team of two, so we’ll pay the cost to help keep the team small.
Savings Plan For Compute Usage - $115,985.76 +18.7% YoY
We saved $47,156.31 in 2022 from purchasing our savings plans, so while we spent +18.7% more on Savings Plan For Compute Usage, we saved +33.3% more on EC2-Instances.
The chart is trending up because we learned from our mistakes. Now we ensure we always keep our savings plans as optimal as possible. Keeping up with our savings plans is a big reason we’re only spending 1.3% of our EC2-Instances cost on on-demand instances.
We didn’t pay anything in February because of the credits we received.
Support - $90,990.09 +38.8% YoY
The cost of support changes with usage. Because our AWS cost increased month-over-month, we spent more on support as we spent more on everything else. $90,990.09 is a very satisfying number to look at, but it’s also a tremendous amount of money.
S3 - $88,725.09 +19.9% YoY
35.4% of our S3 spend is on data transfer out to the internet. 36.6% of our S3 spend is on storage. The remaining 28% is mostly the requests that cause the data transfer out to the internet. Since we use S3 for email attachments, landing pages, and commerce, we can assume that a majority of the combined 63.4% of data transfer and requests comes from those three features. Most of the 36.6% we spend on storage is backups of our Cassandra and Elasticsearch clusters.
EC2-ELB - $43,272.45 +14.7% YoY
Our ELB costs increased by 6.4% from January to December. Kit grew MRR by 14.9% in 2022. This is a good sign because we know that the cost of handling requests to the application doesn’t scale linearly with new accounts and growth. That said, we expect this cost to grow as more people use our service.
Redshift - $43,165.27 +101.5% YoY
Our Redshift costs stayed flat all year despite spending over 100% more YoY. The most significant contributor to the yearly increase was changing instance types. We used to be on a smaller dc2 instance, and now we’re on a bigger ra3 instance. The change was necessary for a better analytics pipeline which has been a big focus of the company for the last few years. With instance reservations on the cluster, we can be confident that the cost will be predictable.
Everything Else - $84,636.18 +161.3% YoY
We started using more AWS services in 2022. We started using GuardDuty to help monitor our AWS accounts, adding about $1,100/month to our bill. We also started using QuickSight in March, adding about $700/month to the bill. The final, significant addition was the Managed Service for Prometheus that we added in November for about $2,300/month. You can see it in the graph. We were billed for 33 other AWS services not listed above.
Conclusion
Our year-over-year spending is much higher, but our month-over-month spending is stable and predictable. We know places where we can improve, but we also know that sometimes spending a little more money is better and safer for stability and peace of mind. We work hard to find the balance between the two. We do our best to be fiscally responsible while maintaining reliability to continue helping creators earn a living.