Kit spent over $1.4 million on AWS in 2022

generalengineeringaws
Kit spent over $1.4 million on AWS in 2022
Kris Hamoud
Kris Hamoud is an Infrastructure Engineer who enjoys building simple and scalable solutions • Kris's website

Overview

We spent $1,471,169.38 on AWS in 2022. This is up 38% from 2021 and was 4.7% of net revenue for the year. We increased spending on every major AWS service that we use. We spend the most on EC2-Instances and Relational Database Service. Those two services account for 55% of our annual spending. If we include EC2-Other and Savings Plan For Compute Usage, which are unavoidable accompanying expenses to EC2-Instances, those four services account for 76.2% ($1,120,380) of our annual spending. Most of our EC2 spend goes toward our Elasticsearch and Cassandra clusters. We use RDS for our primary MySQL database and smaller databases that power our small services. You’ll notice we’re spending significantly more on Redshift. As I mentioned last year, we’re investing heavily in improving our data and analytics pipeline, and having a better Redshift cluster was our first significant change.

Below is the high-level breakdown of our spending. I’ll go over each service and detail how we use it.

High-level breakdown:

  1. EC2-Instances - $480,740.30 +38% YoY
  2. Relational Database Service - $328,634.92 +35.7% YoY
  3. EC2-Other - $195,019.31 +32.4% YoY
  4. Savings Plan For Compute Usage - $115,985.76 +18.7% YoY
  5. Support - $90,990.09 +38.8% YoY
  6. S3 - $88,725.09 +19.9% YoY
  7. EC2-ELB - $43,272.45 +14.7% YoY
  8. Redshift - $43,165.27 +101.5% YoY
  9. Everything Else - $84,636.18 +161.3% YoY

Total Monthly Spend

Our total monthly bill increased by 12.8% from January to December. The leading cause of the increase comes from spending more on many smaller services. Despite spending more on every service year-over-year, the individual services’ monthly costs remained flat throughout 2022. You’ll also notice that February is significantly lower than the rest of the months.

There are two reasons:

  1. February is the shortest month
  2. We got the second tranche of our AWS credits from our account rep

You’ll see the effects of the credits throughout the breakdown.

Monthly Total Spend Chart

EC2-Instances - $480,740.30 +38% YoY

Our EC2 spending increased by +1.9% from January to December. You’ll notice again the effects of the AWS credits we received in February. You’ll also see that our EC2 spending decreased from January to May. This year we worked to reduce the number of on-demand instances we use. As of December 2022, only 1.3% of our EC2 spend is on-demand. Instance reservations, savings plans, or spot instances cover the remaining 98.7%. This improvement is how we reduced EC2 spending while increasing the number of emails we send every month.

However, EC2 costs increased again in June and continued to do so until October because we had to scale out our Elasticsearch and Cassandra clusters multiple times this year.

Monthly EC2 Spend Chart

We use the i3 and i4 instance families to power our Elasticsearch and Cassandra clusters. We don’t use them for anything else, so it’s easy to get a basic idea of how much we spend on those data stores compared to everything else in our fleet. One of our goals for 2023 is to find ways to optimize how much data we store in these databases to help us curb their growth rates. Storage optimization and the use of cold storage is one of the highest leverage things we can do to ensure a healthy and predictable AWS bill.

Elasticsearch and Cassandra compared to everything else

Relational Database Service - $328,634.92 +35.7% YoY

Kit has a love/hate relationship with RDS. In 2019 we spent over $45,000/month on one MySQL database. After years of effort, we’ve gotten ourselves to a stable and predictable RDS spend. RDS is an excellent service for small teams like Kit because it reduces operational complexity in most cases. However, you can see that we have a few months where the cost of RDS increased substantially. That happens every time we have to do database maintenance. We did database maintenance twice in 2022. Once in January for an OS upgrade and the other in September when we upgraded MySQL from 5.7 to 8.0. When performing these maintenance windows, we have two choices. We can either upgrade the instance in place or create a new database instance with all of the necessary upgrades as a read-replica and promote it when the time is right.

Upgrading a database of our size in place via the native RDS methods would cause hours of downtime that is entirely out of our control. Would the upgrade take one hour? Six hours? 15 minutes? It’s impossible to know. Because of the uncertainty and lack of control, we’ve decided that any time we need to do maintenance on our MySQL database that it’s better to pay the extra money to create a new read-replica with all the upgrades and do the promotion process ourselves. It costs more money but results in stress-free upgrades requiring less than one hour of downtime. We can communicate with our users when we’re doing it and are confident about when we’ll be up and running again. Looking at this graph, we average a little more than $24,000/month if we don’t include the months we did maintenance. If we include our maintenance months, the average goes up to about $27,000/month. We view the extra $3,000/month as “peace of mind insurance.”

Monthly RDS Spend Chart

EC2-Other - $195,019.31 +32.4% YoY

Despite spending more money every month on EC2-Other than we did last year, the monthly cost was still mostly flat. We continue to spend the majority of our EC2-Other on regional data transfer. That’s the cost of having a multi-AZ architecture. As we continue to grow, there is no reason to believe this number will ever go down.

We continue to use the AWS-managed NAT gateway. As I said last year, it’s expensive, but it’s also reliable, and it’s easy to use. We’re still only a team of two, so we’ll pay the cost to help keep the team small.

Monthly EC2-Other Spend Chart

Savings Plan For Compute Usage - $115,985.76 +18.7% YoY

We saved $47,156.31 in 2022 from purchasing our savings plans, so while we spent +18.7% more on Savings Plan For Compute Usage, we saved +33.3% more on EC2-Instances.

The chart is trending up because we learned from our mistakes. Now we ensure we always keep our savings plans as optimal as possible. Keeping up with our savings plans is a big reason we’re only spending 1.3% of our EC2-Instances cost on on-demand instances.

We didn’t pay anything in February because of the credits we received.

Monthly Savings Plan Spend Chart

Support - $90,990.09 +38.8% YoY

The cost of support changes with usage. Because our AWS cost increased month-over-month, we spent more on support as we spent more on everything else. $90,990.09 is a very satisfying number to look at, but it’s also a tremendous amount of money.

Monthly Support Spend Chart

S3 - $88,725.09 +19.9% YoY

35.4% of our S3 spend is on data transfer out to the internet. 36.6% of our S3 spend is on storage. The remaining 28% is mostly the requests that cause the data transfer out to the internet. Since we use S3 for email attachments, landing pages, and commerce, we can assume that a majority of the combined 63.4% of data transfer and requests comes from those three features. Most of the 36.6% we spend on storage is backups of our Cassandra and Elasticsearch clusters.

Monthly S3 Spend Chart

EC2-ELB - $43,272.45 +14.7% YoY

Our ELB costs increased by 6.4% from January to December. Kit grew MRR by 14.9% in 2022. This is a good sign because we know that the cost of handling requests to the application doesn’t scale linearly with new accounts and growth. That said, we expect this cost to grow as more people use our service.

Monthly ELB Spend Chart

Redshift - $43,165.27 +101.5% YoY

Our Redshift costs stayed flat all year despite spending over 100% more YoY. The most significant contributor to the yearly increase was changing instance types. We used to be on a smaller dc2 instance, and now we’re on a bigger ra3 instance. The change was necessary for a better analytics pipeline which has been a big focus of the company for the last few years. With instance reservations on the cluster, we can be confident that the cost will be predictable.

Monthly Redshift Spend Chart

Everything Else - $84,636.18 +161.3% YoY

We started using more AWS services in 2022. We started using GuardDuty to help monitor our AWS accounts, adding about $1,100/month to our bill. We also started using QuickSight in March, adding about $700/month to the bill. The final, significant addition was the Managed Service for Prometheus that we added in November for about $2,300/month. You can see it in the graph. We were billed for 33 other AWS services not listed above.

Monthly Everything Else Spend Chart

Conclusion

Our year-over-year spending is much higher, but our month-over-month spending is stable and predictable. We know places where we can improve, but we also know that sometimes spending a little more money is better and safer for stability and peace of mind. We work hard to find the balance between the two. We do our best to be fiscally responsible while maintaining reliability to continue helping creators earn a living.