We have all heard the horror stories of the sysadmin who racked up a multi-thousand dollar bill because of a misconfigured environment, and sadly these tales of woe are indeed not myths. One of my customers very recently discovered a $4000 charge due to sending over 105 TB with of data to S3! What happened there, how did we fix it, and how can we prevent this from happening in the future?
First, the way I describe AWS to my customers is that it’s not one monolithic organization in which one service seamlessly communicates with other services via hidden channels. Instead, it’s easier to think of AWS as a huge conglomerate of separate teams (because that’s really what they are), and those teams’ products talk to each other largely via the use of APIs, accessed over HTTP. That means that if you want to send a file from an instance in EC2 (originating from an EBS block volume) to an S3 bucket, you need to establish a connection to that bucket in the same manner you would access a website over the internet.
Here’s the problem: AWS charges you for every gigabyte of data processed through a NAT gateway as well as egress out of EC2 to the Internet (as well as between regions and other AWS services). If you are moving 105 TB from EC2 over a NAT gateway to S3, you’re going to be facing a serious bill.
Ouch! So how do we avoid something like this from happening? The answer is thankfully very simple, and it’s in the form of a VPC Endpoint. A VPC Endpoint is a way that allows you to securely connect your existing VPC to another AWS service, effectively creating an internal route within AWS’ “walls”. In this fashion, your data never traverses over the gateway of last resort (either being a NAT gateway, your VPN, or IGW), and you are therefore not on the hook for processing that data!
To create a VPC endpoint, select the appropriate region you want to configure, and go to the VPC dashboard here:
Click on Create Endpoint. There are two separate types of endpoints: interface endpoints, which are driven by AWS PrivateLink, and gateway endpoints. Only two services are supported by gateway endpoints: S3 and DynamoDB. Find S3 and select it:
Finally, select the appropriate VPC and the route tables that are associated with the Subnets you want to ensure utilize the endpoint. In the context of Zerto and keeping transfer costs from the ZCA to S3 buckets low, this will be the VPC that your ZCA is installed and the route table(s) that your ZCA’s subnet is associated with. You may also decide to customize your endpoint policy at this time as well. If your security rules require a minimally-permissive policy to be implemented to control which resources may utilize the endpoint, you may edit that policy now or later.
Click on the create endpoint button when finished! The endpoint will be created and S3-bound traffic originating from the subnets that are associated with the route tables you have specified will now flow over the endpoint.
Best of all, gateway endpoints are free! There are edge cases where a VPC endpoint may not be an appropriate fit, and I encourage you to check out https://docs.aws.amazon.com/vpc/latest/userguide/vpce-gateway.html#vpc-endpoints-limitations on the matter, but for Zerto customers wanting to protect their data in AWS, this is a very important topic to understand and I strongly advise you to consider it. It takes literally moments to set up, is free, and will save you the headache of begging your AWS rep for cost forgiveness (or explaining to your boss why there’s suddenly a massive unexpected charge this month).
I welcome any tips, suggestions, or comments here or on Twitter. Hit me up at @vPilotSchenck.