Published 2022-05-26. Last modified 2024-04-08.
Time to read: 12 minutes.

This page is part of the posts collection, categorized under AWS, Internet, Security.

I am a free agent, independent and critical. You can buy my time as a software expert, but not my opinion. Just because Amazon / AWS hired me in the past to help them against a patent troll does not mean they own me. Quite the contrary.

In the time you take a quick shower, your AWS account can be highjacked and tens of thousands of dollars in fees can be incurred. EC2 is the service that provides the greatest financial risk. You can take steps to limit your liability, and this article shows you how.

Out-of-Control Cloud Costs

Out-of-control costs are causing cloud customers to reduce or eliminate their cloud use. The cloud backlash has begun: Why big data is pulling compute back on premises – Thomas Robinson, published by TechCrunch 2023-03-20.

Why we’re leaving the cloud, by David Heinemeier Hansson, creator of Ruby on Rails and CTO of 37signals. Mr. Hansson also wrote We stand to save $7m over five years from our cloud exit.

AWS IAM Users and Roles Have No Budget Limitations

AWS customers are exposed to unlimited financial liability

I encountered two main issues when attempting to secure against AWS account hijacking:

1. AWS does not provide a mechanism to guard against launching expensive services. For example, if an IAM user has a role that allows EC2 instances to be launched, then that user can launch an unlimited number of EC2 instances of any size.
   
   The most expensive Amazon EC2 is currently the p4de.24xlarge, which costs $40.96 USD per hour on-demand in the US East (N. Virginia) region. The instance comes with 96 vCPUs, 1152 GiB of RAM and eight 1TB NVMe SSDs, and has eight NVIDIA A100 Tensor Core GPUs.
   
   Using stolen credentials that allow EC2 instances to be launched, an attacker using scripts could spin up an armada of p4de.24xlarge instances and incur eye-popping costs in minutes.
2. AWS Budgets provides a convenient way to shut down pre-designated services when a total budgetary amount is exceeded. However, the AWS AMI security model is not integrated into real-time cost monitoring. Thus, there is no way to automatically shut down newly launched service instances that exceed a pre-authorized budgetary amount.

Vercel’s Pricing Policy Incorporates Limits

We provide customers with tools to observe, control, and alert on their infrastructure spend with Spend Management. You can define a spend amount (e.g. $40) and receive email, web, and SMS notifications as you reach that amount. When reaching 100%, you can optionally automatically pause all projects with a hard limit. Over the past year, we've added features to the platform based on your feedback to help automatically prevent runaway spend, including recursion protection, improved function defaults, hard spend limits, Attack Challenge Mode, and more.

AWS, Azure, Google App Engine, IBM Cloud and Linode all need a pricing policy like Vercel’s.

Shared Security Responsibility

The AWS Customer Agreement and Shared Responsibility Model describe how AWS shares responsibility for security with users. If you follow the AWS security recommendations, described next, you will not be held accountable for extra charges if your AWS account is hijacked.

The AWS Security Blog published a nice overview, entitled Getting Started: Follow Security Best Practices as You Configure Your AWS Resources. Some of the same information is also provided in Security best practices in IAM.

Even though the AWS instructions do not prevent bad guys taking over your account, follow them anyway so you won't be held liable for the costs incurred if your AWS account is hijacked.

AWS Security Recommendations

Security needs to have depth. Any single measure can fail, and given enough scale, all measures will eventually fail. Layer your security measures so that one failure, no matter how grave, will not be fatal.

AWS recommends the following. I have highlighted what AWS personnel have described to me as being the quickest and easiest path; this article walks through those steps to the extent that I have been able.

1. Set up at least two of the following services to monitor cost and usage:
   2. .
   3. CloudTrail User Guide.
   4. Web Application Firewall (WAF).
   5. Trusted Advisor. I found that Trusted Advisor was somewhat easier to use than CloudWatch.
   For more information about managing your AWS cost and usage, see the AWS Cost Management User Guide.
2. Set up at least one of the following security best practices:
   1. .
   2. AWS Security Hub.
   3. Amazon GuardDuty.

Do The Easiest Things First

I usually prefer to do the easiest things first. Not everyone agrees. Doing even one highlighted item above provides a significant security improvement, so why wait?

The best is the enemy of the good.

 – Voltaire

Better a diamond with a flaw than a pebble without.

 – Confucius

Striving to better, oft we mar what’s well.

 – Shakespeare

Root Credentials

If a bad guy has your root credentials, they can change budget limits. For greatest security, delete your root credentials by selecting Access Keys (access key ID and secret access key) as shown in the image below. However, if you do so, many things become impossible.

Enabling MFA

Yubikey NFC

The easiest highlighted item above is to enable multifactor authentication (MFA).

I use a virtual MFA device, enabled for my root account Google Authenticator for Android and iOS. Google Authenticator features a RFC 6238 standards-based TOTP (time-based one-time password) algorithm.

I use a YubiKey to provide MFA for the AMI user ID that I use to log into the AWS console for everyday work.

Only use root credentials when absolutely necessary.

Trusted Advisor

After adding MFA to root credentials, I found that using AWS Trusted Advisor was the next easiest thing to try to make my AWS account secure.

Once you visit the Trusted Advisor Console, and agree to enable Trusted Advisor, IAM permissions are added to the AWS IAM user that you are logged in as.

Usage seems straightforward; however, I found the information for securing S3 buckets puzzling at first:

The column labeled ACL Allows List means that buckets flagged with Yes have an ACL that allows objects to be listed. AWS recommends that S3 buckets not allow objects to be listed by the public. To correct this:

1. Click on the bucket name.
2. Click on the Permissions tab in the page that opens next.
3. Scroll down to Access Control List.
4. Click on the Edit button.
5. Disable the items marked in red, as shown below.

Referring back to the previous screenshot, the Policy Allows Access column flags all S3 buckets used to serve webpages as insecure. To serve HTML pages and assets stored in an S3 bucket, the permissions must be set to allow objects to be read by everyone. Trusted Advisor flags these buckets as insecure, which does not make sense to me.

Perhaps there is a better security policy for when webpages are served via CloudFront, but as is often the case with AWS documentation, it is difficult to piece together how to optimally configure S3 permissions with CloudFront options.

Best Practices Evolve With Technology

The AWS documentation is written as if the products described have always existed in their present state. In the course of researching this article, I discovered that AWS provides revision histories for their services. For example, the revision history of S3 is documented in the AWS S3 User Guide, and the CloudFront revision history is documented in the AWS CloudFront Developer Guide. Revision histories are useful for experienced technologists to review periodically, so they can keep up with best practices as they evolve.

I started using AWS S3 to serve websites in 2013, and at that time, CloudFront did not exist yet. As the two services matured, best practices, which were at first unknown, evolved and interactions between them became more complex. In particular, CloudFront http/https promotion, how origins are specified, and S3 permissions, all interact in ways that were not well documented for several years. This was a source of security problems and unwanted downtime for me.

I have 15 buckets that are used to serve web pages. On my AWS Support Center console, Trusted Advisor shows that “You have a yellow check affecting 15 of 24 resources”; however, the Trusted Advisor console displays those items as red triangles with exclamation marks.

This is confusing, and became frustrating when an AWS security support technician suggested I disregard anything that seemed annoying. For an IT/security person, that advice is the quickest way to hell that I know.

Rough Spots That Need Love

While researching this article I found:

Under Origin Settings, for Origin Domain Name, choose the Amazon S3 bucket that you created previously. For S3 bucket access, select Yes, use OAI (bucket can restrict access to only CloudFront). For the Origin access identity, you can choose from the list, or choose Create new OAI (both will work). For Bucket policy, select Yes, update the bucket policy.

 – From Use an Amazon CloudFront distribution to serve a static website

However, I found conflicting information near the top of this article:

Important

If you use an Amazon S3 bucket configured as a website endpoint, you must set it up with CloudFront as a custom origin. You can’t use the origin access identity feature described in this topic. However, you can restrict access to content on a custom origin by setting up custom headers and configuring your origin to require them. For more information, see Restricting access to files on custom origins.

 – From Restricting access to Amazon S3 content by using an origin access identity (OAI)

As if this is not confusing enough, Using various origins with CloudFront distributions states that in order for Amazon S3 redirects to work, a non-standard format for the origin must be used:

http://bucket-name.s3-website-region.amazonaws.com

As I discussed in a previous article, AWS S3 buckets support two types of redirects, but there is no indication of which type might or might not work with the non-standard origin format.

Also, there is no mention of whether https would work or not, or if these disparate instructions even work together. As usual, there is lots of uncertainty about how things actually work on AWS, and for how long before things change.

As per standard operating procedure with AWS, one would have to muck about extensively to determine which of the conflicting statements above is generally prevalent, and under what circumstances. I expect that one article is newer than the other, and it likely supercedes the older article somewhat to some degree under as-yet unknown circumstances. If I get lucky, the Trusted Advisor warnings will go away. Trusted Advisor should reference whichever article is currently correct.

Update 2022-05-31

I received the following email from Trusted Advisor. Unfortunately, the contents of the email were different from what I found on the Trusted Advisor console. Perhaps this is because a higher-level, and more expensive, service contract would be required to see those messages. If so, then this is not the way to gain happy customers. The cost of the higher service level would be 4 times more than my total AWS costs at present, clearly not something that I would want to do.

Dear AWS customer:

AWS Trusted Advisor currently shows alerts for 3 checks (1 red and 2 yellow) and $0 of potential monthly savings based on your usage.

Here is a summary of status changes for this week:

The alert severity of these checks has improved:

From red to yellow:
   Amazon S3 Bucket Permissions

From red to green:
   VPC
   Security Groups - Specific Ports Unrestricted

From yellow to green:
   VPC Internet Gateways

The alert status of 5 checks has not changed. For details and recommendations, visit AWS Trusted Advisor.

Sign in to the Trusted Advisor console to view your check results.

Best,

AWS Support
Was this report helpful? Send us your feedback.

To add insult to injury, the final "Send us your feedback" just links to a generic form for AWS sales, where one can pound sand.

There is abundant evidence to demonstrate that whereas amazon.com is consumer-centric, AWS marches to a very different drummer. Perhaps small businesses are not presently viewed by AWS as a significant growth opportunity. My personal experience is that the current risk/value proposition provided by AWS is no longer attractive for that demographic.

Update 2022-06-03

The credit card I used to pay for my AWS usage had a very high limit. In retrospect that was unwise.

AWS took weeks to agree to refund the charge on my credit card, and then they said it would take another week for the reversed charges to appear. This meant that if I only paid for the other charges on that card, the month-to-month credit card interest on the fraudulent AWS charges would have been problematic.

Just before the credit card billing period ended, I called Visa and asked what my options were. Apparently I am not the first person who had this problem with AWS. The answer was immediate: report the charges as fraud. OK, that made sense. This meant my card was immediately cancelled, the charges were forgiven by Visa, and a new card with a new number would arrive in about a week.

Managing Costs With AWS Budgets

Short-lived hijacking is the most serious financial threat, and AWS Budgets was seemingly not designed to guard against that. Ticking off this item will enable AWS to absolve you of financial responsibility for being hacked, but you should not expect that it will do much to slow down an attacker.

AWS Budgets are easy to set up in a passive manner, but they are clumsy to work with and not effective for preventing new services from being launched that would exceed the budget. The cost limiting functionality is not automatic, it must be set up manually. AWS documentation does not provide detailed how-to instructions for common scenarios. Tens of thousands of words of documentation must be digested before the full capabilities can be harnessed.

Another significant issue is that only three types of actions are supported:

1. IAM Policy – I am unclear on how to set this up so the bad guys cannot launch new services, and I am not certain this is possible.
2. Service Control Policy – Hopefully this could prevent new services from being launched that would exceed the budget; however, my head exploded when I researched this.
   
   Attempting to use this option to prevent new, expensive services from being launched would seem to introduce lots of complexity. Perhaps AWS did not consider this use case, or deemed it unimportant.
3. Automate Instances to Stop for EC2 or RDS – If you run a t3.nano instance, for example, bad guys can only incur a certain amount of financial damage. Instead of shutting down the EC2 or RDS instances that I want to run, my main concern is to prevent bad guys from launching expensive new services. This option is therefore unsuitable for my use case.

If AWS Budgets provides an easy way of preventing new services to be launched that would exceed the budget, I could not find it after several hours of reading.

Passively Monitoring Budget Spend

Simply follow the directions and accept the defaults. Daily budgets do not support enabling forecasted alerts, or daily budget planning, so select Monthly Budgets. There is no need to set up SNS alerts or AWS Chatbot alerts, email notification works just as well for most users.

Attaching Actions

Following are my notes from my attempt to enable cost limiting. I was unable to attach an action that prevented expensive new services from being started, and it is unclear if resolving the problem will, in fact, provide the desired benefit. Perhaps someone reading these notes will be able to tell me about an approach that might work for a reasonable amount of effort.

Because AWS will absolve you of financial liability if you create a Budget without any actions, don't stress about the pointless exercise involved in making a Budget that cannot protect you. Just do it and give AWS more time to figure out a better way of preventing account hijacking.

You can stop reading this article now, unless the details interest you.

For PaaS vendors such as AWS, Azure, Digital Ocean, Cloudflare, ScaleWay, etc: “pay-as-you-go” is shorthand for “there is nothing you can do to limit your financial liability”.

1. When you get to the step labeled Attach actions - Optional, choose Add Action.
2. You must choose between using an existing IAM role for AWS Budgets to use, or you can create a new IAM role expressly for this purpose. IAM roles are free. I like the idea of a dedicated IAM role, so I clicked on manually create an IAM role.
   
   
3. A new browser tab opened, and I clicked on the blue Create role button.
4. Now a confusing page appeared:
   
5. I selected the default Trusted entity type, AWS Service, then from the pulldown menu labeled Use cases for other AWS services: I selected Budgets.
6. A new radio button appeared underneath the pull-down, also labeled Budgets, and I clicked on that. This does not win any usability awards.
7. I clicked on the button labeled Next.
8. Managed policy name: AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM
   
   This managed policy is focused on specific actions that AWS Budgets takes on your behalf when completing a specific action. This policy gives AWS Budgets broad permission to control AWS resources. For example, starts and stops Amazon EC2 or Amazon RDS instances by running AWS Systems Manager (SSM) scripts.

    – From Using identity-based policies (IAM policies) for AWS Cost Management
   On the page, I applied AWSBudgetsActionsRolePolicyForResourceAdministrationWithSSM, then I clicked on Next.
   
9. On the final page, I named the IAM role Budget and clicked on Create role.
   
10. Back in the Billing Management Console, I refreshed the list of available IAM roles, then selected the new role called Budget.
    
11. I selected Service Control Policy from the pull-down that appeared next, labeled Which action type should be applied when the budget threshold has been exceeded?. I have no idea how to proceed, or even if going in this direction might provide the desired results.
    

Share to Twitter, Hacker News, LinkedIn.