Tech Talk

Keeping AWS Safe in an Unsafe World

~ by Michael Bailey

Introduction

As more businesses transition to the cloud, administrators find themselves in a position where they have to secure resources well beyond their previous network perimeter prior to adoption.  Despite the fact that, as a whole, AWS can often be easier to manage than many cloud computing platforms, its complex and internet-facing nature requires that it have heightened security in functions ranging from asset management to alerting. This is warranted when you consider that everything from relational databases to S3 buckets in AWS are compromised on a regular basis, as evidenced by a multitude of news reports and our own experience working with The Crypsis Group’s clients.

Unlike similar on-premises solutions, when a misconfigured AWS account is compromised, its assets can be virtually wiped out at a click of a mouse or, conversely, expensive assets can be spun up globally in potentially low-visibility geographic regions. While workloads and services vary substantially between companies, we will explore steps to consider in just about all scenarios.

Methodology

At a high level, regardless of the situation, an AWS administrator generally has to consider the following issues and questions:

  • Network Controls
    • Should that internet-facing EC2 instance be facing the internet?
    • Should that S3 bucket be accessible from any IP address or just some?
  • Visibility
    • If someone’s account access is compromised and a cyber criminal launches a server in Singapore, will it go unnoticed?
    • If an attacker starts downloading data from S3 in Iowa and your company’s operations don’t call for that kind of activity, would you notice it?
  • Auditing
    • Is CloudTrail enabled? What if someone tampers with logs or otherwise deletes them?
    • Beyond simple CloudTrail, if EC2 is in use, are those VPCs being logged? If S3 is being used, is access to those buckets being logged?
  • Thorough Access Control
    • Are access keys treated differently than console logins? If so, what’s going to happen when a user loses his or her password versus their access keys?
    • How granular are employee account permissions? Remember, employees can’t shoot themselves in the foot if you don’t give them the gun to do it.
    • The principle of least privilege extends to the cloud. In AWS it can be more convenient to add employees to existing groups, and grant more rights to that group, rather than making new groups and permissions as capabilities expand and situations change. This can create problems for organizations in the long term.
    • Consider corporate AWS users Kevin and Jane. They have unique requirements of one another and only have minor overlap in permissions necessary to fulfill their job. Does Jane have more permissions as a result of Kevin’s required access or vice versa? Administrators are susceptible to taking shortcuts in AWS where writing policy documents manually is sometimes necessary.

External Resources Worthwhile in AWS

  • AWS Bot - A Ruby AWS Slack bot to interact with AWS. Relatively easy to modify.
  • trufflehog - Scan Git repository and identify secrets in code. Good for mitigating AWS key leakage.
  • AWS Key Disabler - Warns about and deactivates old AWS keys. Note by default it sends poorly formatted reports via email.
  • AWS Security Arsenal - This list (and similar ones) made its rounds on the internet, and Crypsis has drawn from it before.
  • AWS Simple Monthly Calculator - Cost approximation. AWS recommends their TCO calculator instead but it's no replacement.

Let AWS Help You

AWS typically reimburses accounts (though this can’t be depended on) that had their computing power exploited by fraudsters. In the news, when an S3 bucket is hacked, sometimes the title reads “Company X hacked via misconfigured bucket“ and sometimes the title reads “Company X loses data in AWS hack,” and it’s safe to say AWS would like to avoid their services being associated with breaches.

ASW Help

That's why AWS has Config, GuardDuty and Trusted Advisor. From a cost perspective, Config can be very inexpensive, Trusted Advisor starts out free, and GuardDuty's cost depends on usage trends. Other situational tools include AWS Macie for S3 buckets, Web Application Firewall, and Inspector for automated, routine security assessments.

 Trusted Advisor can be considered a baseline security analyzer. It will tell you if there's any cause for concern.  While it may produce a few false positives, they are not burdensome and typically take only seconds to read.

 AWS Config is an excellent tool, not only to maintain a good internal posture but also to ensure employees are following business rules. For instance, if a certain list of expensive EC2 instance types are started, your AWS team can be notified internally, likely triggering an audit. Because these employees require immediate access to starting EC2 instances, and because sometimes their tasking warrants expensive instances, this activity isn't prohibited, but is reviewed. If it was prohibited, Config could be connected to SNS and Lambda to turn those instances off as soon as they start. In addition to the AWS-provided instance type rule, the public-read-prohibited rule is also recommended; notifying an AWS team if someone manages to create a public S3 bucket.

 GuardDuty Panel

Redacted screenshot of a GuardDuty panel, all false positives.

AWS GuardDuty (pictured) uses threat intelligence to identify suspicious activity in an AWS account, such as suspicious source IP addresses acting in the account or unusual activity from an employee. It’s important to note AWS GuardDuty may not be suited for as many environments as the other services discussed here. It is frequently criticized for excessive noise and false positives; however, this is not the result of a product design flaw but rather is a side effect of security through threat intelligence. In a test exercise to determine GuardDuty's worth, Crypsis employees accessed the corporate AWS account through a Tor node and destroyed an EC2 instance, a scenario we've seen in the wild on multiple engagements. GuardDuty reported the activity as highly suspicious, albeit hours late. If it's affordable and properly configured, there's little downside to enabling it. A lesser known fact about AWS GuardDuty is that it accepts a company's known good IP address list and known bad IP address list for additional context beyond what it is configured with upon activation.

Scenario Testing

With IAM's extensibility, it is trivial to manually simulate a disaster event, whether it is an attempted deletion of a bucket or another action from a particular user group. Whenever a key configuration change is made to a Crypsis AWS group, a "scenario test" account is utilized to ensure overgranting hasn't occurred and, more often, that permissions aren't so restrictive it makes it impossible to reliably carry out duties (especially in the web console).

Key Security

One of the most popular means of leaking access keys is through Github and version controlled code in general. Github, like its underlying Git system, is designed to provide a historical record of code, so even if someone accidentally publishes a key, removing it isn't enough. The keys have to be deleted after leaking. This is why you can scan server-side with truffleHog, using this script. Any leaked keys, even to company-only servers, should be immediately revoked. It's also one of many reasons people like to rotate their keys, effectively setting an expiration on any keys that may have accidentally leaked by deleting them account-side and making new ones.

Github CommitsCommits on Github of people publicly removing their private keys. Clicking on commits will display keys, some of which may be valid.

Access keys are a major concern and it's important to accept that there's a good chance you won't know when they leak. The first line of defense for this is obviously limiting how much any particular key can do. Other, more bleeding-edge solutions such as temporary keys are also becoming popular. However, for users and certain applications this doesn't necessarily work. Many environments encourage users to rotate their own keys and enforce hard limits on those who don't. A common procedure is to warn a user twice before deactivating keys. As secrets can't be recovered or "put back," it's important to deactivate rather than delete whenever possible. If a company expects users to rotate their own access keys, make it as easy for them as possible. One thing to consider is to send an email to users with a direct link that allows them to rotate their keys, significantly increasing the likelihood that they will do so. Nobody wins when mission critical keys are deactivated because someone was too busy to click around.

Auditing

It's important to log everything that one can afford to (as much as S3 costs allow) via CloudTrail, and actually do something with those logs. One follow-up action may be to automatically bring them into on-premises Splunk instances to be indexed. After this, consider running a series of checks that may include but are not limited to:

  • API calls made outside of "known good" origins.
    • Accounts accessible from home may be harder to determine "known good" origins, but with the help of authentication activities like successful MFA logins, it should be easier to get closer to a full picture.
    • In cases of applications using keys, user agents, while spoofable, can be used as a potential indicator.
  • Console logins from users that aren't supposed to have the ability to console login, such as service accounts.
  • Geographically suspicious calls, including successful AWS API calls coming from countries where your organization isn't currently doing business, or calls from a user account that are geographically impossible due to travel time.

Conclusion

Hopefully this provides an understanding of how your AWS environment can be proactively managed. As with any security operations, an effective approach combines a constant cycle of improvement with a base requirement high enough to enable your company’s security team to sleep at night.

 

Back to All Posts