Comparing IaC Security Tools

It can be overwhelming looking for the best Infrastructure as code (IaC) security tools. When we started our initiative to secure our cloud environment, we spent a great deal of time evaluating many Cloud Security Posture Management (CSPM) solutions in the market. We immediately ran into “the noise problem”, being flooded with hundreds of alerts, making our security efforts daunting. While there are not as many tools in the market with the focus of Infrastructure as code (IaC) security, these tools also suffer from noise fatigue making them unusable in our CI/CD pipelines. And so, we decided to build Cloudrail to solve our problem, as well as others’.

The primary tenet of Cloudrail is to make security seamless in our DevOps practice. Therefore, solving the noise problem was the key objective of developing Cloudrail. The top questions we get asked by our users are:

  • How is Cloudrail different?
  • How noisy are these tools?

In this blog post, I want to answer these questions by sharing the results from our extensive testing. We compared Indeni Cloudrail with checkov, KICS, terrascan and tfsec.

UPDATE: This blog has been updated on April 14th, 2021, to reflect a more recent comparison with updated versions of all tools.

How is Cloudrail different?

As part of the introduction of Cloudrail, we highlighted Cloudrail’s unique advantages:

  • It “stitches”, or merges the Terraform plan with a snapshot of the cloud account. This means Cloudrail considers resources already existing in the cloud environment (that aren’t described in the plan) and uses them as part of its analysis.
  • It uses a context engine to understand relationships between resources and do a deep analysis of a resource’s security risks.
  • The above two result in 3x less noise than alternatives, while also finding high-severity security risks other tools are not capable of finding.

The above advantages, result in some major benefits:

  • Cloudrail is easy to integrate into CI/CD pipelines, because it won’t stop the pipeline for false positives and low severity issues.
  • Cloudrail can be used to provide visibility into the cloud environment changes being conducted via Terraform across the organization, without interrupting the developers’ flow of work.
  • Cloudrail allows for easy prioritization of issues to be resolved, ensuring devs fix only the issues that truly matter.
Related Article  Understanding the differences between tools like Terraform, CloudFormation, Ansible, and Puppet

The big benefit of Cloudrail: a tool that can actually be relied on by both security people and devs, resulting in an increased cloud environment security while minimizing organizational friction.

Tools and Data Used in Testing

When we introduced Cloudrail, we shared the “cloudrail-demo” repository. In it, there are many different test cases where Cloudrail’s logic can find issues others can’t, or is capable of ignoring false positives others would alert about.

All of the tools were tested against all of the test cases in that repository. We used:

  • Checkov version 2.0.46
  • Cloudrail version 1.2
  • KICS version 1.2.3
  • Terrascan version 1.4.0
  • tfsec version v0.38.3

To reproduce the results, see test/aws/terraform/ in the repository.

Comparison Results

We found a few cases of interest:

  • Redshift without public access

    Checkov, KICS, Terrascan and tfsec will alert you about “Redshift cluster should not be publicly accessible” because it has “publicly_accessible = true”. However, the cluster is in a subnet that has no route to the Internet, and therefore the cluster is not actually publicly accessible.

    Cloudrail understands this context and knows the cluster is not actually accessible publicly, therefore no violation will be listed for this.

  • S3 bucket – is it really public?

    An S3 bucket is being created with a public ACL. Generally, not a good idea. But, if it’s created in an account with a public block at the account level, then it won’t be an issue. Cloudrail takes the account-level public block settings when assessing a bucket’s security posture, which the other IaC tools completely ignore. The result is that Cloudrail generates less noise for the developers to deal with.

  • Same role used with a public EC2 and a private EC2

    Having roles assigned to EC2, and other compute resources, is a good idea. However, assigning the same role to both public and private resources may be asking for trouble. If the private resource needs a specific permission, the public resource will get it as well, and that exposes the capabilities of a potential attacker.

    In this scenario, the other tools were not able to find anything, whereas Cloudrail did.
Related Article  What does "declarative" mean in infrastructure as code?

More examples are available here.

Additional differences between Cloudrail and other tools:

  • All tools will identify when encryption at rest is not being used. However, Cloudrail will make a distinction: it has a map listing all the applicable resources and for each one, which is easy to turn on encryption at rest, and which are hard. For those that are easy, it will flag a violation. For those where it’s hard to turn on encryption, and the resource already exists, it won’t flag a violation.

    The rationale is that if a resource already exists, and in order to turn on encryption at rest you need to destroy and rebuild it from a backup, you’ll likely not going to do that. Kudos to Chris Ferris on highlighting this.

Indeni Cloudrail is a pragmatic tool for DevOps

So, with Cloudrail, you’ll get violations for things you are likely to fix, vs just a list of problems no one will do anything about. This means your devs will actually listen to Cloudrail and fix issues, and your cloud environment will be more secure.

Looking to build secure infrastructure?

Get started with Cloudrail