This is the story of Indeni’s transition to cloud-native development and cloud modernization efforts with security being a key focus. By dogfooding Cloudrail, our Infrastructure as Code (IaC) security solution, we built a mature DevSecOps process.
The end results:
- Achieved zero security violations without hindering agile delivery
- 10x less security investigations
- Continuous delivery with a much improved organization security posture through security compliance automation
Our Journey to the Cloud
Our journey began in 2014 where we leveraged AWS for development and testing. In 2017, we introduced Indeni Insight, a cloud component of our network security infrastructure automation, making cloud infrastructure a production environment. We further increased our reliance on the cloud environment by setting up a proxy in AWS with which all installations of our product communicate with.
In 2019, our cloud transformation kicked into high gear with the greenfield development of Cloudrail. We made significant architectural and strategic decisions about our cloud operations and environments, including:
- Everything as code
We adopted Terraform as the method to provision AWS resources. In a matter of minutes, a developer can create a development environment that is consistent with our production environments (just without real customer data). We added Terraform to our continuous integration workflow to enable deployment of infrastructure alongside software in the same pipeline.
- Transitioned from a single account to multi-account leveraging AWS Organization
Each developer has his/her AWS account along with a main development account, pre-production and production accounts. Segregating the accounts limits blast radius and the ability to have multiple security postures. We can give developers a sandbox environment for experiments while limiting permissions in the pre-production and production environments.
Since we are in a business of building a SaaS product specifically targeting security professionals, securing customer data is of paramount importance. We embarked on a journey to heighten cloud security in 2020.
With cyberattacks on the rise and the risks of supply chain attacks being greater now than ever before, we have to be paranoid about protecting our customer data in the cloud. To gain the trust from our customers, our goal was to obtain certification for SOC2 compliance by Q1 of 2021.
Process for securing customer data
Starting with the list of assets, we identified all the systems (Gmail, Salesforce, AWS, Zendesk, etc.) consisting of customer data. After assessing how these systems were protected and potential ways of accessing them, we went through a mapping exercise to rate the sensitivity of the customer data and decided on our security focus. Securing our AWS environment was identified as a top priority.
Assess – finding a needle in a haystack
To gain visibility into our cloud security posture, we deployed a few Cloud Security Posture Management tools in our environment. We were immediately bombarded by hundreds of alerts making our security efforts daunting. After spending significant time and energy investigating these issues, 97% of them turned out to be either false positives or irrelevant with no impact to our security postures. Worst yet, these tools missed a few true security risks. When we finally found that needle in a haystack, the fun began with the remediation process.
Remediate – playing cat and mouse
With our security heritage as a company, our engineers fully understand the need for security. We were also very selective of which issues to pursue because every security control we implement makes life more difficult for our developers. We ended up limiting ourselves to one or two issues per week. Even with all good intentions, security issues were de-prioritized because they were not immediate problems and they did not impact our customers. Not to mention that some issues were extremely costly to fix after deployment. For example, the cost to change a data store to encrypt in AWS is extremely high once it is provisioned (read this cloud encryption blog post) making it necessary to catch issues before deployment. More importantly, if the security controls inadvertently blocked the developers, we would slow down development and the production of the product.
Maintain – manual process does not scale
With hundreds of Terraform changes per month, it was simply not feasible to perform manual security reviews for every change. Even if manual reviews were not a bottleneck, reviews were missed because developers may not be aware of the need for a security review. Although it was no fault to anybody, it certainly posed potential security risks with changes unreviewed by security experts. To fill the gap, we started evaluating open-source IaC security tools like checkov and tfsec. Once again, we ran into “the noise problem” similar to the CSPM tools.
The Solution – Indeni Cloudrail
We quickly learned from our experience that legacy approaches to security uncover issues too late, making them hard and expensive to fix. A more efficient way is to take a shift left security approach where infrastructure as code is continuously evaluated for the security impact as early as possible in the development cycle. However, when doing so, you must use a tool that doesn’t have a noise issue. Otherwise, developers will reject it like the body rejects a foreign organ.
With that in mind, we integrated Cloudrail into our CI/CD pipeline, as shown below.
When Cloudrail was initially deployed, it was in “learning mode” (default behavior) so it was ‘invisible’ to the developers. The native integration between Cloudrail and the DevOps tools have also made it easier for developers to adopt Cloudrail.
With Cloudrail, we fully automated security reviews. The end result is that we can now enforce certain security behavior without reviewing Terraform code manually. Besides, eating our own dog food would only improve our own product.
Key Learnings and Best Practices
Far too often security efforts concentrate on the technology; while essentially neglecting the people and process aspect of the initiative. Instead of prioritizing one dimension over the others, we took the approach to consider all three dimensions: people, process and technology (in that order.) We truly believe the mullti-dimensional approach is the reason why we were able to make a successful transformation.
People – change always starts with people
A few noteworthy best practices to share:
- Empathize with the developer side more
Being a security centric company, we fundamentally believe security is everything. To ensure we strike a balance between the developer and the security side of the house, we always put ourselves in the developer’s shoes. We collaborated with the developers to introduce security controls at a pace that makes sense to them.
- Have a partner on the developer side to reflect back on a decision
For example, when we transitioned to AWS Organization with the sprawling account structure, a question was raised – how would we safekeep all the credentials with so many accounts. We ended up using AWS SSO (integrated with Google Workspaces for Identity) and aws-vault to bridge the gap.
Be sure you have a partner on the developer side, together you can reflect back if a decision made has a negative impact, and together you can take corrective actions.
- Establish shared goals between Developers and Security teams
We learned from many customers that a common goal between the security and development side of the house was often missing. So early in our journey, we established our shared goal – add security controls without blocking developers.
Net net, maintain the pace of development for the company’s success, while always watching out for the safety of our customers’ data.
With empathy in mind, we followed these best practices to implement the security controls:
- Start with visibility
We integrated the Cloudrail container into our CI/CD workflow to gain insight into our security posture. The integration had no impact to our pipeline initially, since the default behavior was in learning mode. We had twelve violations as opposed to hundreds with two of them being critical. The haystack was a lot smaller so finding the needle was much easier.
- Evaluate where you can add security controls without getting in the way
For example, it was an easy decision to enable SAML authentication since it was so simple with Google, and we use Gmail, Google Drive, etc. already. With a simple button, the developer could sign in. Throughout the entire process, we always think of the user experience.
- Enforce rules with no violations first
While the inclination might be to address the violated rules first, we did not want security controls to impact the developers. Instead, we enforced rules with no violations and were critical in nature. We gradually went from zero control to having real security controls. Effectively, we were able to enforce certain behavior without impacting delivery.
- Enforce violated rules only after they’ve been resolved
We cannot emphasize enough how important it is to not block the developers.
- Do not start with your production environment
Since our top priority was to protect our customer data in the production environment, we made the wrong assumption that the production environment was the only place we needed to enforce security policies. However, the DevOps’ general approach is that nothing should fail in production as it was the last gate. Therefore, Cloudrail shouldn’t be catching security violations when deploying to production. It should always catch them earlier, and only re-verify on the way to production. We should have started with the Developer environment first, followed by the pre-production environment, and eventually the production environment.
- Achieving zero violations for every newly added security control. In other words, making sure that each control that we enable does not break the existing pipeline.
The very first time we enforced the security policy, the pipeline was not impacted, because the violations were addressed while we were in “learning mode”. The same result occurred for subsequent newly added security controls. This is a proof point that we did not enable security controls in a silo and we have collectively implemented them in our environment without hampering speed, agility and other DevOps tenets. This is by far the best way to measure a successful implementation and a must have metric. As time progresses, Cloudrail will let developers know should a security violation occur and stop the pipeline if it is severe.
- With 10 times less security issues to investigate, it translates to significant time saving for the security team.
- Security is ingrained into our DevOps practice in a fully automated fashion. Every time an attempt was made to change our infrastructure, the security impact would be automatically evaluated. Effectively, we have successfully reduced the possibility of cyber threats on an ongoing basis and significantly improved our organization security posture.
At Indeni, I have a dual role – I’m both the CEO and the CISO, due to my background in security. I need to be able to sleep well at night, knowing our customers’ data is protected. Using an automated IaC security analysis tool like Cloudrail, helps me know we’re doing what we need to for our customers.Yoni Leitersdorf, CEO and Founder of Indeni
My team and I have aggressive release targets to meet. We must make sure nothing slows us down and we can deliver new features at the pace we’re expected to. Cloudrail is what I call “the invisible guardrails”, it has been a great way to ensure we don’t slow down, but still meet the organization’s security requirements.Alon Ashkenazi, VP of Engineering