In today’s competitive market, our success depends on how quickly we can innovate and deliver value to customers. It’s all about speeding time to market by shortening the development lifecycle while keeping software deployable. Here at Indeni, we use continuous integration (CI) and continuous delivery and deployment (CD) practice and tools to empower developers to deliver the release of features and bug fixes more frequently and reliably without a manual intervention

With the advent of Infrastructure-as-Code (e.g. Terraform), we automate Infrastructure provisioning and application code delivery as part of our CI/CD pipeline. Moreover, in order to develop fast and secure, we utilize Terraform + AWS to provision a new environment for each developer. This post explains how we at Indeni engineering utilize Terraform in our CI/CD and explain a bit about our SDLC.

Indeni CI/CD Pipeline

We first briefly describe how we have automated our application delivery with CI/CD pipelines, then we add Terraform and security to the picture.

Pull Requests

When a developer is ready to merge code changes, CI helps with merging the code changes back to our shared main development branch. Essentially, a pull request triggers a Jenkins job to run a number of quality gates (automated tests such as unit test, integration test and Lint)

 throttle(['ci']) {
   node('slave-aws') {
       try {
           stage("Checkout") {
               // use env var to give the job a meaningful name
               def new_job_name = env.BRANCH_NAME + " [" + env.CHANGE_BRANCH + " -> " + env.CHANGE_TARGET + "]"
               currentBuild.rawBuild.project.setDisplayName(new_job_name)
               currentBuild.description = env.CHANGE_BRANCH + "->" + env.CHANGE_TARGET
               currentBuild.rawBuild.project.setDescription("<a href=${env.CHANGE_URL}>Link to the PR</a>")
               cleanWs disableDeferredWipeout: true, deleteDirs: true
 
               checkout scm
           }
           stage('Run automated tests') {
               timeout(time: 15, unit: 'MINUTES') {
                   docker.withRegistry('https://<YOUR DOCKER REGISTER>', '<KEY>') {
                       sh 'run-tests.sh' // contains all the needed steps to run your automated tests
                       sh 'docker system prune -f'
                   }
               }
           }
       }
       catch(exc) {
           currentBuild.result='FAILURE'
           echo "ERROR: Something broken... : " + exc.toString()
       }
       finally {
           cobertura coberturaReportFile: 'coverage-results/*.xml'
           junit allowEmptyResults: true, testResults: 'test-results/*.xml'
       }
   }
}

These gates must be passed to ensure the changes did not break the application. This is followed by a peer code review. Upon approval and merge, a Jenkins job is automatically triggered to start a code build. 

Build

Upon a code merge into our main development branch, Jenkins triggers a job that builds the source code and creates the needed artifact for deployment. Every merge that happens in our main development branch gets the build number as a git tag, so it will be easy to track which changes each version contains.

def CRDIR
 
throttle(['build']) {
   node('slave-aws') {
       try {
           stage("Checkout") {
               //similar to ci checkout state
               CRDIR = env.WORKSPACE + "/<YOUR SOURCE CODE FOLDER>"
           }
           state("create git tag") {
               dir(CRDIR) {
                   sshagent (credentials: ['<KEY>']) {
                       sh '''#!/bin/bash -ex
                           export version_tag=''' + env.buildVersion + '''
                           git tag ${version_tag}
                           git push origin ${version_tag}
                       '''
                   }
           }
           stage("Run automated tests") {
               //similar to ci run automated tests state
           }
           stage("Create artifacts") {
               //create lambda artifact
               sh 'zip -r9 ${target_folder}/lambda.zip src'
               //create docker artifact
               docker.withRegistry('https://<YOUR DOCKER REGISTER>', '<USER>') {
                   sh '''
                       docker build -f <DOCKER FILE> -t <TAG>
                       docker push <TAG>
                       docker rm <TAG>
                   '''
               }
           }
       }
       catch(exc) {
           currentBuild.result='FAILURE'
           echo "ERROR: Something broken... : " + exc.toString()
       }
       finally {
           dir(CRDIR) {
               cobertura coberturaReportFile: 'coverage-results/*.xml'
               junit allowEmptyResults: true, testResults: 'test-results/*.xml'
           }
       }
   }
}

The created artifacts are yet to be approved for production use, they first need to be deployed on a development/pre-production environment to make sure that: 

  1. The deployment of the new version succeeded; and 
  2. The version passed a set of regression tests to ensure that the code functions as expected.
Related Article  Indeni Cloudrail Case Study: Eating Dogfood and Enjoying it

Deployment

The deployment to the development environment triggers automatically immediately after we have a new artifact version. We are still running everything in our development environment as we still need to verify that version is deployable and there are no regressions. 

As we are utilizing terraform to provision our infrastructure, a deployment is simply running terraform plan and terraform apply of the IaC code.

def CRDIR
 
node('slave-aws') {
   stage("Checkout") {
       checkout scm
       CRDIR = env.WORKSPACE + "/<YOUR SOURCE CODE FOLDER>"
       //...
 
   }
   stage("Terraform plan") {
       dir(CRDIR + "/<YOUR TERRAFORM FOLDER>") {
           //Preview changes before applying and approve them
           sh 'terragrunt plan --terragrunt-non-interactive ${common_params} -out=${WORKSPACE}/terraform.plan'
       }
   }
   // can stop here for manual review the plan file
   stage("Terraform apply") {
       dir(CRDIR + "/<YOUR TERRAFORM FOLDER>") {
           //Provision infrastructure changes
           sh 'terragrunt apply -auto-approve --terragrunt-non-interactive ${WORKSPACE}/terraform.plan'
       }
   }
   stage("Test regression") {
       try {
           dir(CRDIR + "/<YOUR REGRESSION TEST FOLDER>") {
               sh 'run-smoke-tests-in-parallel.sh'
           }
       } catch (exc) {
           currentBuild.result = "UNSTABLE"
           junit allowEmptyResults: true, testResults: 'test-results/*.xml'
       }
   }           
}

Once the deployment and the extensive regression tests pass, the artifacts can be used for pre-production and later on production.

As we don’t (yet) have UI regression tests, the deployment to pre-production and then to production is coordinated with the team, as we need to manually verify the UI (we are working to automate UI testing as well).

How does Terraform Improve CI/CD?

1. Configure the CI/CD pipeline to spin a developer environment for testing – improving developer productivity

Let’s imagine we have to develop a new service; the developer writes and tests the new code. Before merging the code to the shared development branch, the developer needs to test his feature end to end in a production-like environment. In the past, a separate team was responsible for the cloud infrastructure so coordination needed to happen between teams, to get an environment for testing and to ensure that no one is stepping on each other’s toes. Alternatively, the developer modifies the infrastructure and the source code using the AWS Management Console UI (or any other cloud provider) but there is no guarantee that it reflects the latest version of the infrastructure.

Luckly, we live in a new era, with Terraform, the developer can own the full infrastructure stack and take full control. At indeni, each developer gets his own environment that he can spin up and provision at any given moment. We configure the CI/CD pipeline to refresh the developer cloud environment, this ensures that the cloud infrastructures are similar to the one we run in production. Leveraging the Terraform code we spin up an isolated cloud infrastructure for testing within a matter of minutes. This means the developer is working with a nearly-exact replica of the production environment (without customer data)!

Terraform allows for repeatable infrastructure deployment and the infrastructure deployed always reflects the latest version. With hundreds of infrastructure changes per month, taking advantage of Terraform is the only way we can guarantee an up-to-date environment for ongoing testing. 

2. Integrating Terraform into the CI/CI pipelines

After the new service is successfully tested in the isolated environment, the developer is now ready to commit his new code into the main development environment. Let’s assume the new service requires provisioning of a new Lambda function in AWS. Instead of managing the infrastructure change in its own silo, it makes sense that the updated infrastructure definition is version-controlled and stored alongside the version with the new service code. 

Related Article  Announcing Support for Azure

As part of the delivery process, we integrate Terraform into the pipeline. This allows all changes to the infrastructure to be tracked, along with the source code. With infrastructure treated like software, the infrastructure change can be tested early in the process. Once the change is deployed in the main development environment, regression tests are run automatically to ensure that the infrastructure change hasn’t broken the application. 

After a new version is approved in the development environment, we can choose to release the new service to production along with the needed infrastructure update to production. To do that, we manually initiate a Jenkins job to run Terraform plan/apply, followed automatically by regression tests. Effectively, we have automated the release of an application to production along with the necessary infrastructure updates to support the latest version of the application.

Adding security checks to Improve CI/CD

To further improve the CI/CD pipeline, we implemented Indeni Cloudrail to ensure continuous security and compliance. Think about Cloudrail as an integration test for security. You want to identify security “bugs” as early as possible in your development process. Since our developer writes the Terraform code and they have full responsibility for production, it’s important that they will be aware of the security risks that they might introduce to the system when changing the infrastructure code. 

def CRDIR
 
node('slave-aws') {
   stage("Checkout") {
       checkout scm
       CRDIR = env.WORKSPACE + "/<YOUR SOURCE CODE FOLDER>"
       //...
 
   }
   stage("Terraform plan") {
       dir(CRDIR + "/<YOUR TERRAFORM FOLDER>") {
           //Preview changes before applying and approve them
           sh 'terragrunt plan --terragrunt-non-interactive ${common_params} -out=${WORKSPACE}/terraform.plan'
       }
   }
   state("Run Cloudrail for security tests") {
       dir(CRDIR + "/<YOUR TERRAFORM FOLDER>") {
           cr_image = docker.image("indeni/cloudrail-cli")
           cr_image.pull()
           dir(env.WORKSPACE) {
               cr_image.inside("--entrypoint=''") { c ->
                   withCredentials('... api_key ...') {
                       sh '''#!/bin/bash -exu
                           cloudrail run --api-key ${api_key} \
                               --execution-source-identifier Build-${buildVersion} \
                               --build-link ${BUILD_URL} \
                               --origin ci \
                               --tf-plan ${WORKSPACE}/terraform.plan \
                               --directory ${PWD} \
                               --auto-approve \
                               --output-format junit || {
                                   exit_code=$?
                                   cp /indeni/cli.log ${WORKSPACE}/cli.log
                                   exit $exit_code
                               }
                       '''
                   }
               }
           }
       }
   }
   // can stop here for manual review the plan file
   stage("Terraform apply") {
       dir(CRDIR + "/<YOUR TERRAFORM FOLDER>") {
           //Provision infrastructure changes
           sh 'terragrunt apply -auto-approve --terragrunt-non-interactive ${WORKSPACE}/terraform.plan'
       }
   }
   stage("Test regression") {
       try {
           dir(CRDIR + "/<YOUR REGRESSION TEST FOLDER>") {
               sh 'run-smoke-tests-in-parallel.sh'
           }
       } catch (exc) {
           currentBuild.result = "UNSTABLE"
           junit allowEmptyResults: true, testResults: 'test-results/*.xml'
       }
   }           
}

Now, let’s face it, developers usually don’t have a lot of knowledge about security, so instead of holding them back on things that they might not fully understand, we use Cloudrail with prebuilt security integration tests that keep our infrastructure safe from security vulnerability issues.

Terraform and CI/CD Better Together 

Here at Indeni, when we are talking about the best DevOps practices, we think of the Terraform and CI/CD integration being a key ingredient. Since infrastructure is defined as code with Terraform, there is no reason not to use software developments best practices. Therefore, validating planned infrastructure changes, testing infrastructure early in the development process, applying continuous delivery makes just as much sense for infrastructure as it does for application codes. We truly believe the Terraform and CI/CD integration is a must-have in order to keep your business running.