Validating Configuration of a Growing Cloud Platform

“Infrastructure as Code” is a process of provisioning complex computing platforms in a reliable and repeatable way via configuration files and scripting tools. Cloud computing services such as AWS and Azure expose rich, free APIs that allow their customers to manage fleets of virtual servers without ever having to log into a web-based portal.

Speed and Cost Savings

Infrastructure as Code brings some immediately noticeable benefits – the amount of manual effort required to provision a production environment can be drastically decreased, bringing instant cost savings in terms of effort. Speed is another benefit – a script will always be faster than a human at provisioning servers via a GUI.

Risk Reduction

The most important benefit (in my opinion!) is the reduction of risk. Automation removes the risk associated with human error, leading to less downtime and time wasted on troubleshooting. Code reviews of the infrastructure provisioning scripts help ensure that security standards are adhered to, and automated tests can be run on the infrastructure that the scripts create.

Enter Terraform!

Terraform is a tool created by Hashicorp that is designed to enable engineers to deploy Infrastructure as Code into variety of cloud providers, using the Hashicorp Configuration Language (HCL).

resource "aws_instance" "web" {
	ami = "ami-171987"
	instance_type = "t2.micro"
	tags {
		Name = "HelloWorld"
	}
}

The above snippet of code will create one EC2 instance, and will set the properties of the instance to whatever is defined in the resource block.

Using that same syntax, it’s possible to set up load balancers; security groups; databases – pretty much any resource type that your chosen cloud provider offers API access to! Hashicorp has kindly provided comprehensive documentation to get you started.

Maintaining best Practices

A typical cloud-based infrastructure may contain hundreds of resources that map onto every facet of your platform, especially once the networking aspect of the platform has been mapped out.

At DevOpsGuys we plan our infrastructure design with a “Security First” mindset. Any infrastructure that we design is built from the ground up with security baked in. When we deploy anything into AWS, we have some internal standards that we must abide by in order to keep the infrastructure secure;

1. All cloud databases are encrypted
2. All storage volumes are encrypted
3. All resources are properly tagged

The declarative nature of Terraform ensures that as long as the configuration files are written correctly, all 3 of the mentioned standards will be enforced when the infrastructure is built or updated.

Out of the box, Terraform only has syntax checking to validate what you’re designing. Any code that’s written should be peer reviewed to ensure that the code is written correctly and conforms to standards.

Traditionally, this would be done by a human – which introduces the risk of human error.

Let the Computer do the Work

While I was manually validating that over 50 security groups had the correct tag names, I had an idea – why not write a tool that can automatically check this for me? The terraform_validate is the product of that original idea.

terraform_validate is a python module that aims to reduce the need for human validation of Terraform configuration files.

resource "aws_ebs_volume" "foo" {
	size = 40
}
resource "aws_ebs_volume" "bar" {
	size = 40
	encrypted = True
}

Although both resources are valid within Terraform, the volume resource “foo” is in violation of rule #2 “all storage volumes are encrypted”.

Using the terraform_validate module, it’s straightforward to write an automated tests that checks all the aws_ebs_volume resources and validates that they adhere to standards.

import unittest
import os
import terraform_validate
class TestEncryptionAtRest(unittest.TestCase):
	def test_aws_ebs_volumes_are_encrypted(self):
		self.path = os.path.join(os.path.dirname(os.path.realpath(__file__)),"terraform")
		validator = terraform_validate.Validator(self.path)
		# By default the validator will not fail if the property in question is missing
		validator.error_if_property_missing()
		validator.resources('aws_ebs_volume').property('encrypted').should_equal(True)
if __name__ == '__main__':
	suite = unittest.TestLoader().loadTestsFromTestCase(TestEncryptionAtRest)
	unittest.TextTestRunner(verbosity=0).run(suite)

When the test is executed, the following error is shown;

AssertionError: [aws_ebs_volume.foo.encrypted] should be 'True'. Is: 'False'
======================== 1 failed in 0.07 seconds =========================

Test-driven Infrastructure

Before you sit down and start writing Terraform configuration files, write down (in plain English) what standards you want to conform to. When you have the list of standards, start to create your tests.

It’s recommended that any standards-checking tests are created before any Terraform configuration is written – this reduces the amount of rework that may need to be done as a result of a new test being introduced.

As long as you’re testing one of the following conditions, then terraform_validate will be able to check it;

• A resource property does/doesn’t exist
• A resource property value does/doesn’t match an expected value
• A resource property value matches a regex

The way in which a test is written is mostly self-documenting, however it’s strongly recommended that the test function name is descriptive. Relying heavily on regex matching can make a test confusing if the developers are are not given a context to explain what the test is checking for.

I would also recommend that your validation tests get run on each check-in of Terraform configuration by your Continuous Integration server.

---

Hopefully this blog post will be enough to get you started with test-driven Terraform design! Feel free to make modifications to the terraform_validate library and create a Pull Request.

— Edmund Dipple, DevOps Engineer, DevOpsGroup (aka @elmundio87)