Testing Infrastructure as Code

Testing infrastructure code prevents costly production failures, validates configurations before deployment, and enables confident changes. Comprehensive testing strategies combine static analysis, unit tests, integration tests, and compliance validation.

Why Test Infrastructure Code

Infrastructure failures disrupt services and impact customers. Testing catches issues before production deployment.

Cost of Failures

Infrastructure mistakes have serious consequences:

# Dangerous change without testing
resource "aws_security_group" "api" {
  ingress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]  # Opens all ports to internet!
  }
}

Without testing, this security group exposes all ports publicly, creating major security vulnerability.

Confidence in Changes

Testing enables safe refactoring and improvements:

# Before refactoring
resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  
  user_data = <<-EOF
    #!/bin/bash
    apt-get update
    apt-get install -y nginx
    systemctl start nginx
  EOF
}

# After refactoring with modules
module "web_server" {
  source = "./modules/web-server"
  
  ami_id        = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  install_nginx = true
}

Tests verify the refactored module behaves identically to the original configuration.

graph TD
    A[Write Infrastructure Code] --> B[Run Tests]
    B --> C{Tests Pass?}
    C -->|No| D[Fix Issues]
    D --> A
    C -->|Yes| E[Deploy to Staging]
    E --> F{Staging Validation}
    F -->|Issues Found| D
    F -->|Success| G[Deploy to Production]
    
    style C fill:#fff4e6
    style G fill:#d4f1d4

Static Analysis

Static analysis examines code without executing it, catching syntax errors, security issues, and best practice violations.

Syntax Validation

Verify code syntax correctness:

# Terraform syntax validation
terraform fmt -check
terraform validate

# Example output on success:
# Success! The configuration is valid.

# Example output on error:
# Error: Unsupported argument
#   on main.tf line 15:
#   instance_typ = "t3.medium"
# An argument named "instance_typ" is not expected here.
# Did you mean "instance_type"?

Syntax validation catches typos and structural errors immediately.

Security Scanning

Identify security vulnerabilities in configuration:

# tfsec - Terraform security scanner
tfsec infrastructure/

# Example findings:
# Result #1 CRITICAL Security group rule allows ingress from 0.0.0.0/0
#   infrastructure/security.tf:10-15
#   
#   10 | resource "aws_security_group_rule" "allow_all" {
#   11 |   type              = "ingress"
#   12 |   from_port         = 0
#   13 |   to_port           = 65535
#   14 |   cidr_blocks       = ["0.0.0.0/0"]
#   15 | }
#   
#   Impact: Unrestricted network access increases attack surface
#   Resolution: Set explicit CIDR blocks for source addresses

Security scanners detect common misconfigurations:

• Publicly accessible storage buckets
• Unencrypted data stores
• Overly permissive security groups
• Missing logging configurations
• Weak SSL/TLS configurations

# Checkov - Multi-cloud security scanner
checkov -d infrastructure/

# Example output:
# Check: CKV_AWS_23: "Ensure security group has description"
# FAILED for resource: aws_security_group.api
# File: /security.tf:5-10
# Guide: https://docs.bridgecrew.io/docs/networking_1

Policy as Code

Enforce organizational standards:

// Policy-style validation in C# for Terraform plan JSON
var requiredTags = new[] { "Environment", "Owner", "CostCenter" };
foreach (var resource in terraformPlan.ResourceChanges.Where(r => r.Type == "aws_instance"))
{
    foreach (var tag in requiredTags)
    {
        if (!resource.Change.After.Tags.ContainsKey(tag))
        {
            throw new Exception($"Missing required tag: {tag}");
        }
    }
}

Policy enforcement prevents non-compliant infrastructure:

# This would fail policy
resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  # Missing required tags!
}

# This passes policy
resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t3.medium"
  
  tags = {
    Environment = "production"
    Owner       = "platform-team"
    CostCenter  = "engineering"
  }
}

Open Policy Agent (OPA) provides flexible policy language:

# OPA policy for Kubernetes
package kubernetes.admission

deny[msg] {
  input.request.kind.kind == "Pod"
  not input.request.object.spec.containers[_].resources.limits.memory
  msg := "Containers must have memory limits"
}

deny[msg] {
  input.request.kind.kind == "Service"
  input.request.object.spec.type == "LoadBalancer"
  not input.request.object.metadata.annotations["service.beta.kubernetes.io/aws-load-balancer-internal"]
  msg := "LoadBalancer services must be internal"
}

Unit Testing

Unit tests validate individual components in isolation.

Testing Terraform Modules

Test modules with Terratest (Go):

public class VpcModuleTests
{
    [Fact]
    public async Task VpcModule_ShouldCreateExpectedOutputs()
    {
        var result = await TerraformRunner.InitApplyAsync("../modules/vpc", new Dictionary<string, string>
        {
            ["cidr_block"] = "10.0.0.0/16",
            ["environment"] = "test"
        });

        Assert.Equal("10.0.0.0/16", result.Outputs["vpc_cidr"]);
        Assert.Equal(3, result.OutputLists["subnet_ids"].Count);
    }
}

Run tests:

cd test
go test -v -timeout 30m

Testing Kubernetes Manifests

Validate Kubernetes resources:

public class KubernetesManifestTests
{
  [Fact]
  public void Deployment_ShouldHaveAtLeastThreeReplicas()
  {
    var deployment = ManifestLoader.Load("../k8s/deployment.yaml", "Deployment");
    Assert.True(deployment.Spec.Replicas >= 3, "Production deployments must have at least 3 replicas");
  }

  [Fact]
  public void Deployment_ShouldHaveResourceLimits()
  {
    var deployment = ManifestLoader.Load("../k8s/deployment.yaml", "Deployment");
    var container = deployment.Spec.Template.Spec.Containers.First();
    Assert.NotNull(container.Resources?.Limits?.Cpu);
    Assert.NotNull(container.Resources?.Limits?.Memory);
  }

  [Fact]
  public void Service_ShouldUseClusterIP()
  {
    var service = ManifestLoader.Load("../k8s/deployment.yaml", "Service");
    Assert.Equal("ClusterIP", service.Spec.Type);
  }
}

Run Kubernetes tests:

pytest test_deployment.py -v

Helm Chart Testing

Test Helm charts with helm unittest:

# charts/api-service/tests/deployment_test.yaml
suite: test deployment
templates:
  - deployment.yaml
tests:
  - it: should set replica count from values
    set:
      replicaCount: 5
    asserts:
      - equal:
          path: spec.replicas
          value: 5
  
  - it: should set resource limits
    asserts:
      - exists:
          path: spec.template.spec.containers[0].resources.limits
      - exists:
          path: spec.template.spec.containers[0].resources.requests
  
  - it: should use correct image
    set:
      image:
        repository: myapp
        tag: v2.0.0
    asserts:
      - equal:
          path: spec.template.spec.containers[0].image
          value: myapp:v2.0.0

Run Helm tests:

helm unittest charts/api-service

Integration Testing

Integration tests validate infrastructure in real environments.

End-to-End Testing

Test complete infrastructure stacks:

public class InfrastructureIntegrationTests
{
    [Fact]
    public async Task CompleteInfrastructure_ShouldRespondAndExposeDatabase()
    {
        var result = await TerraformRunner.InitApplyAsync("../infrastructure/staging");
        using var client = new HttpClient();

        var loadBalancerUrl = result.Outputs["load_balancer_url"];
        var response = await client.GetAsync(loadBalancerUrl);
        Assert.Equal(HttpStatusCode.OK, response.StatusCode);

        var databaseEndpoint = result.Outputs["database_endpoint"];
        Assert.False(string.IsNullOrWhiteSpace(databaseEndpoint));
    }
}

Contract Testing

Verify infrastructure meets application requirements:

# infrastructure-contract.yaml
version: 1.0

requirements:
  networking:
    - name: VPC exists
      resource: aws_vpc.main
      assertions:
        - cidr_block_size >= 16
        - enable_dns_hostnames == true
    
    - name: Public subnets available
      resource: aws_subnet.public
      assertions:
        - count >= 2
        - map_public_ip_on_launch == true
  
  database:
    - name: Database encrypted
      resource: aws_db_instance.main
      assertions:
        - storage_encrypted == true
        - multi_az == true
    
    - name: Automated backups enabled
      resource: aws_db_instance.main
      assertions:
        - backup_retention_period >= 7
        - backup_window != null

Validate contracts:

public static class InfrastructureContractValidator
{
  public static List<string> Validate(string contractFile, string terraformStateFile)
  {
    var contract = ContractLoader.Load(contractFile);
    var state = TerraformStateLoader.Load(terraformStateFile);
    var failures = new List<string>();

    foreach (var category in contract.Requirements)
    {
      foreach (var req in category.Value)
      {
        var resource = state.FindResource(req.Resource);
        foreach (var assertion in req.Assertions)
        {
          if (!AssertionEngine.Evaluate(resource, assertion))
          {
            failures.Add($"{req.Name}: {assertion}");
          }
        }
      }
    }

    return failures;
  }
}

Compliance Testing

Verify infrastructure meets regulatory requirements.

Compliance Frameworks

Test against standards:

# InSpec - Compliance testing framework
inspec exec aws-compliance-profile \
  --target aws://us-east-1 \
  --reporter cli json:compliance-report.json

# Example InSpec test
describe aws_s3_bucket('production-data') do
  it { should exist }
  it { should have_default_encryption_enabled }
  it { should_not be_public }
  its('bucket_policy') { should include 'AES256' }
end

describe aws_rds_instance('production-database') do
  it { should be_encrypted }
  it { should have_multi_az_enabled }
  its('backup_retention_period') { should be >= 30 }
end

Custom Compliance Rules

Enforce organization-specific requirements:

public class AwsComplianceTests
{
  [Fact]
  public async Task AllS3Buckets_ShouldHaveEncryptionEnabled()
  {
    var s3 = new Amazon.S3.AmazonS3Client();
    var buckets = await s3.ListBucketsAsync();

    foreach (var bucket in buckets.Buckets)
    {
      var encryption = await s3.GetBucketEncryptionAsync(new Amazon.S3.Model.GetBucketEncryptionRequest
      {
        BucketName = bucket.BucketName
      });
      Assert.NotNull(encryption.ServerSideEncryptionConfiguration);
    }
  }

  [Fact]
  public async Task RdsInstances_ShouldKeepThirtyDayBackups()
  {
    var rds = new Amazon.RDS.AmazonRDSClient();
    var instances = await rds.DescribeDBInstancesAsync();

    foreach (var instance in instances.DBInstances)
    {
      Assert.True(instance.BackupRetentionPeriod >= 30,
        $"Instance {instance.DBInstanceIdentifier} has insufficient backup retention: {instance.BackupRetentionPeriod} days");
    }
  }
}

public static class Ec2TagPolicyTests
{
  [Fact]
  public static async Task Ec2Instances_ShouldHaveRequiredTags()
  {
    var ec2 = new Amazon.EC2.AmazonEC2Client();
    var response = await ec2.DescribeInstancesAsync();
    var requiredTags = new[] { "Environment", "Owner", "CostCenter" };

    foreach (var reservation in response.Reservations)
    {
      foreach (var instance in reservation.Instances)
      {
        var tags = instance.Tags.ToDictionary(t => t.Key, t => t.Value);

        foreach (var requiredTag in requiredTags)
        {
          Assert.True(tags.ContainsKey(requiredTag),
            $"Instance {instance.InstanceId} missing tag: {requiredTag}");
        }
      }
    }
  }
}

Run compliance tests:

pytest compliance_tests.py -v --tb=short

Testing in CI/CD Pipeline

Integrate testing into continuous integration workflows.

Complete Pipeline

# .github/workflows/infrastructure-test.yaml
name: Infrastructure Testing
on:
  pull_request:
    paths:
    - 'infrastructure/**'

jobs:
  static-analysis:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
    
    - name: Terraform Format Check
      run: terraform fmt -check -recursive
      working-directory: infrastructure
    
    - name: Terraform Validate
      run: |
        terraform init -backend=false
        terraform validate
      working-directory: infrastructure
    
    - name: Run tfsec
      uses: aquasecurity/tfsec-action@v1.0.0
      with:
        working_directory: infrastructure
    
    - name: Run Checkov
      uses: bridgecrewio/checkov-action@master
      with:
        directory: infrastructure
        framework: terraform
  
  unit-tests:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Go
      uses: actions/setup-go@v4
      with:
        go-version: '1.21'
    
    - name: Run Terratest
      run: |
        cd test
        go test -v -timeout 30m
      env:
        AWS_DEFAULT_REGION: us-east-1
  
  integration-tests:
    runs-on: ubuntu-latest
    needs: [static-analysis, unit-tests]
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
    
    - name: Deploy to Test Environment
      run: |
        terraform init
        terraform apply -auto-approve
      working-directory: infrastructure/test
      env:
        AWS_DEFAULT_REGION: us-east-1
    
    - name: Run Integration Tests
      run: |
        cd test
        go test -v -run Integration
      env:
        AWS_DEFAULT_REGION: us-east-1
    
    - name: Cleanup Test Environment
      if: always()
      run: terraform destroy -auto-approve
      working-directory: infrastructure/test
  
  compliance-check:
    runs-on: ubuntu-latest
    needs: integration-tests
    steps:
    - name: Checkout
      uses: actions/checkout@v3
    
    - name: Setup Python
      uses: actions/setup-python@v4
      with:
        python-version: '3.11'
    
    - name: Install Dependencies
      run: |
        pip install pytest boto3 pyyaml
    
    - name: Run Compliance Tests
      run: pytest compliance_tests.py -v
      env:
        AWS_DEFAULT_REGION: us-east-1

graph TB
    A[Pull Request] --> B[Static Analysis]
    B --> C{Pass?}
    C -->|No| D[Report Failures]
    C -->|Yes| E[Unit Tests]
    E --> F{Pass?}
    F -->|No| D
    F -->|Yes| G[Integration Tests]
    G --> H{Pass?}
    H -->|No| D
    H -->|Yes| I[Compliance Check]
    I --> J{Pass?}
    J -->|No| D
    J -->|Yes| K[Approve PR]
    
    style K fill:#d4f1d4
    style D fill:#ffe6e6

Testing Best Practices

Effective infrastructure testing requires discipline and structure.

Test Pyramid

Balance different test types:

         /\
        /  \  E2E Tests (Few)
       /____\
      /      \  Integration Tests (Some)
     /________\
    /          \  Unit Tests (Many)
   /__Static____\
  /   Analysis   \  (Most)

• Static Analysis: Run on every commit, fast feedback
• Unit Tests: Test modules independently, quick execution
• Integration Tests: Validate complete stacks, slower
• E2E Tests: Full system validation, slowest and most expensive

Fail Fast

Order tests from fastest to slowest:

# Run quick static analysis first
stages:
  - lint
  - unit-test
  - integration-test
  - compliance

# Static analysis completes in seconds
lint:
  stage: lint
  script:
    - terraform fmt -check
    - terraform validate
    - tfsec

# Unit tests complete in minutes
unit-test:
  stage: unit-test
  script:
    - go test -v -short

# Integration tests take 10-30 minutes
integration-test:
  stage: integration-test
  script:
    - go test -v -run Integration

Isolated Test Environments

Use dedicated environments for testing:

# test/main.tf
terraform {
  backend "s3" {
    bucket = "terraform-state-test"
    key    = "test/infrastructure-${var.test_id}.tfstate"
    region = "us-east-1"
  }
}

resource "random_id" "test_suffix" {
  byte_length = 4
}

module "test_infrastructure" {
  source = "../modules/complete-stack"
  
  environment = "test-${random_id.test_suffix.hex}"
  # Use small, cheap resources for testing
  instance_type = "t3.small"
  database_size = "db.t3.small"
}

Common Pitfalls

Testing Only Happy Path: Tests must cover failure scenarios. Test what happens when resources don't exist, permissions are denied, or dependencies fail.

No Cleanup: Failed tests leave orphaned resources. Always use defer terraform.Destroy() in Go tests or cleanup blocks in other frameworks.

Slow Test Suites: Hour-long test runs discourage running tests. Optimize by parallelizing tests and using smaller resource sizes.

Ignoring Test Failures: Treating test failures as acceptable defeats the purpose. Fix failures immediately or delete unreliable tests.

Key Takeaways

• Test infrastructure code to prevent costly production failures, validate configurations, and enable confident changes
• Static analysis catches syntax errors, security issues, and policy violations without deploying infrastructure
• Unit tests validate individual modules in isolation, using frameworks like Terratest for Terraform or pytest for Kubernetes
• Integration tests verify complete infrastructure stacks work correctly, testing real deployments in isolated environments
• Compliance testing ensures infrastructure meets regulatory requirements using frameworks like InSpec or custom Python tests
• Integrate all testing into CI/CD pipelines, running static analysis first, then unit tests, integration tests, and finally compliance checks
• Follow testing best practices: maintain test pyramid balance, fail fast with quick tests first, use isolated test environments, and always clean up test resources