Version Control for Infrastructure

Version control transforms infrastructure management from ad-hoc changes to a structured, auditable process. Applying Git workflows to infrastructure code enables collaboration, change tracking, and safe deployments.

Why Version Control Matters

Version control provides critical capabilities for infrastructure management that manual processes cannot match.

Change Tracking

Every infrastructure modification creates a permanent record:

# View infrastructure change history
git log --oneline infrastructure/production/

# Example output:
# a3f8b21 Increase RDS instance size for performance
# 7d2c489 Add CloudFront distribution for static assets
# e1b4532 Enable automated backups for production database
# 9c7e6d1 Add monitoring alerts for API latency

# See what changed in specific commit
git show a3f8b21

# Output shows the actual diff:
-  instance_class = "db.t3.medium"
+  instance_class = "db.r5.large"

This history answers critical questions:

Collaboration

Multiple engineers work on infrastructure simultaneously without conflicts:

# Engineer A works on networking
git checkout -b add-vpc-endpoints
# Make changes to VPC configuration
git commit -m "Add VPC endpoints for S3 and DynamoDB"
git push origin add-vpc-endpoints

# Engineer B works on database independently
git checkout -b upgrade-database
# Make changes to database configuration
git commit -m "Upgrade PostgreSQL to version 14"
git push origin upgrade-database

# Both changes reviewed and merged independently

Branches isolate work-in-progress, preventing incomplete changes from affecting others.

Rollback Capability

Revert infrastructure to any previous state:

# Problem detected after deployment
git log --oneline -5
# a3f8b21 (HEAD -> main) Increase RDS instance size
# 7d2c489 Add CloudFront distribution
# e1b4532 Enable automated backups

# Rollback to previous version
git revert a3f8b21
git push origin main

# Or rollback to specific point in time
git checkout 7d2c489 -- infrastructure/database/
git commit -m "Rollback database configuration to pre-upgrade state"
git push origin main

Rollback provides safety net for experimentation and quick recovery from issues.

graph TD
    A[Working State] --> B[Change Applied]
    B --> C{Problem Detected?}
    C -->|No| D[Continue Forward]
    C -->|Yes| E[Git Revert]
    E --> F[Previous Working State]
    
    style E fill:#ffe6e6
    style F fill:#d4f1d4

Audit Trail

Compliance requirements demand change documentation:

# Generate audit report for date range
git log --since="2024-01-01" --until="2024-03-31" \
  --pretty=format:"%h - %an, %ar : %s" \
  infrastructure/production/

# Output provides compliance evidence:
# a3f8b21 - John Doe, 2 weeks ago : Increase RDS instance size
# 7d2c489 - Jane Smith, 1 month ago : Add CloudFront distribution
# e1b4532 - Bob Johnson, 2 months ago : Enable automated backups

# Export detailed audit log
git log --since="2024-01-01" --format=fuller --stat \
  infrastructure/production/ > audit-q1-2024.txt

Repository Structure

Organizing infrastructure code effectively supports maintainability and clarity.

Monorepo Approach

Single repository contains all infrastructure:

infrastructure/
├── environments/
│   ├── production/
│   │   ├── networking/
│   │   │   └── main.tf
│   │   ├── compute/
│   │   │   └── main.tf
│   │   ├── database/
│   │   │   └── main.tf
│   │   └── monitoring/
│   │       └── main.tf
│   ├── staging/
│   │   ├── networking/
│   │   ├── compute/
│   │   ├── database/
│   │   └── monitoring/
│   └── development/
│       ├── networking/
│       ├── compute/
│       └── database/
├── modules/
│   ├── vpc/
│   ├── eks-cluster/
│   ├── rds-database/
│   └── monitoring-stack/
├── policies/
│   ├── security-policies/
│   └── compliance-checks/
└── scripts/
    ├── deploy.sh
    └── validate.sh

Monorepo benefits:

Monorepo challenges:

Multi-Repo Approach

Separate repositories for different concerns:

# Repository: infrastructure-networking
networking/
├── production/
├── staging/
└── modules/

# Repository: infrastructure-compute
compute/
├── production/
├── staging/
└── modules/

# Repository: infrastructure-database
database/
├── production/
├── staging/
└── modules/

Multi-repo benefits:

Multi-repo challenges:

Environment-Based Structure

Organize primarily by environment:

infrastructure-production/
├── networking/
├── compute/
├── database/
└── monitoring/

infrastructure-staging/
├── networking/
├── compute/
└── database/

infrastructure-development/
├── compute/
└── database/

This structure provides strong environment isolation but increases duplication.

Branching Strategies

Different branching models suit different team sizes and release cadences.

Trunk-Based Development

Work primarily on main branch with short-lived feature branches:

# Create short-lived feature branch
git checkout -b add-monitoring-alerts
# Make focused changes
git commit -m "Add CloudWatch alarms for API latency"
# Merge quickly (same day)
git checkout main
git merge add-monitoring-alerts
git push origin main

Trunk-based development works well for:

gitGraph
    commit
    commit
    branch feature-1
    checkout feature-1
    commit
    commit
    checkout main
    merge feature-1
    commit
    branch feature-2
    checkout feature-2
    commit
    checkout main
    merge feature-2
    commit

GitFlow

Structured branching with dedicated release branches:

# Development work on develop branch
git checkout develop
git checkout -b feature/add-cdn
# Implement feature
git commit -m "Add CloudFront CDN configuration"
git checkout develop
git merge feature/add-cdn

# Create release branch
git checkout -b release/v1.2.0 develop
# Final testing and fixes
git commit -m "Update production replica counts"

# Merge to main for production
git checkout main
git merge release/v1.2.0
git tag v1.2.0

# Merge back to develop
git checkout develop
git merge release/v1.2.0

GitFlow suits:

Environment Branches

Branches represent environments:

# Production branch (protected)
git checkout production
# Only promoted changes allowed

# Staging branch
git checkout staging
# Changes deployed to staging first

# Development branch
git checkout development
# Active development happens here

# Promotion flow
git checkout staging
git merge development
git push origin staging
# After validation
git checkout production
git merge staging
git push origin production

Environment branches provide:

Drawbacks:

Code Review Practices

Infrastructure changes require careful review to prevent outages.

Pull Request Template

Standardize review information:

## Infrastructure Change Request

### Description
Brief description of infrastructure changes

### Motivation
Why is this change necessary?

### Changes Made
- List specific resources modified
- Note any resource deletions
- Highlight security implications

### Testing
- [ ] Ran terraform plan locally
- [ ] Validated with tflint
- [ ] Tested in development environment
- [ ] Reviewed security impact

### Rollback Plan
How to revert if problems occur

### Estimated Downtime
Expected impact on running services

### Deployment Schedule
Preferred deployment time window

Templates ensure reviewers have necessary context.

Review Checklist

Systematically evaluate infrastructure changes:

### Security Review
- [ ] No hardcoded secrets in code
- [ ] Appropriate IAM permissions (least privilege)
- [ ] Network security groups properly scoped
- [ ] Encryption enabled for data at rest
- [ ] TLS configured for data in transit

### Cost Review
- [ ] Estimated monthly cost impact documented
- [ ] Resource sizing appropriate for load
- [ ] Auto-scaling configured where applicable
- [ ] Unused resources removed

### Reliability Review
- [ ] High availability configured
- [ ] Backup strategy defined
- [ ] Monitoring and alerting configured
- [ ] Disaster recovery plan updated

### Compliance Review
- [ ] Meets regulatory requirements
- [ ] Audit logging enabled
- [ ] Data residency requirements met
- [ ] Change documented for compliance

Automated Checks

Run validation automatically on pull requests:

# .github/workflows/pr-validation.yaml
name: Infrastructure PR Validation
on:
  pull_request:
    paths:
    - 'infrastructure/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout code
      uses: actions/checkout@v3
    
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
    
    - name: Terraform Format Check
      run: terraform fmt -check -recursive
    
    - name: Terraform Init
      run: terraform init -backend=false
      working-directory: infrastructure/production
    
    - name: Terraform Validate
      run: terraform validate
      working-directory: infrastructure/production
    
    - name: Run tflint
      uses: terraform-linters/setup-tflint@v3
      run: tflint --recursive
    
    - name: Security Scan
      uses: aquasecurity/tfsec-action@v1.0.0
    
    - name: Cost Estimation
      uses: infracost/actions/setup@v2
      run: |
        infracost breakdown --path=infrastructure/production \
          --format=comment --out-file=/tmp/infracost.txt
    
    - name: Post Comment
      uses: actions/github-script@v6
      with:
        script: |
          const fs = require('fs');
          const comment = fs.readFileSync('/tmp/infracost.txt', 'utf8');
          github.rest.issues.createComment({
            issue_number: context.issue.number,
            owner: context.repo.owner,
            repo: context.repo.repo,
            body: comment
          });

Automation catches common issues before human review.

Change Management Workflow

Structure the path from development to production.

Development Cycle

sequenceDiagram
    participant Dev as Developer
    participant Local as Local Environment
    participant Git as Git Repository
    participant CI as CI Pipeline
    participant Review as Code Review
    participant Stage as Staging
    participant Prod as Production
    
    Dev->>Local: Write infrastructure code
    Dev->>Local: terraform plan
    Local->>Dev: Show planned changes
    Dev->>Git: Create feature branch
    Dev->>Git: Push changes
    Git->>CI: Trigger validation
    CI->>CI: Format, validate, lint, scan
    CI->>Git: Report results
    Dev->>Review: Create pull request
    Review->>Review: Team review
    Review->>Git: Approve and merge
    Git->>Stage: Deploy to staging
    Stage->>Stage: Validate functionality
    Stage->>Prod: Promote to production

Deployment Windows

Schedule infrastructure changes carefully:

# deployment-schedule.yaml
environments:
  production:
    allowed_windows:
    - day: Tuesday
      start: "14:00"
      end: "16:00"
      timezone: "America/New_York"
    - day: Thursday
      start: "14:00"
      end: "16:00"
      timezone: "America/New_York"
    
    blackout_periods:
    - start: "2024-11-25"
      end: "2024-11-29"
      reason: "Holiday freeze"
    - start: "2024-12-20"
      end: "2025-01-02"
      reason: "End of year freeze"

Deployment windows reduce risk by:

Secrets Management

Handle sensitive values without committing them to Git.

Environment Variables

Reference secrets from environment:

# variables.tf
variable "database_password" {
  description = "Database master password"
  type        = string
  sensitive   = true
}

# Reference in resource
resource "aws_db_instance" "database" {
  password = var.database_password
}

Provide via environment variable:

export TF_VAR_database_password="SecurePassword123!"
terraform apply

Secret Management Tools

Integrate with dedicated secret stores:

# Fetch from HashiCorp Vault
data "vault_generic_secret" "database" {
  path = "secret/production/database"
}

resource "aws_db_instance" "database" {
  password = data.vault_generic_secret.database.data["password"]
}

# Fetch from AWS Secrets Manager
data "aws_secretsmanager_secret_version" "database" {
  secret_id = "production/database/master-password"
}

resource "aws_db_instance" "database" {
  password = jsondecode(data.aws_secretsmanager_secret_version.database.secret_string)["password"]
}

SOPS for Encrypted Files

Encrypt sensitive files before committing:

# Install SOPS
wget https://github.com/mozilla/sops/releases/download/v3.7.3/sops-v3.7.3.linux
sudo mv sops-v3.7.3.linux /usr/local/bin/sops
sudo chmod +x /usr/local/bin/sops

# Create secret file
cat > secrets.yaml <<EOF
database_password: SuperSecretPassword
api_key: sk-1234567890abcdef
EOF

# Encrypt with AWS KMS
sops --encrypt --kms arn:aws:kms:us-east-1:123456789:key/abc secrets.yaml > secrets.enc.yaml

# Commit encrypted file safely
git add secrets.enc.yaml
git commit -m "Add production secrets"

# Decrypt when needed
sops --decrypt secrets.enc.yaml > secrets.yaml

Common Pitfalls

Committing Secrets: Accidentally committed secrets remain in Git history forever. Use git-secrets or similar tools to prevent commits containing secrets.

No Code Review: Skipping review for "quick fixes" leads to outages. All infrastructure changes must go through review, regardless of urgency.

Ignoring Drift: Manual changes outside Git create divergence. Regularly compare actual infrastructure state with code.

Large Pull Requests: Massive infrastructure changes are difficult to review effectively. Break changes into logical, reviewable pieces.

Key Takeaways