Testing in CI/CD Pipelines

What is CI/CD Testing?

CI/CD testing integrates automated tests directly into continuous integration and deployment pipelines, providing rapid feedback on code quality and functionality. Tests run automatically on every code change, catching issues early before they reach production.

Effective testing in CI/CD ensures code quality, reduces manual QA effort, and enables confident, frequent deployments.

Why Testing in CI/CD Matters

Test Pyramid

The test pyramid guides the balance of different test types in CI/CD pipelines.

graph TD
    A["Manual Exploratory Tests
Expensive, Slow
5%"] -->B["End-to-End Tests
Integration across systems
10%"]
    B -->C["Integration Tests
Component interactions
20%"]
    C -->D["Unit Tests
Individual functions
65%"]
    
    style A fill:#ffcccc
    style B fill:#ffd9b3
    style C fill:#fff3e0
    style D fill:#ccffcc

Key principle: More unit tests (fast, cheap) at the base, fewer E2E tests (slow, expensive) at the top.

Unit Testing in CI/CD

Unit tests verify individual functions or methods in isolation.

Example Unit Tests

// src/utils/math.js
export function add(a, b) {
  return a + b;
}

export function divide(a, b) {
  if (b === 0) {
    throw new Error('Division by zero');
  }
  return a / b;
}

// src/utils/math.test.js
import { add, divide } from './math';

describe('Math utilities', () => {
  test('add should sum two numbers', () => {
    expect(add(2, 3)).toBe(5);
    expect(add(-1, 1)).toBe(0);
  });
  
  test('divide should divide two numbers', () => {
    expect(divide(10, 2)).toBe(5);
    expect(divide(9, 3)).toBe(3);
  });
  
  test('divide should throw error for division by zero', () => {
    expect(() => divide(5, 0)).toThrow('Division by zero');
  });
});

Unit Test Pipeline

# GitHub Actions
name: Unit Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
      
      - run: npm ci
      
      - run: npm run test:unit
        env:
          CI: true
      
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: test-results
          path: test-results/

# Azure Pipelines
stages:
  - stage: UnitTests
    jobs:
      - job: Test
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npm run test:unit
            displayName: 'Run unit tests'
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/junit.xml'

Integration Testing

Integration tests verify interactions between components, services, or modules.

Example Integration Test

// Integration test with database
describe('User API Integration', () => {
  let app;
  let db;
  
  beforeAll(async () => {
    // Start test database
    db = await setupTestDatabase();
    app = createApp(db);
  });
  
  afterAll(async () => {
    await db.close();
  });
  
  test('POST /users creates a new user', async () => {
    const response = await request(app)
      .post('/users')
      .send({ name: 'John Doe', email: 'john@example.com' })
      .expect(201);
    
    expect(response.body).toHaveProperty('id');
    expect(response.body.name).toBe('John Doe');
    
    // Verify in database
    const user = await db.users.findById(response.body.id);
    expect(user).toBeDefined();
  });
  
  test('GET /users/:id returns user details', async () => {
    const user = await db.users.create({
      name: 'Jane Doe',
      email: 'jane@example.com'
    });
    
    const response = await request(app)
      .get(`/users/${user.id}`)
      .expect(200);
    
    expect(response.body.name).toBe('Jane Doe');
  });
});

Integration Test Pipeline

stages:
  - stage: IntegrationTests
    jobs:
      - job: IntegrationTest
        pool:
          vmImage: 'ubuntu-latest'
        
        services:
          postgres:
            image: postgres:15
            env:
              POSTGRES_PASSWORD: postgres
              POSTGRES_DB: testdb
            ports:
              - 5432:5432
        
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npm run test:integration
            displayName: 'Run integration tests'
            env:
              DATABASE_URL: postgres://postgres:postgres@localhost:5432/testdb
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/integration-test-results.xml'

End-to-End (E2E) Testing

E2E tests validate complete user workflows through the application UI.

Example E2E Test

// Using Playwright
import { test, expect } from '@playwright/test';

test.describe('User Authentication', () => {
  test('user can sign up and log in', async ({ page }) => {
    // Navigate to signup page
    await page.goto('https://myapp.com/signup');
    
    // Fill signup form
    await page.fill('input[name="email"]', 'test@example.com');
    await page.fill('input[name="password"]', 'SecurePass123');
    await page.fill('input[name="confirmPassword"]', 'SecurePass123');
    
    // Submit form
    await page.click('button[type="submit"]');
    
    // Verify redirect to dashboard
    await expect(page).toHaveURL('https://myapp.com/dashboard');
    await expect(page.locator('h1')).toContainText('Welcome');
    
    // Log out
    await page.click('button[aria-label="Logout"]');
    
    // Log back in
    await page.goto('https://myapp.com/login');
    await page.fill('input[name="email"]', 'test@example.com');
    await page.fill('input[name="password"]', 'SecurePass123');
    await page.click('button[type="submit"]');
    
    // Verify successful login
    await expect(page).toHaveURL('https://myapp.com/dashboard');
  });
});

E2E Test Pipeline

stages:
  - stage: DeployStaging
    jobs:
      - deployment: DeployStaging
        environment: staging
        strategy:
          runOnce:
            deploy:
              steps:
                - script: kubectl apply -f staging-deployment.yaml

  - stage: E2ETests
    dependsOn: DeployStaging
    jobs:
      - job: E2ETest
        pool:
          vmImage: 'ubuntu-latest'
        
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npx playwright install --with-deps
            displayName: 'Install Playwright browsers'
          
          - script: npm run test:e2e
            displayName: 'Run E2E tests'
            env:
              BASE_URL: https://staging.myapp.com
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/e2e-results.xml'
          
          - task: PublishBuildArtifacts@1
            condition: failed()
            inputs:
              pathToPublish: 'test-results/screenshots'
              artifactName: 'e2e-failure-screenshots'

Test Execution Flow

sequenceDiagram
    participant Dev as Developer
    participant CI as CI Pipeline
    participant Unit as Unit Tests
    participant Int as Integration Tests
    participant E2E as E2E Tests
    participant Deploy as Deployment
    
    Dev->>CI: Push commit
    CI->>Unit: Run unit tests (fast)
    Unit->>CI: Results
    
    alt Unit tests fail
        CI->>Dev: Fail pipeline
    else Unit tests pass
        CI->>Int: Run integration tests
        Int->>CI: Results
        
        alt Integration tests fail
            CI->>Dev: Fail pipeline
        else Integration tests pass
            CI->>Deploy: Deploy to staging
            Deploy->>E2E: Run E2E tests
            E2E->>CI: Results
            
            alt E2E tests pass
                CI->>Deploy: Deploy to production
            else E2E tests fail
                CI->>Dev: Fail pipeline
            end
        end
    end

Code Coverage

Track how much code is executed by tests.

Coverage Configuration

// jest.config.js
module.exports = {
  collectCoverage: true,
  coverageThreshold: {
    global: {
      branches: 80,
      functions: 80,
      lines: 80,
      statements: 80
    }
  },
  coverageReporters: ['text', 'lcov', 'html'],
  collectCoverageFrom: [
    'src/**/*.{js,ts}',
    '!src/**/*.test.{js,ts}',
    '!src/index.js'
  ]
};

Coverage in Pipeline

stages:
  - stage: TestWithCoverage
    jobs:
      - job: Coverage
        steps:
          - script: npm test -- --coverage
            displayName: 'Run tests with coverage'
          
          - task: PublishCodeCoverageResults@1
            inputs:
              codeCoverageTool: 'Cobertura'
              summaryFileLocation: 'coverage/cobertura-coverage.xml'
              reportDirectory: 'coverage'
          
          - script: |
              # Fail if coverage below threshold
              COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
              if (( $(echo "$COVERAGE < 80" | bc -l) )); then
                echo "Coverage $COVERAGE% is below 80% threshold"
                exit 1
              fi
            displayName: 'Check coverage threshold'

Test Matrix

Run tests across multiple configurations.

# GitHub Actions
strategy:
  matrix:
    node-version: [16, 18, 20]
    os: [ubuntu-latest, windows-latest, macos-latest]

runs-on: ${{ matrix.os }}

steps:
  - uses: actions/setup-node@v4
    with:
      node-version: ${{ matrix.node-version }}
  
  - run: npm test

# Azure Pipelines
strategy:
  matrix:
    Linux_Node16:
      vmImage: 'ubuntu-latest'
      nodeVersion: '16.x'
    Linux_Node18:
      vmImage: 'ubuntu-latest'
      nodeVersion: '18.x'
    Windows_Node18:
      vmImage: 'windows-latest'
      nodeVersion: '18.x'

pool:
  vmImage: $(vmImage)

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: $(nodeVersion)
  
  - script: npm test

Performance Testing

Validate application performance under load.

Load Test Example

// Using k6 for load testing
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '1m', target: 50 },   // Ramp up to 50 users
    { duration: '3m', target: 50 },   // Stay at 50 users
    { duration: '1m', target: 100 },  // Ramp up to 100 users
    { duration: '3m', target: 100 },  // Stay at 100 users
    { duration: '1m', target: 0 },    // Ramp down to 0 users
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], // 95% of requests under 500ms
    http_req_failed: ['rate<0.01'],   // Error rate below 1%
  },
};

export default function () {
  const response = http.get('https://myapp.com/api/products');
  
  check(response, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
  
  sleep(1);
}

Performance Test Pipeline

stages:
  - stage: PerformanceTest
    dependsOn: DeployStaging
    jobs:
      - job: LoadTest
        pool:
          vmImage: 'ubuntu-latest'
        
        steps:
          - script: |
              # Install k6
              sudo gpg -k
              sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg \
                --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
              echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | \
                sudo tee /etc/apt/sources.list.d/k6.list
              sudo apt-get update
              sudo apt-get install k6
            displayName: 'Install k6'
          
          - script: k6 run --out json=results.json load-test.js
            displayName: 'Run load test'
            env:
              BASE_URL: https://staging.myapp.com
          
          - script: |
              # Check thresholds
              FAILED=$(jq '.metrics.checks.values.fails' results.json)
              if [ "$FAILED" -gt 0 ]; then
                echo "Performance test failed"
                exit 1
              fi
            displayName: 'Check performance thresholds'

Security Testing

Integrate security scans into CI/CD pipelines.

stages:
  - stage: SecurityTests
    jobs:
      - job: SecurityScan
        steps:
          # Dependency scanning
          - script: npm audit --audit-level=moderate
            displayName: 'NPM audit'
          
          # Static application security testing (SAST)
          - script: |
              npx eslint . --ext .js,.ts \
                --plugin security \
                --rule 'security/detect-object-injection: error'
            displayName: 'Static security analysis'
          
          # Container image scanning
          - script: |
              docker build -t myapp:$(Build.BuildId) .
              trivy image --severity HIGH,CRITICAL myapp:$(Build.BuildId)
            displayName: 'Container security scan'
          
          # OWASP dependency check
          - task: dependency-check-build-task@6
            inputs:
              projectName: 'MyApp'
              scanPath: '$(Build.SourcesDirectory)'
              format: 'HTML'

Smoke Tests

Quick validation after deployment.

// smoke-tests.js
const axios = require('axios');

async function runSmokeTests(baseUrl) {
  const tests = [
    { name: 'Health check', url: `${baseUrl}/health` },
    { name: 'API status', url: `${baseUrl}/api/status` },
    { name: 'Database connectivity', url: `${baseUrl}/api/db-check` },
  ];
  
  let failures = 0;
  
  for (const test of tests) {
    try {
      const response = await axios.get(test.url, { timeout: 5000 });
      if (response.status === 200) {
        console.log(`✓ ${test.name} passed`);
      } else {
        console.error(`✗ ${test.name} failed: status ${response.status}`);
        failures++;
      }
    } catch (error) {
      console.error(`✗ ${test.name} failed: ${error.message}`);
      failures++;
    }
  }
  
  if (failures > 0) {
    process.exit(1);
  }
}

const baseUrl = process.env.BASE_URL || 'http://localhost:3000';
runSmokeTests(baseUrl);

stages:
  - stage: Deploy
    jobs:
      - deployment: DeployProduction
        environment: production
        strategy:
          runOnce:
            deploy:
              steps:
                - script: kubectl apply -f production-deployment.yaml
            
            postRouteTraffic:
              steps:
                - script: |
                    # Wait for deployment to stabilize
                    sleep 30
                    
                    # Run smoke tests
                    node smoke-tests.js
                  displayName: 'Smoke tests'
                  env:
                    BASE_URL: https://production.myapp.com

Flaky Test Management

Handle unreliable tests that intermittently fail.

Retry Strategy

# GitHub Actions
steps:
  - name: Run tests with retry
    uses: nick-invision/retry@v2
    with:
      timeout_minutes: 10
      max_attempts: 3
      command: npm run test:e2e

# Azure Pipelines
steps:
  - powershell: |
      $maxAttempts = 3
      $attempt = 0
      $success = $false
      
      while ($attempt -lt $maxAttempts -and -not $success) {
        $attempt++
        Write-Host "Attempt $attempt of $maxAttempts"
        
        try {
          npm run test:e2e
          $success = $true
        } catch {
          if ($attempt -eq $maxAttempts) {
            throw
          }
          Write-Host "Test failed, retrying..."
          Start-Sleep -Seconds 5
        }
      }
    displayName: 'Run E2E tests with retry'

Quarantine Flaky Tests

// Mark flaky tests
describe.skip('Flaky test suite', () => {
  test('sometimes fails', () => {
    // Test code
  });
});

// Or use test tagging
test('stable test', { tag: '@stable' }, async () => {
  // Reliable test
});

test('flaky test', { tag: '@flaky' }, async () => {
  // Flaky test - run separately
});

// Run only stable tests in main pipeline
// npm test -- --grep "@stable"

Real-World Testing Pipeline

trigger:
  branches:
    include:
      - main
      - develop

stages:
  # Fast tests - run on every commit
  - stage: QuickTests
    jobs:
      - job: Lint
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          - script: npm ci
          - script: npm run lint
      
      - job: UnitTests
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          matrix:
            Node16:
              nodeVersion: '16.x'
            Node18:
              nodeVersion: '18.x'
            Node20:
              nodeVersion: '20.x'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: $(nodeVersion)
          
          - script: npm ci
          
          - script: npm run test:unit -- --coverage
            displayName: 'Unit tests with coverage'
          
          - task: PublishCodeCoverageResults@1
            condition: and(succeeded(), eq(variables['nodeVersion'], '18.x'))
            inputs:
              codeCoverageTool: 'Cobertura'
              summaryFileLocation: 'coverage/cobertura-coverage.xml'
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/junit.xml'

  # Medium tests - run after quick tests pass
  - stage: IntegrationTests
    dependsOn: QuickTests
    jobs:
      - job: IntegrationTest
        pool:
          vmImage: 'ubuntu-latest'
        
        services:
          postgres:
            image: postgres:15
            env:
              POSTGRES_PASSWORD: postgres
            ports:
              - 5432:5432
          
          redis:
            image: redis:7
            ports:
              - 6379:6379
        
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npm run test:integration
            displayName: 'Integration tests'
            env:
              DATABASE_URL: postgres://postgres:postgres@localhost:5432/testdb
              REDIS_URL: redis://localhost:6379
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/integration-results.xml'

  # Security scanning
  - stage: SecurityTests
    dependsOn: QuickTests
    jobs:
      - job: Security
        steps:
          - script: npm audit --audit-level=moderate
          - script: npx snyk test
            env:
              SNYK_TOKEN: $(SNYK_TOKEN)
          - script: |
              docker build -t myapp:$(Build.BuildId) .
              trivy image myapp:$(Build.BuildId)

  # Build and deploy to staging
  - stage: DeployStaging
    dependsOn:
      - IntegrationTests
      - SecurityTests
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployStaging
        environment: staging
        strategy:
          runOnce:
            deploy:
              steps:
                - script: kubectl apply -f staging-deployment.yaml

  # E2E tests on staging
  - stage: E2ETests
    dependsOn: DeployStaging
    jobs:
      - job: E2ETest
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npx playwright install --with-deps
          
          - script: npm run test:e2e
            displayName: 'E2E tests'
            env:
              BASE_URL: https://staging.myapp.com
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/e2e-results.xml'
          
          - task: PublishBuildArtifacts@1
            condition: failed()
            inputs:
              pathToPublish: 'playwright-report'
              artifactName: 'e2e-test-report'

  # Performance testing
  - stage: PerformanceTests
    dependsOn: E2ETests
    condition: succeeded()
    jobs:
      - job: LoadTest
        steps:
          - script: |
              curl -L https://github.com/grafana/k6/releases/download/v0.45.0/k6-v0.45.0-linux-amd64.tar.gz | tar xvz
              sudo mv k6-v0.45.0-linux-amd64/k6 /usr/local/bin
          
          - script: k6 run performance/load-test.js
            env:
              BASE_URL: https://staging.myapp.com

  # Deploy to production
  - stage: DeployProduction
    dependsOn: PerformanceTests
    condition: succeeded()
    jobs:
      - deployment: DeployProduction
        environment: production
        strategy:
          canary:
            increments: [25, 50, 100]
            deploy:
              steps:
                - script: kubectl apply -f production-deployment.yaml
            
            postRouteTraffic:
              steps:
                - script: node smoke-tests.js
                  env:
                    BASE_URL: https://production.myapp.com
                
                - script: sleep 300  # Monitor for 5 minutes

Key Takeaways

Next Steps: Implement the test pyramid in pipelines, add code coverage thresholds, integrate security scanning, and establish performance testing for staging deployments.