Testing in CI/CD Pipelines

What is CI/CD Testing?

CI/CD testing integrates automated tests directly into continuous integration and deployment pipelines, providing rapid feedback on code quality and functionality. Tests run automatically on every code change, catching issues early before they reach production.

Effective testing in CI/CD ensures code quality, reduces manual QA effort, and enables confident, frequent deployments.

Why Testing in CI/CD Matters

• Early Bug Detection - Catch issues immediately after code changes
• Fast Feedback - Developers learn about problems within minutes
• Regression Prevention - Ensure new code doesn't break existing features
• Deployment Confidence - Trust automated tests to validate releases
• Reduced Manual Effort - Automate repetitive testing tasks
• Documentation - Tests serve as executable specifications

Test Pyramid

The test pyramid guides the balance of different test types in CI/CD pipelines.

graph TD
    A["Manual Exploratory Tests
Expensive, Slow
5%"] -->B["End-to-End Tests
Integration across systems
10%"]
    B -->C["Integration Tests
Component interactions
20%"]
    C -->D["Unit Tests
Individual functions
65%"]
    
    style A fill:#ffcccc
    style B fill:#ffd9b3
    style C fill:#fff3e0
    style D fill:#ccffcc

Key principle: More unit tests (fast, cheap) at the base, fewer E2E tests (slow, expensive) at the top.

Unit Testing in CI/CD

Unit tests verify individual functions or methods in isolation.

Example Unit Tests

public static class MathUtils
{
    public static int Add(int a, int b) => a + b;

    public static decimal Divide(decimal a, decimal b)
    {
        if (b == 0) throw new DivideByZeroException("Division by zero");
        return a / b;
    }
}

public class MathUtilsTests
{
    [Fact]
    public void Add_ShouldSumTwoNumbers()
    {
        Assert.Equal(5, MathUtils.Add(2, 3));
        Assert.Equal(0, MathUtils.Add(-1, 1));
    }

    [Fact]
    public void Divide_ShouldThrowForZeroDivisor()
    {
        Assert.Throws<DivideByZeroException>(() => MathUtils.Divide(5, 0));
    }
}

Unit Test Pipeline

# GitHub Actions
name: Unit Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'
      
      - run: npm ci
      
      - run: npm run test:unit
        env:
          CI: true
      
      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: test-results
          path: test-results/

# Azure Pipelines
stages:
  - stage: UnitTests
    jobs:
      - job: Test
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npm run test:unit
            displayName: 'Run unit tests'
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/junit.xml'

Integration Testing

Integration tests verify interactions between components, services, or modules.

Example Integration Test

public class UserApiIntegrationTests : IClassFixture<ApiFactory>
{
    private readonly HttpClient _client;
    private readonly TestDb _db;

    public UserApiIntegrationTests(ApiFactory factory)
    {
        _client = factory.CreateClient();
        _db = factory.Db;
    }

    [Fact]
    public async Task PostUsers_CreatesUser()
    {
        var response = await _client.PostAsJsonAsync("/users", new { name = "John Doe", email = "john@example.com" });
        Assert.Equal(HttpStatusCode.Created, response.StatusCode);
    }
}

Integration Test Pipeline

stages:
  - stage: IntegrationTests
    jobs:
      - job: IntegrationTest
        pool:
          vmImage: 'ubuntu-latest'
        
        services:
          postgres:
            image: postgres:15
            env:
              POSTGRES_PASSWORD: postgres
              POSTGRES_DB: testdb
            ports:
              - 5432:5432
        
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npm run test:integration
            displayName: 'Run integration tests'
            env:
              DATABASE_URL: postgres://postgres:postgres@localhost:5432/testdb
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/integration-test-results.xml'

End-to-End (E2E) Testing

E2E tests validate complete user workflows through the application UI.

Example E2E Test

using Microsoft.Playwright;

[Fact]
public async Task UserCanSignUpAndLogIn()
{
    using var playwright = await Playwright.CreateAsync();
    await using var browser = await playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions { Headless = true });
    var page = await browser.NewPageAsync();

    await page.GotoAsync("https://myapp.com/signup");
    await page.FillAsync("input[name='email']", "test@example.com");
    await page.FillAsync("input[name='password']", "SecurePass123");
    await page.ClickAsync("button[type='submit']");

    Assert.Equal("https://myapp.com/dashboard", page.Url);
}

E2E Test Pipeline

stages:
  - stage: DeployStaging
    jobs:
      - deployment: DeployStaging
        environment: staging
        strategy:
          runOnce:
            deploy:
              steps:
                - script: kubectl apply -f staging-deployment.yaml

  - stage: E2ETests
    dependsOn: DeployStaging
    jobs:
      - job: E2ETest
        pool:
          vmImage: 'ubuntu-latest'
        
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npx playwright install --with-deps
            displayName: 'Install Playwright browsers'
          
          - script: npm run test:e2e
            displayName: 'Run E2E tests'
            env:
              BASE_URL: https://staging.myapp.com
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/e2e-results.xml'
          
          - task: PublishBuildArtifacts@1
            condition: failed()
            inputs:
              pathToPublish: 'test-results/screenshots'
              artifactName: 'e2e-failure-screenshots'

Test Execution Flow

sequenceDiagram
    participant Dev as Developer
    participant CI as CI Pipeline
    participant Unit as Unit Tests
    participant Int as Integration Tests
    participant E2E as E2E Tests
    participant Deploy as Deployment
    
    Dev->>CI: Push commit
    CI->>Unit: Run unit tests (fast)
    Unit->>CI: Results
    
    alt Unit tests fail
        CI->>Dev: Fail pipeline
    else Unit tests pass
        CI->>Int: Run integration tests
        Int->>CI: Results
        
        alt Integration tests fail
            CI->>Dev: Fail pipeline
        else Integration tests pass
            CI->>Deploy: Deploy to staging
            Deploy->>E2E: Run E2E tests
            E2E->>CI: Results
            
            alt E2E tests pass
                CI->>Deploy: Deploy to production
            else E2E tests fail
                CI->>Dev: Fail pipeline
            end
        end
    end

Code Coverage

Track how much code is executed by tests.

Coverage Configuration

<!-- Directory.Build.props -->
<Project>
  <PropertyGroup>
    <CollectCoverage>true</CollectCoverage>
    <CoverletOutputFormat>cobertura</CoverletOutputFormat>
    <Threshold>80</Threshold>
    <ThresholdType>line,branch,method</ThresholdType>
    <ThresholdStat>Total</ThresholdStat>
  </PropertyGroup>
</Project>

Coverage in Pipeline

stages:
  - stage: TestWithCoverage
    jobs:
      - job: Coverage
        steps:
          - script: npm test -- --coverage
            displayName: 'Run tests with coverage'
          
          - task: PublishCodeCoverageResults@1
            inputs:
              codeCoverageTool: 'Cobertura'
              summaryFileLocation: 'coverage/cobertura-coverage.xml'
              reportDirectory: 'coverage'
          
          - script: |
              # Fail if coverage below threshold
              COVERAGE=$(cat coverage/coverage-summary.json | jq '.total.lines.pct')
              if (( $(echo "$COVERAGE < 80" | bc -l) )); then
                echo "Coverage $COVERAGE% is below 80% threshold"
                exit 1
              fi
            displayName: 'Check coverage threshold'

Test Matrix

Run tests across multiple configurations.

# GitHub Actions
strategy:
  matrix:
    node-version: [16, 18, 20]
    os: [ubuntu-latest, windows-latest, macos-latest]

runs-on: ${{ matrix.os }}

steps:
  - uses: actions/setup-node@v4
    with:
      node-version: ${{ matrix.node-version }}
  
  - run: npm test

# Azure Pipelines
strategy:
  matrix:
    Linux_Node16:
      vmImage: 'ubuntu-latest'
      nodeVersion: '16.x'
    Linux_Node18:
      vmImage: 'ubuntu-latest'
      nodeVersion: '18.x'
    Windows_Node18:
      vmImage: 'windows-latest'
      nodeVersion: '18.x'

pool:
  vmImage: $(vmImage)

steps:
  - task: NodeTool@0
    inputs:
      versionSpec: $(nodeVersion)
  
  - script: npm test

Performance Testing

Validate application performance under load.

Load Test Example

using NBomber.CSharp;
using NBomber.Http.CSharp;

var scenario = Scenario.Create("api_load", async context =>
{
    var response = await Http.Send(Http.CreateRequest("GET", "https://myapp.com/api/products"), context);
    return response.IsError ? Response.Fail() : Response.Ok();
})
    .WithLoadSimulations(
        Simulation.RampingInject(rate: 50, interval: TimeSpan.FromMinutes(1), during: TimeSpan.FromMinutes(1)),
        Simulation.Inject(rate: 100, interval: TimeSpan.FromSeconds(1), during: TimeSpan.FromMinutes(3))
    );

NBomberRunner.RegisterScenarios(scenario).Run();

Performance Test Pipeline

stages:
  - stage: PerformanceTest
    dependsOn: DeployStaging
    jobs:
      - job: LoadTest
        pool:
          vmImage: 'ubuntu-latest'
        
        steps:
          - script: |
              # Install k6
              sudo gpg -k
              sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg \
                --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
              echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | \
                sudo tee /etc/apt/sources.list.d/k6.list
              sudo apt-get update
              sudo apt-get install k6
            displayName: 'Install k6'
          
          - script: k6 run --out json=results.json load-test.js
            displayName: 'Run load test'
            env:
              BASE_URL: https://staging.myapp.com
          
          - script: |
              # Check thresholds
              FAILED=$(jq '.metrics.checks.values.fails' results.json)
              if [ "$FAILED" -gt 0 ]; then
                echo "Performance test failed"
                exit 1
              fi
            displayName: 'Check performance thresholds'

Security Testing

Integrate security scans into CI/CD pipelines.

stages:
  - stage: SecurityTests
    jobs:
      - job: SecurityScan
        steps:
          # Dependency scanning
          - script: npm audit --audit-level=moderate
            displayName: 'NPM audit'
          
          # Static application security testing (SAST)
          - script: |
              npx eslint . --ext .js,.ts \
                --plugin security \
                --rule 'security/detect-object-injection: error'
            displayName: 'Static security analysis'
          
          # Container image scanning
          - script: |
              docker build -t myapp:$(Build.BuildId) .
              trivy image --severity HIGH,CRITICAL myapp:$(Build.BuildId)
            displayName: 'Container security scan'
          
          # OWASP dependency check
          - task: dependency-check-build-task@6
            inputs:
              projectName: 'MyApp'
              scanPath: '$(Build.SourcesDirectory)'
              format: 'HTML'

Smoke Tests

Quick validation after deployment.

using System.Net;

var baseUrl = Environment.GetEnvironmentVariable("BASE_URL") ?? "http://localhost:3000";
var checks = new[] { "/health", "/api/status", "/api/db-check" };
var failures = 0;

using var client = new HttpClient { Timeout = TimeSpan.FromSeconds(5) };
foreach (var path in checks)
{
    var response = await client.GetAsync($"{baseUrl}{path}");
    if (response.StatusCode != HttpStatusCode.OK)
    {
        failures++;
    }
}

if (failures > 0) Environment.Exit(1);

stages:
  - stage: Deploy
    jobs:
      - deployment: DeployProduction
        environment: production
        strategy:
          runOnce:
            deploy:
              steps:
                - script: kubectl apply -f production-deployment.yaml
            
            postRouteTraffic:
              steps:
                - script: |
                    # Wait for deployment to stabilize
                    sleep 30
                    
                    # Run smoke tests
                    node smoke-tests.js
                  displayName: 'Smoke tests'
                  env:
                    BASE_URL: https://production.myapp.com

Flaky Test Management

Handle unreliable tests that intermittently fail.

Retry Strategy

# GitHub Actions
steps:
  - name: Run tests with retry
    uses: nick-invision/retry@v2
    with:
      timeout_minutes: 10
      max_attempts: 3
      command: npm run test:e2e

# Azure Pipelines
steps:
  - powershell: |
      $maxAttempts = 3
      $attempt = 0
      $success = $false
      
      while ($attempt -lt $maxAttempts -and -not $success) {
        $attempt++
        Write-Host "Attempt $attempt of $maxAttempts"
        
        try {
          npm run test:e2e
          $success = $true
        } catch {
          if ($attempt -eq $maxAttempts) {
            throw
          }
          Write-Host "Test failed, retrying..."
          Start-Sleep -Seconds 5
        }
      }
    displayName: 'Run E2E tests with retry'

Quarantine Flaky Tests

// Mark flaky tests
public class FlakySuite
{
    [Fact(Skip = "Flaky - under investigation")]
    public void SometimesFails()
    {
    }
}

// Or use test traits
public class TaggedTests
{
    [Fact, Trait("Category", "Stable")]
    public void StableTest() { }

    [Fact, Trait("Category", "Flaky")]
    public void FlakyTest() { }
}

// Run only stable tests in main pipeline
// dotnet test --filter "Category=Stable"

Real-World Testing Pipeline

trigger:
  branches:
    include:
      - main
      - develop

stages:
  # Fast tests - run on every commit
  - stage: QuickTests
    jobs:
      - job: Lint
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          - script: npm ci
          - script: npm run lint
      
      - job: UnitTests
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          matrix:
            Node16:
              nodeVersion: '16.x'
            Node18:
              nodeVersion: '18.x'
            Node20:
              nodeVersion: '20.x'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: $(nodeVersion)
          
          - script: npm ci
          
          - script: npm run test:unit -- --coverage
            displayName: 'Unit tests with coverage'
          
          - task: PublishCodeCoverageResults@1
            condition: and(succeeded(), eq(variables['nodeVersion'], '18.x'))
            inputs:
              codeCoverageTool: 'Cobertura'
              summaryFileLocation: 'coverage/cobertura-coverage.xml'
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/junit.xml'

  # Medium tests - run after quick tests pass
  - stage: IntegrationTests
    dependsOn: QuickTests
    jobs:
      - job: IntegrationTest
        pool:
          vmImage: 'ubuntu-latest'
        
        services:
          postgres:
            image: postgres:15
            env:
              POSTGRES_PASSWORD: postgres
            ports:
              - 5432:5432
          
          redis:
            image: redis:7
            ports:
              - 6379:6379
        
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npm run test:integration
            displayName: 'Integration tests'
            env:
              DATABASE_URL: postgres://postgres:postgres@localhost:5432/testdb
              REDIS_URL: redis://localhost:6379
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/integration-results.xml'

  # Security scanning
  - stage: SecurityTests
    dependsOn: QuickTests
    jobs:
      - job: Security
        steps:
          - script: npm audit --audit-level=moderate
          - script: npx snyk test
            env:
              SNYK_TOKEN: $(SNYK_TOKEN)
          - script: |
              docker build -t myapp:$(Build.BuildId) .
              trivy image myapp:$(Build.BuildId)

  # Build and deploy to staging
  - stage: DeployStaging
    dependsOn:
      - IntegrationTests
      - SecurityTests
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    jobs:
      - deployment: DeployStaging
        environment: staging
        strategy:
          runOnce:
            deploy:
              steps:
                - script: kubectl apply -f staging-deployment.yaml

  # E2E tests on staging
  - stage: E2ETests
    dependsOn: DeployStaging
    jobs:
      - job: E2ETest
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: NodeTool@0
            inputs:
              versionSpec: '18.x'
          
          - script: npm ci
          
          - script: npx playwright install --with-deps
          
          - script: npm run test:e2e
            displayName: 'E2E tests'
            env:
              BASE_URL: https://staging.myapp.com
          
          - task: PublishTestResults@2
            condition: succeededOrFailed()
            inputs:
              testResultsFormat: 'JUnit'
              testResultsFiles: '**/e2e-results.xml'
          
          - task: PublishBuildArtifacts@1
            condition: failed()
            inputs:
              pathToPublish: 'playwright-report'
              artifactName: 'e2e-test-report'

  # Performance testing
  - stage: PerformanceTests
    dependsOn: E2ETests
    condition: succeeded()
    jobs:
      - job: LoadTest
        steps:
          - script: |
              curl -L https://github.com/grafana/k6/releases/download/v0.45.0/k6-v0.45.0-linux-amd64.tar.gz | tar xvz
              sudo mv k6-v0.45.0-linux-amd64/k6 /usr/local/bin
          
          - script: k6 run performance/load-test.js
            env:
              BASE_URL: https://staging.myapp.com

  # Deploy to production
  - stage: DeployProduction
    dependsOn: PerformanceTests
    condition: succeeded()
    jobs:
      - deployment: DeployProduction
        environment: production
        strategy:
          canary:
            increments: [25, 50, 100]
            deploy:
              steps:
                - script: kubectl apply -f production-deployment.yaml
            
            postRouteTraffic:
              steps:
                - script: node smoke-tests.js
                  env:
                    BASE_URL: https://production.myapp.com
                
                - script: sleep 300  # Monitor for 5 minutes

Key Takeaways

• Follow the test pyramid with more unit tests than integration or E2E tests
• Run fast tests first to provide quick feedback to developers
• Integration tests verify component interactions with databases and services
• E2E tests validate complete user workflows through the application UI
• Code coverage tracks test completeness but should not be the only quality metric
• Matrix testing validates code across multiple versions and platforms
• Performance testing ensures applications handle expected load
• Security testing catches vulnerabilities early in the development cycle
• Smoke tests provide quick validation after deployment
• Handle flaky tests with retries or quarantine strategies
• Parallel test execution reduces pipeline duration significantly
• Publish test results and coverage reports for visibility

Next Steps: Implement the test pyramid in pipelines, add code coverage thresholds, integrate security scanning, and establish performance testing for staging deployments.