---

name: test-strategy description: Comprehensive test strategy guidance including test pyramid design, coverage goals, test categorization, flaky test diagnosis, test infrastructure architecture, and risk-based prioritization. Absorbed expertise from eliminated senior-qa-engineer. Use when planning testing approaches, setting up test infrastructure, optimizing test suites, diagnosing flaky tests, or designing test architecture across domains (API, data pipelines, ML models, infrastructure). Trigger keywords: test strategy, test pyramid, test plan, what to test, how to test, test architecture, test infrastructure, coverage goals, test organization, CI/CD testing, test prioritization, testing approach, flaky test, test optimization, test parallelization, API testing strategy, data pipeline testing, ML model testing, infrastructure testing.

Test Strategy

Overview

Test strategy defines how to approach testing for a project, balancing thoroughness with efficiency. A well-designed strategy ensures critical functionality is covered while avoiding over-testing trivial code. This skill covers the test pyramid, coverage metrics, test categorization, and integration with CI/CD pipelines.

Instructions

1. Design the Test Pyramid

Structure tests in layers with appropriate ratios:

Copy         /\
        /  \        E2E Tests (5-10%)
       /----\       - Critical user journeys
      /      \      - Cross-system integration
     /--------\     Integration Tests (15-25%)
    /          \    - API contracts
   /------------\   - Database interactions
  /              \  - Service boundaries
 /----------------\ Unit Tests (65-80%)
                    - Business logic
                    - Pure functions
                    - Edge cases

Recommended Ratios:

• Unit tests: 65-80% of test suite
• Integration tests: 15-25%
• E2E tests: 5-10%

2. Set Coverage Goals

Coverage Targets by Component Type:

Coverage Anti-patterns to Avoid:

• Chasing 100% coverage for coverage's sake
• Testing getters/setters without logic
• Testing framework or library code
• Writing tests that don't verify behavior

3. Decide What to Test vs What Not to Test

Always Test:

• Business logic and domain rules
• Input validation and error handling
• Security-sensitive operations
• Data transformations
• State transitions
• Edge cases and boundary conditions
• Regression scenarios from bug fixes

Consider Not Testing:

• Simple pass-through functions
• Framework-generated code
• Third-party library internals
• Trivial getters/setters
• Configuration constants
• Logging statements (unless critical)

Test Smell Detection:

Copy// BAD: Testing trivial code
test("getter returns value", () => {
  const user = new User("John");
  expect(user.getName()).toBe("John");
});

// GOOD: Testing meaningful behavior
test("user cannot change name to empty string", () => {
  const user = new User("John");
  expect(() => user.setName("")).toThrow(ValidationError);
});

4. Categorize and Organize Tests

Directory Structure:

Copytests/
├── unit/
│   ├── services/
│   ├── models/
│   └── utils/
├── integration/
│   ├── api/
│   ├── database/
│   └── external-services/
├── e2e/
│   ├── flows/
│   └── pages/
├── fixtures/
│   ├── factories/
│   └── mocks/
└── helpers/
    ├── setup.ts
    └── assertions.ts

Test Tagging System:

Copy// Jest example with tags
describe("[unit][fast] UserService", () => {});
describe("[integration][slow] DatabaseRepository", () => {});
describe("[e2e][critical] CheckoutFlow", () => {});

// Run specific categories
// npm test -- --grep="\[unit\]"
// npm test -- --grep="\[critical\]"

Naming Conventions:

Copy[ComponentName].[scenario].[expected_result].test.ts

Examples:
UserService.createUser.returnsNewUser.test.ts
PaymentProcessor.invalidCard.throwsPaymentError.test.ts

5. Integrate with CI/CD

Pipeline Stage Configuration:

Copy# .github/workflows/test.yml
name: Test Pipeline

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Unit Tests
        run: npm test -- --grep="\[unit\]" --coverage
      - name: Upload Coverage
        uses: codecov/codecov-action@v3

  integration-tests:
    runs-on: ubuntu-latest
    needs: unit-tests
    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: test
    steps:
      - uses: actions/checkout@v4
      - name: Run Integration Tests
        run: npm test -- --grep="\[integration\]"

  e2e-tests:
    runs-on: ubuntu-latest
    needs: integration-tests
    steps:
      - uses: actions/checkout@v4
      - name: Run E2E Tests
        run: npm run test:e2e

CI Test Optimization:

• Run unit tests first (fast feedback)
• Parallelize test suites
• Cache dependencies and build artifacts
• Use test splitting for large suites
• Fail fast on critical tests

6. Risk-Based Test Prioritization

Risk Matrix for Prioritization:

Risk Factors to Consider:

• Business Impact: Revenue, user trust, legal compliance
• Complexity: Code complexity, integration points
• Change Frequency: Actively developed areas
• Historical Bugs: Components with bug history
• Dependencies: Critical external services

Prioritized Test Categories:

1. Critical (P0): Run on every commit

   • Authentication/authorization
   • Payment processing
   • Data integrity

2. High (P1): Run on PR merge

   • Core business workflows
   • API contract tests

3. Medium (P2): Run nightly

   • Edge cases
   • Performance tests

4. Low (P3): Run weekly

   • Backward compatibility
   • Deprecated feature coverage

7. Domain-Specific Testing Strategies

API Testing Strategy

Test Layers:

1. Contract Tests (P0)

   • Request/response schema validation
   • HTTP status codes for all endpoints
   • Error response formats
   • Authentication/authorization rules

2. Business Logic Tests (P0)

   • Valid input processing
   • Business rule enforcement
   • State transitions via API calls

3. Integration Tests (P1)

   • Database operations via API
   • External service integration
   • Transaction rollback scenarios

4. Performance Tests (P2)

   • Response time under load
   • Concurrent request handling
   • Rate limiting behavior

API Test Organization:

Copytests/api/
├── contracts/          # Schema validation tests
├── endpoints/          # Per-endpoint behavior tests
├── auth/               # Authentication flows
├── integration/        # Cross-service scenarios
└── performance/        # Load and stress tests

Data Pipeline Testing Strategy

Test Focus Areas:

1. Data Quality Tests (P0)

   • Schema validation at each stage
   • Data type correctness
   • Null/missing value handling
   • Duplicate detection

2. Transformation Tests (P0)

   • Input → output correctness
   • Edge case handling
   • Data loss detection
   • Aggregation accuracy

3. Integration Tests (P1)

   • Source extraction correctness
   • Sink loading verification
   • Idempotency checks
   • Failure recovery

4. Performance Tests (P2)

   • Processing throughput
   • Memory usage with large datasets
   • Partition handling

Data Pipeline Test Pattern:

Copydef test_user_data_transformation():
    # Arrange: Create test input data
    raw_input = create_test_dataset(
        rows=1000,
        include_nulls=True,
        include_duplicates=True
    )

    # Act: Run transformation
    result = transform_user_data(raw_input)

    # Assert: Verify output quality
    assert_no_nulls(result, required_fields=["user_id", "email"])
    assert_no_duplicates(result, key="user_id")
    assert_schema_matches(result, UserSchema)
    assert len(result) == expected_output_count(raw_input)

ML Model Testing Strategy

Test Layers:

1. Data Validation Tests (P0)

   • Feature schema validation
   • Label distribution checks
   • Data leakage detection
   • Train/test split correctness

2. Model Behavior Tests (P0)

   • Prediction on known examples
   • Invariance tests (e.g., case-insensitive text)
   • Directional expectation tests
   • Boundary condition handling

3. Model Quality Tests (P1)

   • Accuracy/precision/recall thresholds
   • Fairness metrics across groups
   • Performance on edge cases
   • Regression detection (vs baseline)

4. Integration Tests (P1)

   • Model loading and serving
   • Prediction API contract
   • Feature engineering pipeline
   • Model versioning

ML Test Example:

Copydef test_sentiment_model_invariance():
    """Model should be case-insensitive"""
    model = load_sentiment_model()

    test_cases = [
        ("This is GREAT!", "This is great!"),
        ("TERRIBLE service", "terrible service"),
    ]

    for text1, text2 in test_cases:
        pred1 = model.predict(text1)
        pred2 = model.predict(text2)
        assert pred1 == pred2, f"Case sensitivity detected: {text1} vs {text2}"

Infrastructure Testing Strategy

Test Focus:

1. Infrastructure-as-Code Tests (P0)

   • Syntax validation (terraform validate)
   • Security policy checks
   • Resource naming conventions
   • Cost estimation validation

2. Deployment Tests (P1)

   • Smoke tests post-deployment
   • Health check endpoints
   • Configuration validation
   • Rollback procedures

3. Resilience Tests (P2)

   • Service restart handling
   • Network partition recovery
   • Resource exhaustion scenarios
   • Chaos engineering tests

4. Observability Tests (P1)

   • Metrics collection verification
   • Log aggregation correctness
   • Alert rule validation
   • Dashboard functionality

Infrastructure Test Pattern:

Copy# terraform test example
run "verify_security_group_rules" {
  command = plan

  assert {
    condition     = length([for rule in aws_security_group.main.ingress : rule if rule.cidr_blocks[0] == "0.0.0.0/0"]) == 0
    error_message = "Security group should not allow ingress from 0.0.0.0/0"
  }
}

8. Flaky Test Diagnosis and Prevention

Common Causes of Flakiness:

Flaky Test Diagnosis Workflow:

Copy1. Reproduce Locally
   ├─ Run test 100 times: `for i in {1..100}; do npm test -- TestName || break; done`
   ├─ Run with different seeds: `npm test -- --seed=$RANDOM`
   └─ Run in parallel: `npm test -- --maxWorkers=4`

2. Identify Pattern
   ├─ Always fails at same point? → Logic bug, not flaky
   ├─ Fails under load? → Timing/resource issue
   ├─ Fails with other tests? → Shared state pollution
   └─ Fails on specific data? → Data-dependent bug

3. Instrument Test
   ├─ Add verbose logging
   ├─ Capture timing information
   ├─ Record test environment state
   └─ Save failure artifacts (screenshots, logs)

4. Fix Root Cause
   ├─ Eliminate race conditions
   ├─ Add proper synchronization
   ├─ Isolate test state
   └─ Mock external dependencies

5. Verify Fix
   ├─ Run fixed test 1000 times
   ├─ Run in CI 10 times
   └─ Monitor over 1 week

Flaky Test Prevention Checklist:

• Tests use deterministic test data (fixed seeds, no random())
• Async operations use explicit waits (not setTimeout/sleep)
• Tests create unique resources (UUIDs in names/IDs)
• Cleanup always runs (try/finally, afterEach hooks)
• No hardcoded timing assumptions (sleep(100) is a code smell)
• External services are mocked or use test doubles
• Time-dependent logic uses injected/fake clocks
• Tests do not depend on execution order
• Shared state is reset between tests
• Test environment is reproducible (containerized)

Example: Fixing a Flaky Test

Copy// FLAKY: Race condition with async operation
test("user profile loads", async () => {
  renderUserProfile(userId);
  // Race: profile might not be loaded yet
  expect(screen.getByText("John Doe")).toBeInTheDocument();
});

// FIXED: Proper async handling
test("user profile loads", async () => {
  renderUserProfile(userId);
  // Wait for async operation to complete
  const userName = await screen.findByText("John Doe");
  expect(userName).toBeInTheDocument();
});

// FLAKY: Shared state pollution
test("creates user with default role", () => {
  const user = createUser({ name: "Alice" });
  expect(user.role).toBe("user"); // Fails if previous test modified default
});

// FIXED: Isolated state
test("creates user with default role", () => {
  resetDefaultRole(); // Ensure clean state
  const user = createUser({ name: "Alice" });
  expect(user.role).toBe("user");
});

// FLAKY: Time-dependent logic
test("expires session after 1 hour", () => {
  const session = createSession();
  // Flaky: Depends on current time
  expect(session.expiresAt).toBe(Date.now() + 3600000);
});

// FIXED: Inject time dependency
test("expires session after 1 hour", () => {
  const mockClock = installFakeClock();
  mockClock.setTime(new Date("2024-01-01T12:00:00Z"));

  const session = createSession();
  expect(session.expiresAt).toBe(new Date("2024-01-01T13:00:00Z").getTime());

  mockClock.uninstall();
});

9. Test Infrastructure Architecture

Test Environment Management:

Copy# docker-compose.test.yml
version: '3.8'
services:
  test-db:
    image: postgres:15
    environment:
      POSTGRES_DB: test_db
      POSTGRES_USER: test_user
      POSTGRES_PASSWORD: test_pass
    ports:
      - "5433:5432"
    tmpfs:
      - /var/lib/postgresql/data  # In-memory for speed

  test-redis:
    image: redis:7-alpine
    ports:
      - "6380:6379"

  test-app:
    build: .
    environment:
      DATABASE_URL: postgres://test_user:test_pass@test-db:5432/test_db
      REDIS_URL: redis://test-redis:6379
    depends_on:
      - test-db
      - test-redis

Test Data Management:

Copy// Factory pattern for test data
class UserFactory {
  private sequence = 0;

  create(overrides?: Partial<User>): User {
    return {
      id: overrides?.id ?? `user-${this.sequence++}`,
      email: overrides?.email ?? `user${this.sequence}@test.com`,
      name: overrides?.name ?? `Test User ${this.sequence}`,
      role: overrides?.role ?? "user",
      createdAt: overrides?.createdAt ?? new Date(),
    };
  }

  createBatch(count: number, overrides?: Partial<User>): User[] {
    return Array.from({ length: count }, () => this.create(overrides));
  }
}

// Usage ensures unique data per test
test("user search works", () => {
  const factory = new UserFactory();
  const users = factory.createBatch(10);
  // Each test gets unique users, no conflicts
});

Test Parallelization Strategy:

Example: Parallel Test Configuration

Copy// jest.config.js with parallel optimization
module.exports = {
  maxWorkers: process.env.CI ? "50%" : "75%", // Conservative in CI
  testTimeout: 30000, // Longer timeout for CI

  // Run fast tests first
  testSequencer: "./custom-sequencer.js",

  // Database isolation per worker
  globalSetup: "./tests/setup/create-test-dbs.js",
  globalTeardown: "./tests/setup/drop-test-dbs.js",

  // Shard tests in CI
  shard: process.env.CI_NODE_INDEX
    ? `${process.env.CI_NODE_INDEX}/${process.env.CI_NODE_TOTAL}`
    : undefined,
};

Test Optimization Techniques:

1. Reduce Test Startup Time

   • Cache compiled code
   • Lazy-load test dependencies
   • Use in-memory databases for unit tests

2. Optimize Test Execution

   • Batch database operations
   • Reuse expensive fixtures (connections, containers)
   • Skip unnecessary setup for focused tests

3. Parallelize Safely

   • Unique identifiers per test (UUIDs)
   • Separate database schemas per worker
   • Avoid shared file system access

4. Smart Test Selection

   • Run only affected tests during development
   • Use coverage mapping to determine affected tests
   • Cache test results for unchanged code

Copy# Run only tests affected by changes
npm test -- --changedSince=origin/main

# Run tests for specific module and dependents
npm test -- --selectProjects=user-service --testPathPattern=user

# Watch mode with smart re-running
npm test -- --watch --changedSince=HEAD

Best Practices

1. Test Behavior, Not Implementation

   • Tests should verify outcomes, not internal mechanics
   • Refactoring should not break tests if behavior unchanged

2. Keep Tests Independent

   • No shared mutable state between tests
   • Each test sets up its own context
   • Tests can run in any order

3. Use Test Doubles Appropriately

   • Stubs for providing test data
   • Mocks for verifying interactions
   • Fakes for complex dependencies
   • Real implementations when feasible

4. Maintain Test Quality

   • Apply same code quality standards to tests
   • Refactor test code for readability
   • Remove obsolete tests promptly

5. Fast Feedback Loop

   • Optimize for quick local test runs
   • Use watch mode during development
   • Prioritize fast tests in CI

6. Document Test Intent

   • Clear test names describe behavior
   • Add comments for non-obvious setup
   • Link tests to requirements/tickets

Examples

Example: Feature Test Strategy Document

Copy# Feature: User Registration

## Risk Assessment

- Business Impact: HIGH (user acquisition)
- Complexity: MEDIUM (email validation, password rules)
- Change Frequency: LOW (stable feature)

## Test Coverage Plan

### Unit Tests (P0)

- [ ] Email format validation
- [ ] Password strength requirements
- [ ] Username uniqueness check logic
- [ ] Profile data sanitization

### Integration Tests (P1)

- [ ] Database user creation
- [ ] Email service integration
- [ ] Duplicate email handling

### E2E Tests (P0)

- [ ] Happy path: complete registration flow
- [ ] Error path: duplicate email shows error

## Coverage Targets

- Line coverage: 85%
- Branch coverage: 80%
- Critical paths: 100%

Example: Test Organization Configuration

Copy// jest.config.js
module.exports = {
  projects: [
    {
      displayName: "unit",
      testMatch: ["<rootDir>/tests/unit/**/*.test.ts"],
      setupFilesAfterEnv: ["<rootDir>/tests/helpers/unit-setup.ts"],
    },
    {
      displayName: "integration",
      testMatch: ["<rootDir>/tests/integration/**/*.test.ts"],
      setupFilesAfterEnv: ["<rootDir>/tests/helpers/integration-setup.ts"],
      globalSetup: "<rootDir>/tests/helpers/db-setup.ts",
      globalTeardown: "<rootDir>/tests/helpers/db-teardown.ts",
    },
  ],
  coverageThreshold: {
    global: {
      branches: 75,
      functions: 80,
      lines: 80,
      statements: 80,
    },
    "./src/services/": {
      branches: 90,
      lines: 90,
    },
  },
};

Example: Risk-Based Test Selection Script

Copy// scripts/select-tests.ts
interface TestFile {
  path: string;
  priority: "P0" | "P1" | "P2" | "P3";
  tags: string[];
}

function selectTestsForPipeline(
  context: "commit" | "pr" | "nightly" | "weekly",
): TestFile[] {
  const allTests = getTestManifest();

  const priorityMap = {
    commit: ["P0"],
    pr: ["P0", "P1"],
    nightly: ["P0", "P1", "P2"],
    weekly: ["P0", "P1", "P2", "P3"],
  };

  return allTests.filter((test) =>
    priorityMap[context].includes(test.priority),
  );
}