---

name: api-test-runner description: Run CI-friendly API test suites (REST + GraphQL) from a single manifest, delegating to rest.sh/gql.sh and emitting JSON (+ optional JUnit) results. Use when the user asks to reduce CI boilerplate and provide a simple, composable suite runner for other tools (pytest/node/LLM) to call.

API Test Runner (REST + GraphQL)

Contract

Prereqs:

• bash and jq available on PATH.
• REST runner: skills/rest-api-testing/scripts/rest.sh (requires curl).
• GraphQL runner: skills/graphql-api-testing/scripts/gql.sh (requires xh/http/curl).

Inputs:

• Suite selection: --suite <name> or --suite-file <path>.
• Optional filters/flags: --tag, --allow-writes, --out, --junit.
• Optional auth secret env (when configured in suite): API_TEST_AUTH_JSON (or custom secretEnv).

Outputs:

• JSON results always emitted to stdout.
• Optional files: --out <path> (results JSON) and --junit <path> (JUnit XML).
• Per-run logs/artifacts under out/api-test-runner/<runId>/ when executed.

Exit codes:

• 0: all selected cases passed
• 2: one or more cases failed
• 1: invalid inputs / schema / missing files

Failure modes:

• Missing/invalid suite or case files (JSON schema errors, missing request/op files).
• Missing dependencies (jq, curl, or no supported GraphQL HTTP client).
• Auth missing/invalid (401/403) or write-capable case blocked by safety defaults.

Goal

Run a suite of API checks in CI (and locally) via a single manifest file, reusing existing callers:

• REST: skills/rest-api-testing/scripts/rest.sh
• GraphQL: skills/graphql-api-testing/scripts/gql.sh

The runner:

• Executes selected cases deterministically
• Produces machine-readable JSON results (and optional JUnit XML)
• Applies safe defaults (no secret leakage; guardrails for write-capable cases)

TL;DR

Bootstrap a minimal setup/api/ (when your repo already has setup/rest and/or setup/graphql):

Copymkdir -p setup
cp -R "$AGENT_KIT_HOME/skills/api-test-runner/template/setup/api" setup/

Bootstrap a runnable public-endpoint smoke suite (includes setup/api, setup/rest, setup/graphql):

Copycp -R "$AGENT_KIT_HOME/skills/api-test-runner/template/setup" .

Run a canonical suite:

Copy$AGENT_KIT_HOME/skills/api-test-runner/scripts/api-test.sh --suite smoke-demo --out out/api-test-runner/results.json

Emit JUnit for CI reporters:

Copy$AGENT_KIT_HOME/skills/api-test-runner/scripts/api-test.sh --suite smoke-demo --junit out/api-test-runner/junit.xml

Generate a human-friendly summary (CI logs + $GITHUB_STEP_SUMMARY), based on the results JSON:

Copy$AGENT_KIT_HOME/skills/api-test-runner/scripts/api-test-summary.sh \
  --in out/api-test-runner/results.json \
  --out out/api-test-runner/summary.md \
  --slow 5

Include skipped cases (optional):

Copy$AGENT_KIT_HOME/skills/api-test-runner/scripts/api-test-summary.sh \
  --in out/api-test-runner/results.json \
  --show-skipped

Suite Manifests

Canonical location (committed):

• setup/api/suites/*.suite.json

Runner entrypoints:

• --suite <name> → resolves to setup/api/suites/<name>.suite.json
• --suite-file <path> → explicit override (recommended only when canonical path is not possible)

Notes:

• REST cases point at *.request.json (same schema used by rest.sh).
• GraphQL cases point at *.graphql + variables *.json (same inputs used by gql.sh).

Suite schema v1

Copy{
  "version": 1,
  "name": "smoke",
  "defaults": {
    "env": "staging",
    "noHistory": true,
    "rest": { "configDir": "setup/rest", "url": "", "token": "" },
    "graphql": { "configDir": "setup/graphql", "url": "", "jwt": "" }
  },
  "cases": [
    {
      "id": "rest.health",
      "type": "rest",
      "tags": ["smoke"],
      "env": "",
      "noHistory": true,
      "allowWrite": false,
      "configDir": "",
      "url": "",
      "token": "",
      "request": "setup/rest/requests/health.request.json"
    },
    {
      "id": "rest.auth.login_then_me",
      "type": "rest-flow",
      "tags": ["smoke"],
      "env": "",
      "noHistory": true,
      "allowWrite": true,
      "configDir": "",
      "url": "",
      "loginRequest": "setup/rest/requests/login.request.json",
      "tokenJq": ".accessToken",
      "request": "setup/rest/requests/me.request.json"
    },
    {
      "id": "graphql.countries",
      "type": "graphql",
      "tags": ["smoke"],
      "env": "",
      "noHistory": true,
      "allowWrite": false,
      "configDir": "",
      "url": "",
      "jwt": "",
      "op": "setup/graphql/operations/countries.graphql",
      "vars": "setup/graphql/operations/countries.variables.json",
      "expect": { "jq": "(.errors? | length // 0) == 0" }
    }
  ]
}

Notes:

• defaults.* are optional; per-case fields override defaults.
• defaults.noHistory (and case noHistory) map to --no-history on underlying rest.sh / gql.sh.
• defaults.rest.configDir / defaults.graphql.configDir default to setup/rest / setup/graphql.
• For REST and GraphQL endpoint selection, prefer url for CI determinism (avoids env preset drift).
• rest-flow runs loginRequest first, extracts a token (via tokenJq), then runs request with ACCESS_TOKEN=<token> (token is not printed in command snippets).

CI auth (GitHub Secrets / JWT login)

If your JWT expires, prefer logging in at runtime in CI using a suite-level auth block + a single JSON GitHub Secret.

How it works:

• You provide credentials via a JSON env var (default: API_TEST_AUTH_JSON).
• The runner logs in once per referenced profile (cached for the run) using either a REST or GraphQL login provider.
• For cases that specify token (REST) or jwt (GraphQL), the runner injects ACCESS_TOKEN for that case and does not rely on tokens(.local).env / jwts(.local).env.

Recommended secret schema (example):

Copy{
  "profiles": {
    "admin": { "username": "admin@example.com", "password": "..." },
    "member": { "username": "member@example.com", "password": "..." }
  }
}

Suite example (REST provider):

Copy{
  "version": 1,
  "name": "auth-smoke",
  "auth": {
    "provider": "rest",
    "secretEnv": "API_TEST_AUTH_JSON",
    "required": true,
    "rest": {
      "loginRequestTemplate": "setup/rest/requests/login.request.json",
      "credentialsJq": ".profiles[$profile] | select(.) | { username, password }",
      "tokenJq": ".accessToken"
    }
  },
  "defaults": {
    "noHistory": true,
    "rest": { "url": "https://<host>", "token": "member" },
    "graphql": { "url": "https://<host>/graphql", "jwt": "member" }
  },
  "cases": [
    { "id": "rest.me.member", "type": "rest", "token": "member", "request": "setup/rest/requests/me.request.json" },
    { "id": "graphql.me.admin", "type": "graphql", "jwt": "admin", "op": "setup/graphql/operations/me.graphql" }
  ]
}

Suite example (GraphQL provider):

Copy{
  "version": 1,
  "name": "auth-smoke",
  "auth": {
    "provider": "graphql",
    "secretEnv": "API_TEST_AUTH_JSON",
    "required": true,
    "graphql": {
      "loginOp": "setup/graphql/operations/login.ci.graphql",
      "loginVarsTemplate": "setup/graphql/operations/login.ci.variables.json",
      "credentialsJq": ".profiles[$profile] | select(.) | { email, password }",
      "tokenJq": ".. | objects | (.accessToken? // .access_token? // .token? // .jwt? // empty) | select(type==\"string\" and length>0) | ."
    }
  }
}

Defaults:

• Fail fast: if auth is configured but the secret env var is missing/empty, the runner exits 1 with a clear error.
• Optional override: set auth.required=false to disable suite auth when the secret is missing (useful for forks / local runs).

Assertions

• REST: assertions live in the request file:
  • expect.status (required when expect is present)
  • expect.jq (optional; evaluated with jq -e against the JSON response)
• GraphQL: the runner enforces .errors is empty by default, plus optional per-case expect.jq.

CLI flags

Core:

• --suite <name> / --suite-file <path>
• --out <path> (optional; JSON results file)
• --junit <path> (optional; JUnit XML)

Selection:

• --only <id1,id2,...>
• --skip <id1,id2,...>
• --tag <tag> (repeatable; all tags must match)

Control:

• --fail-fast (stop after first failure)
• --continue (continue after failures; default)

CI examples

Generic shell (write JSON + JUnit as CI artifacts):

Copy$AGENT_KIT_HOME/skills/api-test-runner/scripts/api-test.sh \
  --suite smoke \
  --out out/api-test-runner/results.json \
  --junit out/api-test-runner/junit.xml

GitHub Actions (runs the bundled public smoke suite using the template bootstrap):

• Example workflow file: .github/workflows/api-test-runner.yml

If you want to run your own suite in CI, replace the bootstrap step with your repo’s committed setup/api/.

Notes:

• Keep out/api-test-runner/results.json as the primary machine-readable artifact.
• Only upload per-case response files as artifacts if they are known to be non-sensitive.

GitHub Actions: matrix sharding by tags

For large suites, you can split a single suite into multiple CI jobs by tagging cases and running the runner in a matrix.

Suite tagging pattern (make shard tags mutually exclusive):

Copy{
  "cases": [
    { "id": "graphql.health", "type": "graphql", "tags": ["staging", "shard:0"], "op": "..." },
    { "id": "graphql.notifications", "type": "graphql", "tags": ["staging", "shard:1"], "op": "..." }
  ]
}

Workflow example:

Copystrategy:
  fail-fast: false
  matrix:
    shard: ["0", "1"]

steps:
  - name: Run suite shard
    env:
      AGENT_KIT_HOME: ${{ github.workspace }}
      API_TEST_AUTH_JSON: ${{ secrets.API_TEST_AUTH_JSON }}
    run: |
      skills/api-test-runner/scripts/api-test.sh \
        --suite my-suite \
        --tag staging \
        --tag "shard:${{ matrix.shard }}" \
        --out "out/api-test-runner/results.shard-${{ matrix.shard }}.json" \
        --junit "out/api-test-runner/junit.shard-${{ matrix.shard }}.xml"

Notes:

• --tag is repeatable and uses AND semantics (a case must include all tag filters to run).
• Make shard tags mutually exclusive to avoid duplicate coverage across jobs.
• Use per-shard output filenames to avoid artifact collisions.

Safety defaults

Write-capable cases are denied by default.

• REST write detection: HTTP methods other than GET/HEAD/OPTIONS.
• GraphQL write detection: operation type mutation (best-effort).

Allow writes only when:

• The case has allowWrite: true, AND
• Either:
  • effective env is local, OR
  • the runner is invoked with --allow-writes (or API_TEST_ALLOW_WRITES=1).

Results

• JSON results are always emitted to stdout (single JSON object). A one-line summary is emitted to stderr.
• Use --out <path> to also write the JSON results to a file.
• Use --junit <path> to write a JUnit XML report.

Result JSON schema (v1)

Top-level fields:

• version: integer
• suite: suite name
• suiteFile: path to suite file (relative to repo root)
• runId: timestamp id
• startedAt / finishedAt: UTC timestamps
• outputDir: output directory (relative to repo root)
• summary: { total, passed, failed, skipped }
• cases[]: per-case status objects

Per-case fields:

• id, type, status (passed|failed|skipped), durationMs
• tags (array)
• command (replayable snippet; no secrets)
• message (reason for fail/skip; stable-ish tokens)
• assertions (GraphQL only; includes defaultNoErrors and optional jq)
• stdoutFile / stderrFile (paths under out/api-test-runner/<runId>/ when executed)

Exit codes:

• 0: all selected cases passed
• 2: one or more cases failed
• 1: invalid inputs / schema / missing files

References

• Guide: skills/api-test-runner/references/API_TEST_RUNNER_GUIDE.md
• Progress plan: docs/progress/archived/20260109_ci-api-test-runner.md