Automating in CI

Tests that depend on someone remembering to run them stop being run. Continuous integration (CI) runs them on every change, by itself, and blocks the change when they fail. ApiWorkbench's command-line runner, apiworkbench-cli, is made for that.

How the pieces fit

Piece	Role
The collection file	Your requests and tests, exported from ApiWorkbench and committed
An environment file, or --env-var	The address and credentials for the target
apiworkbench-cli	Runs the collection and exits with a result code
The CI system	Runs the command on each change, and reads the exit code

The exit code is the whole contract between the runner and CI:

Code	Meaning
0	Every request and test passed
1	A request or test failed
2	Wrong arguments, an unreadable file, or a report that could not be written

Step 1: get the tests into the repository

1. In ApiWorkbench, export the collection. See Export and share.

2. Check the file for secrets before committing. Tokens belong in the CI system's secret store, not in the file.

3. Commit it, for example as tests/shop-api.apiworkbench.json. Commit any data files beside it.

The runner also reads Postman collections, so an existing Postman suite can run unchanged.

Step 2: run it locally first

Get the command right on your own machine before putting it in CI:

apiworkbench-cli run tests/shop-api.apiworkbench.json

With the snap, the command is apiworkbench.cli. Useful options:

Option	Does
-e <file>	Use an environment file
--env-var name=value	Set one environment variable. Repeatable
-d <file>	Use a CSV or JSON data file
-n <count>	Run that many iterations
--folder <path>	Run only that folder. Repeatable
--bail	Stop at the first failed request
--timeout-request <ms>	Give up on a request after this long
-r cli,junit,html,json	Choose the reporters
--reporter-junit-export <file>	Where the JUnit report goes

apiworkbench-cli run --help prints the full list, and Command-line runner explains each one.

To see it work end to end, start the Shop API sample mock and run its collection:

apiworkbench-cli run shop-api.apiworkbench.json -d shop-orders.csv

Step 3: add it to CI

GitHub Actions

name: API tests

on: [push, pull_request]

jobs:
  api-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install apiworkbench-cli
        run: |
          curl -fsSLO https://github.com/Dhiva-Labs/ApiWorkbench/releases/download/v1.4.0/apiworkbench_1.4.0_amd64.deb
          sudo apt-get update
          sudo apt-get install -y ./apiworkbench_1.4.0_amd64.deb

      - name: Run the API tests
        env:
          API_TOKEN: ${{ secrets.API_TOKEN }}
        run: |
          apiworkbench-cli run tests/shop-api.apiworkbench.json \
            --env-var "baseUrl=${{ vars.API_BASE_URL }}" \
            --env-var "token=$API_TOKEN" \
            -r cli,junit,html \
            --reporter-junit-export results/junit.xml \
            --reporter-html-export results/report.html

      - name: Keep the reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: api-test-reports
          path: results/

if: always() keeps the reports when the tests fail, which is exactly when you want them.

GitLab CI

api-tests:
  image: ubuntu:24.04
  before_script:
    - apt-get update
    - apt-get install -y curl ca-certificates
    - curl -fsSLO https://github.com/Dhiva-Labs/ApiWorkbench/releases/download/v1.4.0/apiworkbench_1.4.0_amd64.deb
    - apt-get install -y ./apiworkbench_1.4.0_amd64.deb
  script:
    - >
      apiworkbench-cli run tests/shop-api.apiworkbench.json
      --env-var "baseUrl=$API_BASE_URL"
      --env-var "token=$API_TOKEN"
      -r cli,junit
      --reporter-junit-export results/junit.xml
  artifacts:
    when: always
    reports:
      junit: results/junit.xml

GitLab reads the JUnit file and shows each failed test in the merge request.

What the tests run against

CI needs a server to call. Three common answers:

Target	When	How
A deployed test environment	You have a staging server	Pass its address as baseUrl
The API started inside the job	The API can run in CI	Start it in an earlier step, wait for its health check, then run the tests against localhost
A mock	You are testing a client, or the API is not ready	Run the mock where CI can reach it

For the second, wait for the service before testing, or the first requests fail for the wrong reason:

until curl -fs http://localhost:8080/health > /dev/null; do sleep 1; done

Secrets

• Store tokens and passwords in the CI system's secrets, and pass them with --env-var.
• Never commit them, and never print them. The runner masks secret variables in its reports.
• Use an account made for CI, with the least access the tests need.
• Check that an uploaded report does not contain response bodies with private data before making it public.

Keeping the pipeline useful

• Fast first. Run a small smoke set on every push, and the full suite on merge or nightly.
• Fail loudly, with detail. Keep the JUnit or HTML report as an artifact.
• No flaky tests. A test that fails at random teaches everyone to ignore red. Fix or remove it. Common causes: depending on leftover data, on timing, or on the order of a list.
• Each run cleans up after itself, or uses fresh data from dynamic variables.
• Pin the runner's version, as the examples do, so a new release cannot change your results unannounced. Update it on purpose.

Check yourself

QuizAutomating in CI

Question 1 of 6

apiworkbench-cli finishes with exit code 1. What happened?

Score: 0

Next steps

• Command-line runner
• Monitoring
• Data-driven testing