Load testing
A load test sends many requests at once and measures what happens to speed and errors. An API that answers one user in 50 milliseconds may take five seconds, or fall over, with five hundred. This page explains the ideas, how to design a test, and how to read the numbers.
Only load-test what you may
A load test is, mechanically, a small denial-of-service attack. Run one only against a system you own or have written permission to test, and never against production without the agreement of the people who run it. Practise on a mock on your own machine.
The words
| Term | Meaning |
|---|---|
| Virtual user | One simulated client, sending requests one after another |
| Iteration | One pass through the steps by one virtual user |
| Ramp-up | Starting the users gradually over a period, not all at once |
| Think time | A pause between steps, like a person reading a page |
| Throughput | Requests completed per second |
| Latency | How long one request took |
| Error rate | The share of requests that failed |
| Percentile | See below |
Percentiles, plainly
Sort every response time from fastest to slowest. Then:
- p50 (the median) is the middle one. Half the requests were faster.
- p95 is the one 95% of the way along. Only 5 in 100 were slower.
- p99 is 99% of the way along. Only 1 in 100 was slower.
Why not the average? Take ten requests: nine at 100 ms and one at 5,000 ms. The average is 590 ms, a number that describes none of them. p50 is 100 ms (the typical experience) and the slowest is 5,000 ms (the bad one). Percentiles show both; the average hides both.
Targets are written as percentiles for that reason: "p95 under 300 ms" means 95 of every 100 requests finish within 300 ms.
Kinds of load test
| Kind | Shape | Question |
|---|---|---|
| Baseline | A few users, a short time | What is normal? |
| Load | The expected number of users | Does it cope with a normal busy day? |
| Stress | More and more users until it breaks | Where is the limit, and how does it fail? |
| Spike | A sudden jump | Does it survive a burst, and recover? |
| Soak | Normal load for hours | Does it degrade over time? |
Start with a baseline. Without one, no later number means anything.
Designing a test
Write the goal as a number. "With 50 users, p95 stays under 500 ms and the error rate under 1%."
Choose a realistic journey, not one endpoint in isolation. Sign in, browse, order.
Decide the users. Start low: 5. Then 10, 25, 50. Jumping straight to 500 tells you it broke, not when.
Add a ramp-up, so the users arrive over several seconds. Everyone starting at the same instant is a spike test, whether or not you meant one.
Add think time, unless you want the maximum possible pressure. Real users pause.
Decide the length. A number of iterations per user, or a duration. Thirty seconds is too short to trust; a few minutes is better.
Decide what a failure is. By default, a status of 400 or above. Add tests for the status you expect.
Run one in ApiWorkbench
Open the ⋯ menu of a collection or folder, or the ⋮ menu beside Send, and choose Load test.
Set the users, the iterations or duration, the ramp-up and the think time.
Choose One by one to run the requests in order, so that captured values flow from step to step, or All at once to fire every request of an iteration together.
Start the run and watch the live numbers.
ApiWorkbench runs up to 1,000 users. Extract rules carry values between steps, as captures do. Scripts do not run in a load test; no-code tests do. Every setting is described in Load testing in ApiWorkbench.
Run one in Mock Studio
Mock Studio's Flows run a scripted journey with many virtual users against the mock, or against any address you put in Target base URL. Values are extracted from one step and used in the next as {{vars.name}}; {{vu}} is the virtual user's number and {{iteration}} the pass.
The Shop API sample includes a ready-made flow. See Flows.
Reading the results
| Number | Healthy | Worrying |
|---|---|---|
| Requests per second | Rises as users are added | Stays flat or falls as users are added: the limit is reached |
| p50 | Steady as load grows | Climbing |
| p95 and p99 | A small multiple of p50 | Many times p50: a slow tail, some users suffering |
| Max | Near p99 | Enormous: single requests stalling |
| Failed | Zero, or close | Any steady share. Read which errors |
Three patterns to recognise:
- The knee. Add users and throughput stops rising while latency shoots up. The number of users just before that is the capacity.
- The long tail. p50 is fine but p99 is ten times larger. Most users are happy; some are waiting. Look for a lock, a slow query or a cold cache.
- Errors under load only. Everything passes with 5 users and 3% fail with 50. Look for timeouts, connection limits and rate limiting (429).
Always read the errors, not only their count. A wall of 429s means a rate limiter did its job, which is a different story from 500s.
Reports
ApiWorkbench can log every call and save the run as an HTML report, a CSV log or a JSON export. Keep the baseline's report: comparing a later run with it is how you spot a regression.
A report can contain request and response details. Read it before sharing, in case it holds tokens or personal data.
Getting numbers you can trust
- Run it twice. One run is an anecdote.
- Change one thing at a time: users, or the code, not both.
- Know what you are measuring. A test against a mock on your own machine measures your machine. It is practice, not evidence about the real API.
- Watch the server too. CPU, memory and database connections explain what the response times only report.
- Mind your own machine. If the computer running the test is at its limit, the numbers describe it, not the server.
- Use test data and test accounts. A load test creates a great many records.
Next steps
- Load testing in ApiWorkbench
- Flows in Mock Studio
- Flaky Payments lab: read percentiles on an endpoint with random delay