Flaky Payments lab

Real services fail: they time out, answer with errors now and then, refuse you for calling too often, or send back rubbish. This mock does all of that on purpose, so you can see what each failure looks like from the client's side and learn to test for it.

Level: intermediate · Time: about 45 minutes · Before this: Todo API lab

Some tests here are meant to fail

In the other labs, green means good. Here, several requests are built to fail so that you can read the failure. The lab says which.

Download

• flaky-payments.mockstudio.json — the mock API, for Mock Studio
• flaky-payments.apiworkbench.json — the requests and tests, for ApiWorkbench

Set up

1. In Mock Studio, select Import, choose Mock Studio project file and pick flaky-payments.mockstudio.json. Select Start server. The mock is at http://localhost:4030.

2. In ApiWorkbench, select Import, choose flaky-payments.apiworkbench.json and select Import.

The API

Method	Path	Behaviour	Built with
POST	/payments	About 7 in 10 calls: 201. About 2 in 10: 500. About 1 in 10: 503	Random / weighted
GET	/payments/:id/status	pending, pending, completed, then round again	Sequence
GET	/rates	Every third call: 429 with Retry-After	Sequence
GET	/slow	Answers after 3 seconds	Latency
GET	/wobbly	Answers after 0.1 to 1 second, at random	Latency with jitter
GET	/broken-json	200, but the body is not valid JSON	Fault: malformed body
GET	/empty	200 with no body	Fault: empty response
GET	/drop	Closes the connection without answering	Fault: drop connection
GET	/hang	Never answers	Fault: hang

Part 1: random failures

A service that works most of the time is the hardest kind to test by hand, because one try tells you nothing.

1. Open Charge a card and select Send a few times. Most answers are 201. Sooner or later you get a 500 or a 503.

2. Open the ⋮ menu beside Send, choose Runner, set 20 iterations and select Run.

3. Count the failures. Expect around six, but not exactly: it is random.

In Mock Studio, open Charge a card. The strategy is Random / weighted, and the three responses have weights 7, 2 and 1. A weight is a share, not a percentage: 7 out of a total of 10.

What to take from it: a client that calls this API must expect a 5xx and decide what to do: retry, tell the user, or give up. Look at the 503: it carries a Retry-After: 2 header, which is the server asking for a pause before the next try. See Resilience testing.

Retrying a payment

POST is not idempotent. Retrying a failed charge blindly can charge twice. Real payment APIs solve this with an idempotency key: a unique id the client sends so the server can recognise a repeat. A good test asks: what happens if this request is sent twice?

Part 2: a sequence, for polling

Some work finishes later. The client asks "done yet?" until it is.

1. Open Poll the payment status and select Send three times.

2. Read the status field each time: pending, pending, completed.

3. Send a fourth time: pending again. The sequence starts over.

In Mock Studio the strategy is Sequence (round-robin): each call gets the next response in the list. This is the simplest way to test a client that polls or retries, because the outcome of each attempt is known in advance.

Part 3: rate limits

1. Open Exchange rates (rate limited), open the ⋮ menu beside Send, choose Runner, set 3 iterations and select Run.

2. The first two pass. The third fails: expected 200, got 429.

3. Send it once by hand until you get the 429, and open the response's Headers tab. Find Retry-After: 1.

A 429 is not a bug. It is the API protecting itself. The test worth writing is not "never get a 429" but "when a 429 comes, the client waits for Retry-After and tries again".

Part 4: slow answers

1. Open Slow answer and select Send. Watch the three seconds pass.

2. The status is 200, but open the response's Tests tab: Time below (ms) failed, with took 3005 ms (limit 2000 ms) or close to it.

This test fails on purpose. The response is correct and still unacceptable. Speed is part of the contract, and only a time test catches a slow success.

Now open Wobbly answer and send it several times. The time changes on every call. One measurement of a variable thing is meaningless, so measure many:

1. Open the ⋮ menu beside Send and choose Load test.

2. Set 10 users and 20 iterations, and run it.

3. Read p50 (median), p95 and p99.

The median will be near 550 ms and p99 near 1000 ms. The average hides the slow tail; the percentiles show it. Load testing explains how to read them.

Part 5: answers that are not answers

These four are the failures a status code cannot describe.

Request	What you see in ApiWorkbench	What it teaches
Broken JSON	200, and the JSON test fails with body is not valid JSON	A 200 does not mean the body is usable. Always check content
Empty answer	200, and the body test fails with not found in body	The same, for a missing body
Dropped connection	No status at all: a connection error	Some failures have no response. The client needs a plan for that too
Never answers	Nothing, until you select Cancel or the timeout ends it	Without a timeout, a client waits for ever

Send each one and read the result.

For Never answers, the wait ends when the response timeout in ApiWorkbench's Settings runs out. Shorten it to a few seconds to see the timeout message quickly, then set it back. See App settings and Per-request settings.

Use this mock to test your own app

Point the app you are building at http://localhost:4030 and watch what it does on a hang, a drop or a broken body. Does it show a helpful message? Does it freeze? That is resilience testing, and it needs no code on the server side. To reach the mock from a phone, see Share on LAN or the internet.

Part 6: build a failure yourself

1. In Mock Studio, add an endpoint: GET /refunds/:id.

2. Choose Sequence (round-robin) and add three responses: a 503, another 503, then a 200 with a body of your choice.

3. Call it three times from ApiWorkbench. You have built "fails twice, then works": the standard case for testing retry logic.

4. Now set the second 503's fault to Hang (never respond) and try again.

Latency and faults lists every fault and what the client sees for each.

Check yourself

QuizFlaky Payments lab

Question 1 of 6

Charge a card uses Random / weighted with weights 7 (201), 2 (500) and 1 (503). You run it 20 times. What should you expect?

Score: 0

What you learned

• Random failures need many runs, not one.
• A sequence makes retries and polling testable, because each attempt's result is known.
• A 429 with Retry-After is the server asking you to wait.
• A correct answer that arrives too late is still a failure.
• Some failures have no status code. Clients need timeouts.

Next steps

• Resilience testing
• Load testing
• Response strategies