API retry and idempotency testing: an acceptance checklist

Before accepting an API integration, test what happens when a request succeeds but its response never reaches the caller. Repeat the same logical operation, inspect the stored business result, and check any downstream effects. A successful HTTP response on the retry is insufficient evidence that the system avoided a duplicate reservation, payment, or work order.

This checklist is for engineering leads and prime contractors reviewing a software work package. It supplies a proposed failure matrix and a worked review procedure. The examples are hypothetical engineering aids, not Rainier client requirements or reported test results.

What does idempotency guarantee?

An idempotent operation has the same intended server effect when repeated as it does when performed once. It need not produce identical response bytes, and separate access-log entries are compatible with idempotency. HTTP Semantics, RFC 9110 section 9.2.2 defines this distinction and cautions against automatically retrying non-idempotent methods without knowledge that the operation is safe to repeat or was never applied.

For an application programming interface (API) that creates records through POST, require an explicit retry contract. Merely sending a header named Idempotency-Key does not make an endpoint implement one. The caller and service must agree how an operation is identified and how repeat attempts are handled.

An idempotency key identifies one logical operation across attempts. A trace or request identifier can identify an individual attempt. Keep those concepts separate in the review: the evidence should let an engineer follow several attempts to one business outcome.

Define the contract before writing the tests

Use this worksheet in the interface review. The proposed decisions below are engineering recommendations, not a universal protocol specification.

Do not copy one provider's retention period or error behavior into every integration. For example, Stripe's idempotent-request reference says API v1 retains the first status and body after execution begins, including 500 results. Its v1 keys may be removed once at least 24 hours old, and reuse after pruning creates a new request. The same page describes different retry behavior for API v2. Record the actual API version and contract you are integrating with.

Key expiry deserves a business decision. If an offline client can replay work after protection has expired, the receiving system may need a durable business reference or a reconciliation step. A short-lived retry key alone cannot establish that a much later submission is a new customer instruction.

A failure matrix for API retry acceptance

Choose observable invariants before injecting faults. For the hypothetical reservation below, they are one reservation for the logical operation, one inventory hold, and a recoverable outcome for the authorized caller. Record notification delivery separately if it crosses another service boundary.

For each row, record pass, fail, or not tested. An unavailable dependency leaves an evidence gap. A passing stub test establishes the local handling of that stub's behavior, not the remote service's duplicate protection.

Run concurrency checks against the persistence mechanism and deployment topology being accepted. A single-process test with an in-memory map does not exercise two application instances competing over the same key. Use a barrier or test hook to create the relevant overlap; sending requests close together and hoping for a race makes failures difficult to reproduce.

Worked review: a reservation commits, then the connection drops

Assume a hypothetical service with POST /reservations, an authenticated tenant, and a client-generated operation key. The agreed contract returns the original reservation reference when a completed request is repeated within its retention window. A reservation and its inventory hold are stored in one database transaction. Email is outside this example's initial acceptance boundary.

Rainier's published Ever engagement describes reservation and sell flows with a typed PostgreSQL data layer. That is relevant delivery context only; the following endpoint, key design, and fault scenario are illustrative and do not describe Ever's implementation.

Use this review procedure:

1. Create a synthetic tenant, available inventory, and a fresh operation key. Capture the initial reservation and hold counts for that fixture.
2. Submit the request through a test harness that can suppress the response after a confirmed commit. Correlate the commit with the operation key. A client-side timeout without server evidence does not establish this fault location.
3. Verify that the client received no completed result. Preserve its attempt identifier and the service's commit evidence.
4. Retry with the same tenant, key, and business input. Do not generate a new key in the retry loop.
5. Read the authoritative reservation and inventory state. Confirm that the retry resolves to the original reservation and did not create another hold.
6. Restart an application instance and repeat within the retention window. Confirm that duplicate protection survives the restart as required by the contract.
7. Save the fixture setup, fault-injection point, request sequence, redacted responses, state assertions, and revision together. Remove the synthetic records through the test environment's cleanup procedure.

A useful acceptance assertion is: “After the response is lost following commit, the repeat request returns reservation R, and the fixture contains exactly one reservation and one hold for this logical operation.” State the observation window and how pending work is drained before checking counts. Otherwise a delayed duplicate may arrive after the assertion.

A hypothetical failure would be two reservation rows with successful responses on both attempts. The HTTP checks passed; the business invariant failed. Another failure would be one row but no supported way for the caller to discover it, leaving the operator unable to decide whether to retry or begin again.

Inspect the write boundary and downstream effects

Ask the implementer to show how reserving the key and recording the business change stay consistent under a crash. Amazon's explanation of idempotent APIs discusses caller-provided operation identifiers and the need for atomic recording of the token and associated mutations. It also covers parameter mismatch and late-arriving requests.

For this worksheet, review three implementation questions:

• Can two instances both decide that a key is unused before either records it? Ask for the persistence constraint or equivalent coordination and the concurrent test result.
• Can the business write commit while the duplicate-protection record disappears? Ask for the transaction boundary and a crash-recovery test at the boundary that matters.
• Can the key be marked complete while a required external effect is still unknown? Ask which component owns reconciliation and how its state is exposed to operations staff.

A local database transaction does not include an unrelated provider's operation. If the service also charges a payment provider or publishes work to a separate system, extend the evidence boundary. Review the provider's retry contract, the persisted operation reference, and what recovery does after an acknowledgment is lost. An outbox can preserve work awaiting delivery, but the receiving effect still needs duplicate handling or a reconciliation strategy.

Avoid an unqualified “exactly once” acceptance statement. Name the effect, system boundary, retention window, and tested failure conditions. This makes the promise something a receiving engineer can inspect.

Review retry limits as well as duplicate protection

Retries can amplify load on an already struggling dependency. Amazon's guidance on timeouts, backoff, and jitter explains that tradeoff and the use of delays with random variation to spread retry traffic.

For the delivered integration, request a small policy table identifying the retryable conditions, maximum attempts or elapsed-time budget, delay policy, and terminal recovery action. Document whether the client library, application, or gateway owns each retry. Verify actual request counts when more than one layer can retry.

Do not prescribe one retry count for every workflow. A user waiting at a checkout and a background reconciliation job have different deadlines. Once the agreed budget ends, the application should expose a usable pending or unknown state with an authorized recovery path, rather than inviting a blind resubmission with a new key.

Turn the results into a handover decision

Use this record for the review package:

Operation / API version / tenant scope:
Revision / deployment / test date:
Key retention and retry policy:
Fixture and initial business state:
Fault location and proof it was reached:
Attempt identifiers / redacted operation reference:
Expected response and business effects:
Observed response, persisted state, downstream state:
Observation window / pending-work drain condition:
Pass, fail, or not tested / evidence links:
Known boundary or unavailable dependency:
Finding owner / recovery procedure / retest revision:
Acceptance decision and authorized reviewer:

Keep credentials, customer payloads, and sensitive identifiers out of shared test evidence. Use synthetic data and a redacted correlation reference sufficient for authorized engineers to locate the underlying records.

The broader software subcontractor acceptance checklist connects this technical record to integration ownership, deployment, rollback, and handover. For an inherited application, the legacy assessment worksheet helps establish which system owns the data before changing retry behavior.

Bring the operation contract, dependencies, and failure matrix to the scoping discussion. Rainier's API and integration delivery scope provides a starting point for a bounded implementation or review package with explicit acceptance evidence.

Discuss an API integration and its failure scenarios→