Most APIs are designed for the happy path and discover the unhappy one in production, at 3am, during a traffic spike. Resilience is not a feature you add later; it is a set of decisions you make in the first design review. Here are the ones that matter most.
Idempotency is not optional
The network will retry your requests whether you plan for it or not. Clients time out and retry, load balancers replay, users double-click. If a POST /payments can run twice, eventually it will.
The mechanism is simple: the client generates a unique key, the server records the result against that key, and replays return the stored result instead of re-executing.
async function createPayment(req: PaymentRequest, key: string) {
const existing = await store.get(key);
if (existing) return existing; // replay: return the original result
const result = await processPayment(req);
await store.put(key, result, { ttl: '24h' });
return result;
}The subtlety is the race: two concurrent requests with the same key. Use a conditional write (INSERT ... ON CONFLICT DO NOTHING) so exactly one wins, and have the loser wait for the winner’s result.
Backpressure beats buffering
When a service is overwhelmed, the instinct is to buffer: queue the excess and work through it. This feels resilient and is quietly catastrophic. Unbounded queues convert a latency problem into a memory problem, and a memory problem into an outage.
The healthier response is to reject early and clearly. A fast 429 Too Many Requests with a Retry-After header lets clients back off gracefully. A silent 30-second hang does not.
Set a concurrency limit
Cap in-flight requests per instance based on measured capacity, not hope.
Bound every queue
A full bounded queue is a signal. An unbounded queue is a time bomb.
Shed load explicitly
Return 429 with backoff guidance rather than degrading everyone equally.
Version at the boundary, not everywhere
API versioning terrifies teams because they imagine maintaining every version forever. The trick is to keep versioning at the edge and normalize to a single internal representation immediately.
Your business logic should never contain an if (version === 'v1') branch. Adapters translate inbound requests to the canonical shape and outbound responses back to the requested version. When you retire a version, you delete an adapter, not perform surgery through the core.
Design for graceful degradation
The goal is not “never fail.” It is “fail small.” A product page that loads without its recommendation carousel is a minor annoyance; one that returns a 500 because the recommendation service is slow is an outage.
| Dependency | Criticality | Strategy on failure |
|---|---|---|
| Auth service | Critical | Fail closed, return 503 |
| Product catalog | Critical | Serve cached, flag staleness |
| Recommendations | Optional | Omit section, log, move on |
| Analytics beacon | Cosmetic | Fire-and-forget, never block |
Classify every dependency by criticality before you wire it in. The optional ones should be behind timeouts short enough that their failure is invisible to the user.
The most reliable systems are not the ones with the fewest failures. They are the ones where failures stay contained.
The checklist
Before an API goes live, I want yes answers to all of these:
- Can every write be safely retried?
- Is every queue and buffer bounded?
- Does every outbound call have a timeout and a fallback?
- Can we deploy a breaking change without a client rewrite?
- When a dependency dies, does the blast radius stay local?
None of this is exotic. It is a collection of small, deliberate decisions that compound. Make them early, and the 3am page never comes.
