The Reality of Network Partitions

In a distributed system, network failures are not an anomaly; they are a statistical certainty. When a client sends a POST request to an API, and the connection drops before the server can return a response, the client is left in an unknown state.

Did the server process the request before the connection died? Or did the request never reach the server at all?

The Retries Dilemma: The standard behavior for robust clients is to retry the request. However, if the API is not designed to handle duplicate requests safely, a single network hiccup can result in a user being charged twice, or two identical servers being provisioned.

Understanding Idempotency

An API endpoint is considered idempotent if making multiple identical requests has the same effect on the server state as making a single request.

• GET, PUT, and DELETE are naturally idempotent by HTTP design specifications.
• POST (typically used for creation or complex mutations) is not naturally idempotent.

To safely scale distributed infrastructure, every state-mutating POST endpoint must be engineered for idempotency.

The Idempotency Key Architecture

The industry-standard solution is the introduction of an Idempotency-Key header.

When a client initiates a mutation, it generates a unique identifier (usually a UUIDv4) and includes it in the request headers.

POST /v1/servers/provision HTTP/1.1
Host: api.example.com
Authorization: Bearer token_123
Idempotency-Key: f3b4d8c1-9a2e-4e7a-8b1c-3f2d1e4a5b6c

{"region": "eu-west", "size": "large"}

The Server-Side Implementation

When the API gateway receives this request, it follows a strict operational flow:

1. Check the Cache: The server queries a high-speed, distributed Key-Value store (like Redis or an Edge KV) for the Idempotency-Key.
2. In-Progress Lock: If the key exists and the status is "processing," the server immediately returns a 409 Conflict or holds the connection until the original request finishes.
3. Return Cached Response: If the key exists and the status is "completed," the server returns the exact HTTP response (status code and body) that was generated by the original successful request, without executing the core business logic again.
4. Execute and Store: If the key does not exist, the server executes the business logic, caches the response against the Idempotency-Key with a TTL (e.g., 24 hours), and returns the payload to the client.

Cross-Service Consistency

This pattern becomes absolutely critical when orchestrating actions across multiple microservices.

For instance, if a user requests the programmatic creation of a new marketing campaign, the orchestration layer must provision a landing page via MyFunnelAPI and set up the corresponding DNS routing. If the orchestration process crashes halfway through, it must be able to resume and retry those API calls safely without accidentally creating duplicate funnels.

By enforcing strict idempotency keys at the gateway layer, distributed systems can self-heal and retry operations aggressively without data corruption.