Iurii Deduplicates: Idempotency Keys: Building Retries That Don't Double-Charge

A customer clicks "Pay €49." Your API charges their card, writes the order, and starts sending the response. Somewhere on the way back, the connection drops — a flaky mobile network, a load balancer timeout, a laptop lid closing. The client never sees a 200. So it does the only reasonable thing: it retries. Your API charges the card again.

Now you have one angry customer, two charges, and a refund to process. The code worked perfectly both times. That's the trap. This is the problem an idempotency key API solves: making a repeated request safe to send.

The painful part is that nothing in your stack is broken. The card processor did its job twice because it was asked twice. The retry logic did its job because, from the client's point of view, the first request genuinely might have failed. The bug isn't in any one component — it's in the gap between "the work happened" and "the client knows the work happened." That gap is where money gets charged twice.

This is a different problem from the one I covered in Stripe webhooks in production. That article is about incoming events — Stripe pushing notifications to your endpoint, and how you deduplicate them by event ID. This article is the mirror image: outgoing requests from a client to your API, and how you make a retry safe with a client-supplied Idempotency-Key. Same underlying concern, opposite direction of traffic.

A Timeout Doesn't Mean It Failed

The first thing to internalize: when a request times out, you have learned almost nothing. The request might have never reached the server. It might have reached the server, done the work, and died on the response path. From the caller's side these two cases are indistinguishable — both look like silence.

This isn't a flaw you can fix by being careful. It's a property of networks. The classic framing is the Two Generals problem: you cannot build a protocol over an unreliable channel that guarantees both sides agree on whether a message was delivered, using a finite number of messages. There is no acknowledgment scheme that closes the gap completely, because the acknowledgment itself can be lost.

So retries are not optional. If you don't retry on timeout, you lose legitimate operations every time the network hiccups — and it always hiccups. Mobile clients especially: a request that times out on a train going through a tunnel did not fail in any meaningful sense; the user just moved. The correct behaviour is to retry. Which means your server will receive the same logical operation more than once. You cannot prevent that. You can only decide what happens when it arrives.

At-Least-Once Is What You Get; Exactly-Once Is What You Build

There's a phrase that causes a lot of confusion: "exactly-once delivery." It mostly doesn't exist. Networks and message systems give you, at best, at-least-once delivery — the message arrives one or more times. The transport cannot promise it arrives exactly once, for the same Two Generals reason.

What you actually want is not exactly-once delivery — it's exactly-once effect. The card gets charged once even if the request arrives three times. And that property is built on the receiving side, not granted by the wire. You make the operation idempotent: applying it N times produces the same result as applying it once.

That distinction matters because it tells you where to spend your effort. You will not find a transport, queue, or framework that makes double-charges impossible for you. You build the guarantee yourself, in your handler, with a record of what you've already done. The rest of this article is how.

Idempotency-Key as a Contract Between Client and Server

The mechanism is a single HTTP header. The client generates a unique key per logical operation and sends it with the request:

POST /v1/charges HTTP/1.1
Content-Type: application/json
Idempotency-Key: a1b2c3d4-9f8e-7d6c-5b4a-3e2f1d0c9b8a
 
{ "amount": 4900, "currency": "eur", "customer": "cus_8812" }

The contract has two halves. The client promises: a retry of this same operation carries the same key. The server promises: it will execute the operation at most once per key, and a repeated key returns the original result instead of doing the work again.

This is exactly how Stripe's own API does it — you pass an Idempotency-Key header, and Stripe guarantees the request runs once even if you send it repeatedly — within the key's retention window. It's worth copying because it's a proven, widely understood pattern, and if your clients have ever integrated Stripe, they already know the shape of it.

The key is a UUID — any unique, collision-resistant value works, but a UUID is the obvious default. (If you want a key that's also sortable by creation time for easier debugging, UUID v7 is a good choice.) The one rule the client must not break: generate the key once per logical operation, not once per HTTP attempt. If a retry generates a fresh key, the server sees a brand-new operation and charges again — the whole mechanism collapses. The key belongs to "the user is paying for order #4821," not to "this particular TCP connection."

This also handles the humbler case: the double-click. A user mashing the Pay button twice should produce one charge. If the button is wired to a single operation with a single key, the second click carries the same key and the server short-circuits. No special debounce logic needed — idempotency covers it for free.

The Server Side: A Key Table in Postgres

The server needs durable memory of which keys it has seen and what it answered. A table:

CREATE TABLE idempotency_keys (
  key                  text PRIMARY KEY,
  request_fingerprint  text        NOT NULL,
  status               text        NOT NULL DEFAULT 'in_progress',
  response_code        integer,
  response_body        jsonb,
  created_at           timestamptz NOT NULL DEFAULT now(),
  expires_at           timestamptz NOT NULL DEFAULT now() + interval '24 hours',
  CONSTRAINT status_check CHECK (status IN ('in_progress', 'completed'))
);
 
CREATE INDEX idx_idempotency_expires ON idempotency_keys (expires_at);

A few fields earn their place:

• key is the primary key. The unique constraint is the heart of the mechanism — two requests with the same key cannot both insert.
• request_fingerprint is a hash of the request body. It catches a dangerous case: the same key sent with a different body. That's a client bug, and silently returning the old response would hide it. More on this below.
• status distinguishes a request that's still running (in_progress) from one that finished (completed). This is what protects you against two retries arriving at the same time.
• response_code / response_body store the answer so a repeat can replay it without re-running the work.
• expires_at bounds how long the key is honoured. After the retry window closes, the row can go.

Claiming the Key Atomically

The first thing the handler does is try to claim the key. The trick is to do the claim and the duplicate-check in a single atomic statement, so two concurrent requests can't both think they're first:

// Illustrative handler — Node.js + node-postgres (pg)
import { createHash } from "node:crypto";
import type { Pool } from "pg";
 
function fingerprint(body: unknown): string {
  return createHash("sha256").update(JSON.stringify(body)).digest("hex");
}
 
async function handleCharge(pool: Pool, key: string, body: ChargeRequest) {
  const fp = fingerprint(body);
 
  // Try to claim the key. ON CONFLICT DO NOTHING means: if the row already
  // exists, this inserts nothing and returns zero rows.
  const claim = await pool.query(
    `INSERT INTO idempotency_keys (key, request_fingerprint, status)
     VALUES ($1, $2, 'in_progress')
     ON CONFLICT (key) DO NOTHING
     RETURNING key`,
    [key, fp]
  );
 
  const weClaimedIt = claim.rowCount === 1;
 
  if (weClaimedIt) {
    return runChargeAndStore(pool, key, fp, body);
  }
 
  // Key already existed — someone got here first. Read its current state.
  return replayOrReject(pool, key, fp);
}

If INSERT ... ON CONFLICT DO NOTHING returns a row, this request won the race and owns the operation. If it returns nothing, the key already existed — either from a previous completed request or from a concurrent in-flight one — and we branch into replay logic.

Replaying or Rejecting a Repeat

When the key already exists, three things can be true, and each needs a different answer:

async function replayOrReject(pool: Pool, key: string, fp: string) {
  const { rows } = await pool.query(
    `SELECT request_fingerprint, status, response_code, response_body
     FROM idempotency_keys WHERE key = $1`,
    [key]
  );
  const row = rows[0];
 
  // 1. Same key, DIFFERENT body — client bug. Reject loudly, don't replay.
  if (row.request_fingerprint !== fp) {
    return {
      code: 422,
      body: { error: "Idempotency-Key reused with a different request body" },
    };
  }
 
  // 2. Original request is still running. Tell the client to retry later.
  if (row.status === "in_progress") {
    return {
      code: 409,
      body: { error: "A request with this key is still being processed" },
    };
  }
 
  // 3. Completed — replay the stored response verbatim.
  return { code: row.response_code, body: row.response_body };
}

The three branches are the whole design:

• Different body, same key → 422. This is a programming error on the client. Returning the old response would be worse than failing, because it would silently answer a question the client didn't ask. Reject it so the bug surfaces.
• Still in progress → 409. Two retries are racing and the first one hasn't finished. The second one must not start a parallel charge. Tell it to back off and retry — by then the first will have completed and the next attempt replays the stored result.
• Completed → replay. The work is done; hand back exactly what we sent the first time. The client can't tell its retry from the original, which is the entire point.

Running the Work and Storing the Result

The half that won the claim does the work and records the answer:

async function runChargeAndStore(pool: Pool, key: string, fp: string, body: ChargeRequest) {
  const client = await pool.connect();
  try {
    await client.query("BEGIN");
 
    // The actual side effect and the idempotency result share one transaction.
    const charge = await insertCharge(client, body); // your business logic
 
    const responseBody = { id: charge.id, status: "paid" };
 
    await client.query(
      `UPDATE idempotency_keys
       SET status = 'completed', response_code = $2, response_body = $3
       WHERE key = $1`,
      [key, 200, responseBody]
    );
 
    await client.query("COMMIT");
    return { code: 200, body: responseBody };
  } catch (err) {
    await client.query("ROLLBACK");
    throw err;
  } finally {
    client.release();
  }
}

Note what's inside the transaction. That brings us to the part most implementations get subtly wrong.

Atomicity: The Effect and the Record Must Commit Together

Here's the failure window. Suppose you charge the card, then — in a separate transaction — mark the idempotency key completed. If the process dies between those two steps, you've charged the customer but the key still says in_progress. The retry arrives, sees in_progress, and... now you're stuck. Worse, if you'd designed it to re-run on a stale in_progress, you charge twice.

The fix is to make the side effect and the idempotency record part of the same database transaction, as in the code above. The charge row and the status = 'completed' update commit together or not at all. If the process crashes mid-transaction, Postgres rolls both back — no charge, no completed key — and the retry safely starts over.

That works cleanly when the side effect is a database write. The honest complication is when the side effect lives outside your database — calling an external payment processor, for instance. You can't enroll a third-party API call in your Postgres transaction. The standard answer is the transactional outbox pattern: inside the transaction you write an "intent to charge" row; a separate worker reads it, performs the external call, and records the outcome. The external call itself must be idempotent (this is exactly why payment processors offer their own idempotency keys — you pass one through). It's more moving parts, and I won't pretend otherwise — but it's the only honest way to keep an external effect and a local record consistent across a crash.

TTL and Cleanup: Keys Don't Live Forever

Idempotency keys are not permanent. A retry that arrives 24 hours after the original almost certainly isn't a genuine retry — it's a stale client, a replayed log, or something you'd rather treat as a new request anyway. So the keys get a TTL, which is why the table has expires_at.

Pick a window that comfortably covers your real retry behaviour. A client that retries with exponential backoff over a few minutes is well inside a 24-hour window. Stripe honours keys for 24 hours; that's a reasonable default to copy unless you have a specific reason to differ.

Cleanup is a periodic delete:

DELETE FROM idempotency_keys WHERE expires_at < now();

Run it on a schedule — a cron job, a background worker, or pg_cron. Without cleanup, the table grows unbounded; every operation your API ever served leaves a row. On a high-volume endpoint that's millions of rows of dead weight dragging down the very index that makes the claim fast. The idx_idempotency_expires index keeps the delete cheap.

The TTL also quietly solves the stuck-in_progress problem. If a process dies before committing, the transaction rolls back and the row never persisted — fine. But if you ever end up with a genuinely orphaned in_progress row (a bug, a non-transactional path), it will expire and get swept, rather than blocking that key forever. Belt and suspenders.

The Honest Edge Cases (and Where You Don't Need This)

Idempotency keys are not a magic exactly-once wand. Here's what they don't do, and what they cost.

Idempotency stops at your boundary. The key guarantees your handler runs once. It says nothing about downstream side effects you don't control. The canonical example: your handler sends a confirmation email through a third-party provider, and the process crashes after the email goes out but before the transaction commits. The retry re-runs, sends a second email. Your charge is exactly-once; your emails are at-least-once. The mitigation is to push non-transactional side effects out of the critical path — enqueue them via the outbox and make those idempotent too — but you have to think about each one. There's no single switch.

A reused key with a different body is a real trap. I made the handler reject it with a 422, and I'd argue strongly against any alternative. The tempting shortcut — "just return the cached response" — means a client that accidentally reuses a key gets the answer to a question it never asked, with no error to tell it. That's the kind of bug that surfaces three weeks later as "why did this customer get charged for the wrong amount." Reject loudly.

Concurrent retries can deadlock or thrash if you're careless. The ON CONFLICT DO NOTHING claim is the clean approach precisely because it doesn't take a lock you have to manage. If you reach instead for SELECT ... FOR UPDATE or advisory locks, you're now responsible for lock ordering and timeouts, and two retries hammering the same key under load can stack up. The atomic insert sidesteps most of that. The 409-and-retry path handles the rest: the loser of the race doesn't wait, it bounces.

The crash-between-lock-and-commit window is real, not hypothetical. I described the mitigation (single transaction) and the cleanup (TTL sweep), but be clear-eyed: if your side effect is external and your outbox worker has a bug, you can still produce a stuck key or a missed effect. Idempotency narrows the failure window dramatically; it does not erase it. Anyone who tells you their idempotency layer is bulletproof either hasn't run it under real failure injection or isn't being honest.

And the most important caveat: not every endpoint needs this. Idempotency is a cost — a table, a transaction wrapper, replay logic, cleanup. Spend it where a double-execution actually hurts:

• GET requests are already idempotent. Reading data twice is harmless. Don't add keys to reads.
• Naturally idempotent writes don't need it either. PUT /users/42 { "name": "Iurii" } sets the name to the same value no matter how many times it runs. A retry is harmless by construction.
• The endpoints that need it are non-idempotent state changes with real consequences: charges, order creation, sending money, provisioning, anything that increments a counter or fires a one-time side effect.
• A purely internal, low-stakes endpoint — an admin tool that re-runs a report, say — probably isn't worth a key table. Match the machinery to the cost of getting it wrong.

If you find yourself adding idempotency keys to every route reflexively, stop. The discipline is knowing which operations are dangerous when repeated, and protecting exactly those.

The Whole Design in One Glance

Takeaways

1. A timeout tells you nothing. The request may have succeeded and lost its response on the way back. Design as if every timed-out request might have run, because it might have. That single assumption is what makes retries safe instead of dangerous.

2. Exactly-once is an effect you build, not a delivery you buy. The network gives you at-least-once and no transport will change that. The guarantee lives in your handler — a record of what you've done and the discipline to check it first.

3. The atomic claim beats locking. INSERT ... ON CONFLICT DO NOTHING does the dedup-check and the claim in one statement, with no lock to manage and no deadlock to chase. Reach for explicit locks only when this genuinely can't express what you need.

4. Commit the effect and the record together. If the side effect and the idempotency result aren't in the same transaction, there's a crash window where you've done the work but lost the proof. When the effect is external, use an outbox and make the external call idempotent too.

5. Don't idempotency everything. GETs and naturally-idempotent PUTs don't need it. Spend the machinery on the handful of operations where a repeat charges money, ships a box, or sends a thing twice. Knowing which those are is the actual skill.

If you're building an API where a retry can't be allowed to charge twice — payments, ordering, anything that moves money or inventory — getting the idempotency layer right is the difference between a quiet system and a recurring refund queue. It's the kind of correctness work I do on API and integration projects, and if it'd be a real incident for you to get it wrong, it's worth getting right from the start.