The scary backend bug is not a worker crashing. A crash is visible. The scarier bug is a retry that looks successful while repeating the side effect: a duplicate charge, a second withdrawal transfer, or two ledger rows for one event.

While building CryptoEx, the invariant I wanted was simple to say and annoyingly difficult to guarantee: one accepted withdrawal request must create at most one provider transfer, even if the queue redelivers the job or the process dies halfway through it. That is the problem idempotency solves.

the invariant before the implementation

Idempotency means one logical operation can run more than once but only produce one final side effect.

Example:

Event Unsafe result Idempotent result
Worker charges user, then crashes before saving success Retry charges user again Retry sees the same idempotency key and returns the first result
Payment webhook arrives twice Two ledger rows One ledger row tied to one event ID
Email job retries after timeout Customer gets duplicate emails Job checks whether the message ID was already sent

The key is not “never retry.” The key is “retry safely.”

retries are not optional

Any real worker system eventually retries jobs.

The process restarts. Redis disconnects. A payment API times out. A webhook arrives twice. A queue library decides the job failed and puts it back into the waiting set.

For harmless jobs, this is fine. Recalculating a cache twice is annoying, not dangerous.

For money, inventory, user credits, or email, retries need a contract.

define one logical operation

An idempotency key is a stable identifier for one logical operation.

For an invoice payment, that key might be:

payment:user_123:invoice_456

That key should not change between retries. If the same invoice payment is attempted five times, all five attempts use the same key.

This lets your worker ask a simple question:

Has this exact operation already been claimed or completed?

the naive version has a race condition

This looks reasonable at first:

const key = `payment:${userId}:${invoiceId}`;

const existing = await redis.get(key);

if (existing) {
  return;
}

await chargeUser(userId, amount);
await redis.set(key, "done");

The bug is the gap between get and set.

Two workers can run at the same time:

  1. Worker A checks Redis. No key.
  2. Worker B checks Redis. No key.
  3. Worker A charges the user.
  4. Worker B charges the user.
  5. Both write "done".

The final Redis state looks clean, but the user was charged twice.

claim the work atomically

The check and claim need to happen in one operation.

Redis can do that with SET plus NX:

const claimed = await redis.set(key, "processing", {
  NX: true,
  EX: 60 * 10
});

if (!claimed) {
  return;
}

NX means set the key only if it does not exist. EX gives the claim a timeout so a dead worker does not hold the lock forever.

Only one worker wins that claim.

keep a durable database constraint too

Redis is useful for a fast claim, but I would not make it the only record protecting a money movement. Keys expire, caches can be flushed, and an operator may need to reconstruct an operation long after the queue job disappears.

Give the durable operation its own unique key in the database:

CREATE UNIQUE INDEX withdrawals_idempotency_key_unique
ON withdrawals (idempotency_key);

Now two workers cannot create two withdrawal operations with the same logical identity, even if the Redis claim is missing. The application should handle the unique-conflict path by loading and returning the existing operation.

This is deliberate overlap. Redis reduces concurrent work. The database protects durable state. The payment provider’s idempotency support protects the external transfer. Each boundary answers the duplicate question for the state it owns.

store useful state, not just done

A string like "done" is better than nothing, but it is not very useful when debugging.

Store a small result:

{
  "status": "completed",
  "chargeId": "ch_123",
  "invoiceId": "invoice_456"
}

Now a retry can return the previous result, and an operator can understand what happened without reading logs from three services.

I like thinking in these states:

processing
completed
failed

Do not overbuild the state machine too early. Just make sure the state answers the question you will have during an incident: did the side effect happen?

the payment provider needs the same key

Your Redis key protects your worker from duplicate local execution. It does not automatically protect you from uncertainty at the payment provider.

This is the awkward case:

  1. Worker sends charge request.
  2. Provider charges successfully.
  3. Network drops before your worker receives the response.
  4. Worker retries.

If the provider supports idempotency keys, send the same key there too.

await paymentProvider.charge({
  userId,
  amount,
  idempotencyKey: key
});

That lets the provider return the original charge instead of creating a second one.

a safer worker shape

This is still simplified, but it has the right bones:

async function processPaymentJob(job: PaymentJob) {
  const key = `payment:${job.userId}:${job.invoiceId}`;

  const claimed = await redis.set(key, "processing", {
    NX: true,
    EX: 60 * 10
  });

  if (!claimed) {
    return;
  }

  try {
    const charge = await paymentProvider.charge({
      userId: job.userId,
      amount: job.amount,
      idempotencyKey: key
    });

    await redis.set(key, JSON.stringify({
      status: "completed",
      chargeId: charge.id,
      invoiceId: job.invoiceId
    }));
  } catch (error) {
    await redis.set(key, JSON.stringify({
      status: "failed",
      invoiceId: job.invoiceId
    }), { EX: 60 * 30 });

    throw error;
  }
}

The important parts are:

  • The operation has a stable key.
  • The worker claims the key atomically.
  • The external API receives the same idempotency key.
  • The completed state stores a useful result.
  • Failures are visible instead of disappearing.

when to delete a failed key

There is no universal answer.

If the failure happened before any side effect, deleting the key may allow a clean retry.

If the failure happened after a side effect might have happened, deleting the key can be dangerous. You may need reconciliation instead: ask the provider what happened, compare records, then decide whether to retry.

This is where payment systems stop being a code snippet and become operations work.

where this pattern applies

Use idempotency for any retried job with side effects:

  • Payment charges.
  • Webhook handling.
  • Email sends.
  • Reward credits.
  • Ledger writes.
  • Subscription renewals.
  • File processing that updates permanent records.

If a duplicate would confuse a user or corrupt business data, design the retry path before you ship.

what I would do differently next time

I would design idempotency keys before writing the worker handler. It is much harder to add the key later after jobs, logs, retries, and database rows already exist.

The practical rule is simple: queues make retries easy, but retries make duplicates possible. Idempotency is how you get the reliability without the duplicate side effects.