A withdrawal worker receives a job, calls a provider, and gets an invalid-account response. Retrying every minute will not repair the account number. Dropping the job loses the customer’s operation. Leaving it in the main queue forever burns capacity and hides a business failure inside infrastructure noise.

A dead-letter workflow gives exhausted jobs a durable, inspectable state. It is not a second queue where failures go to be forgotten. It is a recovery system with enough evidence to decide whether to retry, repair, compensate, or close the operation.

classify failures before setting retry counts

Retries help transient failures: connection resets, temporary 503 responses, lock contention, or a dependency timeout before any side effect occurred. They do not help invalid input, revoked credentials, missing required data, or code that cannot parse a new event version.

Represent that distinction explicitly:

class PermanentJobError extends Error {}
class RetryableJobError extends Error {
  constructor(message: string, readonly retryAfterMs?: number) {
    super(message);
  }
}

The worker can discard or dead-letter permanent errors immediately while retrying temporary errors with exponential backoff and jitter. Jitter prevents thousands of jobs from returning at the same millisecond after an outage.

function retryDelay(attempt: number): number {
  const base = Math.min(60_000, 1_000 * 2 ** (attempt - 1));
  return Math.round(base * (0.8 + Math.random() * 0.4));
}

Set a maximum elapsed retry period, not only an attempt count. Five retries over ten seconds and five retries over two days have different operational meaning.

store a useful failure envelope

The original payload is not enough. A recovery record should answer what failed, where, how often, and what may already have happened:

type DeadLetterRecord = {
  jobId: string;
  queue: string;
  jobType: string;
  payloadVersion: number;
  payload: unknown;
  attempts: number;
  firstAttemptAt: string;
  failedAt: string;
  errorCode: string;
  errorMessage: string;
  stack?: string;
  workerVersion: string;
  correlationId: string;
  idempotencyKey?: string;
  sideEffectState: "not_started" | "confirmed" | "unknown";
};

Redact passwords, tokens, payment credentials, and unnecessary personal data before storage. A dead-letter store often has wider operational access and longer retention than the original request path.

preserve business state outside the queue

In CryptoEx-style withdrawal design, the queue is not the source of truth. The database owns a withdrawal record with states such as pending, processing, requires_review, completed, and failed. Moving a message to a dead-letter queue should update the operation to a visible recoverable state.

That prevents a dangerous mismatch where the queue says “failed” while the user-facing record remains “processing” forever. Use a stable job or operation ID to connect both records.

UPDATE withdrawals
SET status = 'requires_review',
    failure_code = $2,
    updated_at = now()
WHERE id = $1
  AND status = 'processing';

The guarded transition matters. An old worker should not overwrite a withdrawal that reconciliation has already marked complete.

make replay a deliberate command

A “retry all” button is an incident multiplier. Replaying should require a reason, current job version, and a destination environment. Keep the original dead-letter record immutable and create a new attempt linked to it.

Before replay, an operator or repair job should answer:

  1. Is the root cause fixed?
  2. Is the payload still valid under the current schema?
  3. Did the external side effect possibly happen?
  4. Is the handler idempotent with the original operation key?
  5. How many similar jobs will this release?

For an unknown provider timeout, do not blindly replay a transfer. Reconcile with the provider using the stable operation key first. If the transfer exists, record the result; if it definitely does not, then create a controlled retry.

use bounded batches and pause controls

Release repaired jobs in small batches and watch success rate, latency, downstream errors, and new dead letters. A replay worker should support pause, rate limits, and a maximum concurrency lower than the normal queue if the downstream service is recovering.

Keep poison messages isolated. If one event crashes the deserializer before your normal error wrapper, the consumer must still capture its raw bytes or broker metadata and move past it. Otherwise one malformed message can block an entire ordered partition.

distinguish a retry queue from a dead-letter store

A delayed retry queue contains work expected to succeed later. A dead-letter store contains work that exhausted normal automation or was classified as unsafe to retry. Mixing them makes alerts and ownership unclear.

State Expected owner Next action
Scheduled retry Worker system Run automatically after delay
Dead letter: bad payload Producing team Fix producer or migration
Dead letter: dependency rejected Integration owner Repair credentials or request
Dead letter: side effect unknown Operator Reconcile before retry
Dead letter: code bug Service team Deploy fix, canary replay

alert on rate and age

One failed marketing email and one failed withdrawal are not equal. Alert by job class and business impact. Useful signals include dead-letter arrival rate, oldest unresolved age, repeated error codes, retry exhaustion rate, and the ratio of dead letters to completed jobs.

Create a runbook linked from the alert. It should identify the queue owner, dashboards, replay command, redaction rules, and escalation path. If nobody owns the dead-letter backlog, it is data loss with a nicer name.

test the uncomfortable transitions

Test a permanent validation failure, repeated dependency timeouts, process death after a side effect, duplicate delivery, malformed payload, unknown event version, dead-letter-store outage, replay with a changed schema, and two operators replaying the same item. Verify that the original business operation ends in one explainable state.

The operating rule is to retry only what time can repair. Everything else needs evidence, ownership, and a controlled path back into the system.