A dashboard says last night’s database backup succeeded. That proves a backup job wrote something somewhere. It does not prove that the file contains the expected data, that encryption keys are available, that the database version can read it, or that the application can run after restoration.

A restore test turns backup from hope into evidence. The test creates an isolated database, restores a selected recovery point, verifies integrity and important business invariants, starts a compatible application version, and records how long the process took.

define recovery targets before tooling

Two numbers shape the design:

  • Recovery Point Objective (RPO): how much recent data the system can afford to lose.
  • Recovery Time Objective (RTO): how long service recovery may take.

If the RPO is 15 minutes, one nightly snapshot is insufficient. You may need continuous write-ahead-log archiving or managed point-in-time recovery. If the RTO is one hour but downloading and restoring the backup takes three hours, the dashboard’s green status is misleading.

Choose targets per system. A portfolio blog and a financial ledger have different consequences, dependencies, and budgets.

inventory everything recovery needs

The database is rarely the whole product. Record:

  • Database engine and version.
  • Snapshot or dump location and retention.
  • Encryption-key ownership and recovery process.
  • Schema migration version.
  • Required extensions and roles.
  • Object storage or uploaded files referenced by rows.
  • Search indexes and caches that can be rebuilt.
  • Application version compatible with the recovery point.
  • Secrets and configuration needed to start safely.

Keep infrastructure definitions and restore instructions outside the database being recovered. A runbook stored only in the failed system is not a runbook.

restore into an isolated environment

Never test by overwriting production. Create a temporary network and database with no access to real payment, email, webhook, or analytics endpoints. Restored jobs and outbox rows may try to repeat old side effects if an application starts against them.

For a PostgreSQL logical backup, a drill might begin like this:

createdb restore_drill_20260713
pg_restore \
  --exit-on-error \
  --no-owner \
  --dbname restore_drill_20260713 \
  backup.dump

The exact commands depend on backup type and provider. Physical snapshots, point-in-time recovery, and logical dumps have different version and extension requirements. Use credentials dedicated to the drill and destroy the environment afterward according to data-handling policy.

check more than row counts

A successful command exit is the start of verification. Test structural and business integrity:

SELECT COUNT(*) FROM users;
SELECT COUNT(*) FROM withdrawals;
SELECT MAX(created_at) FROM audit_events;

SELECT id
FROM withdrawals
WHERE status = 'completed'
  AND provider_transfer_id IS NULL;

The last query expresses an invariant: a completed withdrawal should have a provider reference. Similar checks can look for orphaned foreign keys, missing required settings, duplicate operation keys, negative balances where forbidden, or an outbox cursor ahead of available events.

Compare counts and checksums with metrics captured near backup time. Exact equality may not be possible for a point-in-time recovery target, so document the expected cutoff and acceptable differences.

start the application safely

Use the application build and schema version expected by the restored data. Disable outbound side effects, background schedulers, and consumers until their behavior is understood. Then run read-only smoke tests and controlled mutations against synthetic records.

A useful smoke suite checks login, one important object read, a representative transaction, migration compatibility, and health checks. It should fail loudly when required extensions, roles, or configuration are absent.

Do not automatically run the newest migrations before inspecting the restored state. A destructive migration may make an otherwise valid older backup impossible to diagnose. Record the restored schema version first, then test the documented upgrade path separately.

test point-in-time recovery with a marker

To verify an RPO, write a harmless marker at a known timestamp, make a second change later, and restore to a time between them. The marker should exist and the later change should not.

INSERT INTO recovery_markers (id, created_at)
VALUES ('drill-2026-07-13-a', now());

This proves more than selecting the newest available timestamp in a provider interface. Record the requested recovery time, actual latest recovered transaction, and observed data gap.

measure the complete timeline

RTO includes detection, decision, access, provisioning, download, restore, verification, application start, DNS or traffic changes, and communication. Measure each stage rather than reporting only database import duration.

Capture:

Evidence Why it matters
Backup ID and creation time Identifies what was tested
Restore start and finish Measures technical duration
Latest recovered record time Measures observed RPO
Integrity and smoke-test results Proves usability
Manual steps and owners Reveals automation gaps
Errors and workarounds Creates the next improvement

Store the report where the engineering team can find it during an incident.

rehearse failure inside the recovery process

Try an expired credential, missing encryption permission, incompatible database version, insufficient disk space, corrupted dump copy, absent extension, and unavailable team member. A drill that assumes every dependency works does not test the incident path.

At least two people should be able to perform recovery. Use a second reviewer for destructive or traffic-changing steps. Keep break-glass access controlled, audited, and tested without exposing credentials in the runbook.

choose a realistic cadence

Run automated restore checks frequently for critical systems and a fuller human drill on a slower cadence, such as monthly or quarterly depending on risk. Test different recovery points, not the same known-good snapshot forever. Include a restore after major database-version, encryption, provider, or migration changes.

Alert when backups stop, shrink unexpectedly, cannot be decrypted, exceed the RPO, or fail an automated restore. Retention should protect against accidental deletion and account compromise, using separate credentials or immutability controls where appropriate.

The honest status is not “backups enabled.” It is “this named recovery point was restored on this date, these invariants passed, the observed RPO was X, and recovery took Y.” Until that evidence exists, the recovery plan remains an assumption.