The PostgreSQL administration field guideField notes · Runbooks · Free certification
pgdba.dev
Menu

Operational runbook · Quick reference

Backup or restore tests are failing

Protect the recovery chain, locate the failing stage, and prove recovery objectives with a fresh rehearsal.

backup failedrestore failedarchive gapRTO missed

Before you begin

Prerequisites

  • Confirm the target cluster, database, PostgreSQL major version, and your read-only access to the required statistics views.
  • Record the current user impact, incident owner, UTC timestamp, and a baseline before running diagnostics.

Control the blast radius

Safety boundary

Start with read-only observations. Do not restart, terminate, delete, fail over, or change configuration until ownership, blast radius, and an approved recovery path are explicit.

Triage

Establish impact

  1. Record the affected recovery or production objective, scope, start time, and current user impact.
  2. Preserve logs, identifiers, timestamps, and before/after evidence before retrying or restarting work.
  3. Identify the owning application, change, job, or infrastructure boundary and its rollback constraints.

Observe

Gather evidence

Diagnostic query

Record recovery and WAL context

Capture whether this server is primary or recovering and identify its current WAL boundary.

SELECT
  pg_is_in_recovery() AS is_in_recovery,
  CASE
    WHEN NOT pg_is_in_recovery() THEN pg_current_wal_lsn()
    ELSE NULL
  END AS primary_wal_lsn,
  pg_last_wal_receive_lsn() AS last_received_lsn,
  pg_last_wal_replay_lsn() AS last_replayed_lsn,
  pg_last_xact_replay_timestamp() AS last_replay_timestamp;
low observation cost

Interpret

Interpret the evidence

  • NULL values can be expected on a primary; interpret columns from the server role.
  • A base backup without the required WAL, keys, roles, extensions, and validation does not prove recovery.

Act

Take the safest useful action

  1. Preserve the last known-good backup and WAL chain before retrying or changing retention.
  2. Repair the proven permission, storage, archive, dependency, or capacity boundary.
  3. Run a fresh isolated restore, time every stage, and validate application and data invariants.

Prove the outcome

Verification

  1. Repeat the baseline observation over a known interval and confirm that the measured queue or risk is moving in the intended direction.
  2. Verify the user-facing objective and every affected replica or dependency before closing or handing over the incident.

Keep recovery close

Rollback

Record the original state and rollback owner before acting. If verification worsens or the objective is missed, reverse only the bounded change, confirm the baseline is restored, and escalate with the before-and-after evidence.

Escalate

Escalate when

  • Escalate when required WAL or encryption material is missing, the last good recovery point breaches policy, or the rehearsal misses RPO or RTO.

Prevent recurrence

Study the underlying system

replication postgres health

Certification competency

Refresh the assessed operating model

Backup strategy and validation Restore and point-in-time recovery Safe changes, upgrades, and handover

Reference assurance

Version and review

Compatible versions
PostgreSQL 16–18
Content version
2026.08
Reviewed
2026-08-04

Catalog columns, wait events, and operational controls can vary by PostgreSQL major version, extensions, and orchestration layer. Verify commands against the deployed version.

Verify independently

Sources