Disaster recovery engineering

What evidence supports a five-minute recovery point objective?

The age of the latest complete, accessible, recoverable data point under the specified failure scenario. Backup scheduling frequency alone does not prove the objective.

When measuring recovery time, why include detection and decision time?

Users remain affected before restoration commands begin. Define the measurement boundary and include activation, provisioning, restoration, validation, and traffic recovery.

What does PostgreSQL point-in-time recovery need besides a base backup?

The continuous required WAL sequence through the target point, compatible recovery tooling, and correct timeline information. One missing required segment can prevent reaching the target.

Can a PostgreSQL logical dump be used as the base for physical WAL replay?

No. Logical dumps and physical base backups are different recovery mechanisms. Select a supported chain and test the actual restore procedure.

Why restore a suspected corruption incident to an isolated environment first?

It allows checking the selected recovery point and data integrity without overwriting the live system or exposing inconsistent data. Preserve evidence and account for valid later changes.

An encrypted backup is intact but its key service is unavailable. What does this reveal?

Recoverability depends on credentials, keys, permissions, and their failure domains. Include those dependencies in backup and disaster exercises.

Why can a standby region fail to start despite replicated data?

Identity, secrets, image registries, DNS, quotas, or control-plane services may still depend on the failed region. Inventory the complete bootstrap path.

Why must the old primary be fenced before accepting writes on a replacement?

A partitioned old primary may still serve clients. Prevent concurrent writers through an authoritative ownership or fencing mechanism rather than assuming the old process is dead.

What must operators establish before promoting an asynchronously replicated standby?

The last durable replicated position and the possible acknowledged writes absent from it. Promotion may trade data loss for availability; reconcile ambiguous operations afterward.

Why can changing DNS leave some clients connected to the failed destination?

Resolver caches and existing connections outlive the change. Test client reconnection, connection lifetime, TTL behavior, and alternate traffic controls.

Why can a small successful restore test underestimate production RTO?

Full data volume, log replay, index work, bandwidth, and validation may scale poorly. Measure representative size and contention rather than extrapolating from a tiny sample alone.

What should be checked after a database starts successfully from backup?

Expected records and transaction boundaries, application invariants, permissions, dependencies, and real read/write workflows. Process startup establishes only a narrow part of recovery.

What risk remains when production administrators can delete every backup with the same credentials?

A compromised or mistaken production identity can destroy both live data and recovery copies. Separate authority and use appropriate retention controls for the threat model.

Why should backup retention account for how long corruption can go undetected?

If every retained backup already contains the defect, backups cannot recover a clean state. Choose retention and integrity checks based on detection delay and repair needs.

Why restore application configuration and infrastructure definitions alongside data?

A usable service requires compatible schemas, binaries, routing, credentials, and policies. Data without the configuration needed to interpret and serve it may not be operationally recoverable.

Why can replaying a recovery log accidentally repeat external actions?

Emails, payments, and webhooks may already have completed outside the restored database state. Preserve effect identities and reconcile with external systems before reissuing actions.

Why is moving traffic back to the original region a new migration rather than undoing failover?

New writes accumulated in the recovery region. Reestablish replication, verify convergence, fence writers appropriately, and test capacity before switching authority again.

What makes a disaster-recovery exercise realistic enough to reveal hidden dependencies?

Disable the assumed failed components, use actual recovery access and artifacts, include unfamiliar operators, and measure validation and traffic restoration. A tabletop alone cannot prove restore throughput.

When is a finding from a failed restore exercise actually resolved?

After the fix is implemented and the relevant restore step is rerun successfully with recorded evidence. Updating the runbook without exercising it leaves the guarantee unverified.