Disaster Recovery Runbook

Project	PTX-CM
Last Updated	2026-02-17

---

Quick Reference

Scenario	Time to recover	Procedure
API crash	~1 min	Automatic (Docker restart + replicas)
Host reboot	~2-3 min	Automatic (Docker restart policies)
Bad migration	~15 min	Restore from backup
Disk failure	~30-60 min	Full server rebuild
Data corruption	~15 min	Restore from backup

---

Prerequisites

Before disaster strikes, ensure:

• [ ] scripts/backup.sh runs daily via cron
• [ ] Backups are copied offsite (S3/GDrive/rclone)
• [ ] Better Stack heartbeat confirms backups are running
• [ ] .env file is stored in a secure vault (1Password, Bitwarden, etc.)
• [ ] This document is accessible outside the server (printed/cloud copy)

---

Restore from Backup

Use when: bad migration, accidental data deletion, data corruption.

Step 1: Find the latest backup

ls -lht backups/*.sql.gz | head -5

If local backups are gone, download from offsite:

# rclone copy remote:ptx-cm-backups ./backups --max-age 7d
# aws s3 sync s3://ptx-cm-backups/ ./backups/

Step 2: Restore

./scripts/restore.sh backups/ptx_cm_YYYYMMDD_HHMMSS.sql.gz

The script will:

1. Stop API containers (prevent writes)
2. Drop and recreate the database
3. Restore from the backup file
4. Restart API containers

Step 3: Verify

# Check API health
curl http://localhost/health

# Check readiness (DB + Redis)
curl http://localhost/health/ready

# Check containers
docker compose ps

---

Full Server Rebuild

Use when: server hardware failure, need to migrate to new server.

Step 1: Provision new server

• Ubuntu 22.04+ LTS (or your preferred Linux)
• Docker Engine + Docker Compose v2
• Git
• Minimum: 2 CPU, 4 GB RAM, 40 GB SSD
• Open ports: 80 (HTTP), 443 (HTTPS)
• SSH access configured

# Install Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER

# Install Docker Compose plugin
sudo apt install docker-compose-plugin

Step 2: Clone repository

git clone <your-repo-url> /opt/ptx-cm
cd /opt/ptx-cm

Step 3: Restore secrets

Retrieve .env from your secure vault and place it in the project root:

# Required secrets:
# DATABASE_URL, JWT_SECRET, JWT_REFRESH_SECRET,
# ENCRYPTION_KEY, REDIS_PASSWORD, SMTP credentials
cp /path/to/vault/.env .env

Step 4: Restore database backup

# Start only postgres first
docker compose up -d postgres
# Wait for healthy
docker compose exec postgres pg_isready -U ptx_cm

# Download latest backup from offsite
# rclone copy remote:ptx-cm-backups ./backups --max-age 1d

# Restore
./scripts/restore.sh backups/ptx_cm_LATEST.sql.gz

Step 5: Start all services

docker compose up -d

Step 6: Verify

# All containers running?
docker compose ps

# API healthy?
curl http://localhost/health

# DB + Redis connected?
curl http://localhost/health/ready

# Web accessible?
curl -I http://localhost/

Step 7: DNS / Proxy

Update DNS or load balancer to point to the new server's IP.

Step 8: Restore cron

# Add backup cron
crontab -e
# 0 2 * * * /opt/ptx-cm/scripts/backup.sh >> /var/log/ptx-cm-backup.log 2>&1

---

Secrets to Recover

Secret	Where to find it
.env file	Secure vault (1Password/Bitwarden)
JWT_SECRET	In .env
JWT_REFRESH_SECRET	In .env
ENCRYPTION_KEY	In .env — critical, decrypts OTA credentials
REDIS_PASSWORD	In .env
SMTP credentials	In .env
Better Stack heartbeat URL	Better Stack dashboard
Git repo credentials	Git hosting platform

⚠️ CRITICAL: If ENCRYPTION_KEY is lost, all encrypted OTA credentials become unrecoverable. Ensure this is backed up in your vault.

---

Post-Recovery Checklist

• [ ] All containers healthy (docker compose ps)
• [ ] /health returns 200
• [ ] /health/ready returns 200 (DB + Redis OK)
• [ ] Login works in browser
• [ ] Dashboard loads with data
• [ ] Booking sync is running (check sync jobs page)
• [ ] Better Stack monitors are green
• [ ] Backup cron is scheduled
• [ ] Better Stack heartbeat is receiving pings