Incident Response Runbook for a Live Crypto Trading Bot Outage

A bot going dark mid-session is not a hypothetical — it is a scheduled event you have not prepared for yet. The difference between a 4-minute recovery and a 40-minute recovery is almost never technical skill; it is whether your team has a documented decision tree they can execute half-asleep. What follows is the runbook format we actually use when a production bot stops filling orders.

Immediate Triage: The First 90 Seconds

Before you touch any code, kill the noise. Open your alerting dashboard and answer four questions in order:

1. Is the process alive? Check your process supervisor (PM2, systemd, or a Kubernetes pod) for exit codes. An exit code 1 is usually an unhandled exception; code 137 is OOM; code 0 means the process exited cleanly on purpose.
2. Is the exchange reachable? Ping the REST health endpoint. On Hyperliquid this is GET /info with a {"type":"meta"} body. On Solana RPC, check slot lag — if your provider is more than 5 slots behind the cluster head, you have an RPC issue, not a bot issue.
3. Are positions open? Query the exchange directly, not your internal state. If you have naked delta and the bot is dead, that is your critical path. Everything else is secondary.
4. What changed in the last deploy? Run git log --oneline -10. If anything shipped in the last 90 minutes, that is your prime suspect until proven otherwise.

Do not restart anything yet.

Classifying the Incident

Severity drives the response. A three-tier classification keeps everyone aligned without a war room call:

• P1 — Open exposure with no hedge: Bot is down, positions are live, no kill switch has fired. Immediate manual intervention. Someone gets on the exchange UI and closes or hedges within 5 minutes, no committee needed.
• P2 — Bot is down, flat book: Revenue loss from missed fills, but no market risk. You have 15–30 minutes to diagnose before restarting.
• P3 — Degraded performance: Bot is running but fill rate or latency metrics are outside normal bounds. Investigate during business hours.

The most common mistake is treating a P2 like a P3 because the book looks flat. Flat now does not mean flat in 30 seconds on a volatile asset.

Rollback Procedure

If the last deploy is the culprit, the rollback sequence matters. On a containerized deployment:

# 1. Tag the broken image for post-mortem
docker tag mybot:latest mybot:broken-$(date +%Y%m%d%H%M)

# 2. Pull the last known-good image
docker pull mybot:stable

# 3. Restart with the previous config — not current env vars
docker run --env-file .env.stable mybot:stable

Never restart with the same config that caused the crash. If the bot died on a bad config value, restarting it will reproduce the crash. Keep a .env.stable pinned to the last production-verified configuration and rotate it deliberately, not in the same commit as code changes.

On Solana, watch for state desync after a rollback. If your bot tracks order book state in memory across websocket messages, a restart drops that state. On DEXes with on-chain order books like Phoenix, re-fetch the full slab before resuming. Skipping this step causes phantom fills and incorrect inventory tracking.

Exchange Escalation

For CEX integrations (Hyperliquid, dYdX), know your escalation path before you need it. Hyperliquid support is reachable via their Discord #api-support channel; response times during Asian trading hours are typically under 20 minutes for verified API partners. For issues that look like exchange-side (orders accepted but not matching, anomalous latency on the matching engine), pull your WebSocket message logs and attach timestamps before opening a ticket — support cannot act on "my bot stopped working."

For Solana on-chain issues, the escalation path is your RPC provider first, not the Solana Foundation. Providers like Helius and Triton maintain status pages and have direct support channels. If you are seeing consistent transaction simulation failures that did not exist an hour ago, check whether a recent validator client update shipped — slot timing behavior can shift across minor versions.

Keep escalation contact info in the runbook itself, not in someone's head. People leave.

Post-Mortem Template

Run a post-mortem within 48 hours, while the details are fresh. Five sections, no blame:

1. Timeline — UTC timestamps from first alert to full recovery. Be precise: "approximately 3 AM" is useless.
2. Root cause — One specific, falsifiable statement. "Config value MAX_POSITION_SIZE was set to 0 after a merge conflict resolution" not "there was a configuration issue."
3. Contributing factors — What made this possible? Missing config validation, no integration test for that code path, deploy happened during high-volatility window.
4. Impact — Missed fills quantified, open exposure duration, estimated PnL impact if calculable.
5. Action items — Each item has an owner and a due date. "Add config schema validation before process start" assigned to a specific person by a specific date.

A post-mortem with no action items is a story, not a process improvement.

Monitoring Baselines That Make Triage Faster

The fastest incident response starts before the incident. Three metrics that cut triage time significantly:

• Fill rate by symbol, 1-minute buckets — a sudden drop to zero is unambiguous. Latency spikes are subtle; zero fills are not.
• Heartbeat latency to exchange — track round-trip time on your keepalive pings. On Hyperliquid WebSocket, expect under 15ms from a co-located instance. If you see 200ms, your network path has changed.
• Position state checksum — hash your internal inventory state and compare it against the exchange's reported positions every 60 seconds. Divergence is a silent killer that compounds over hours.

Alert on deviation, not just on zero. A fill rate that drops from 85% to 40% is already an incident, even if the bot has not crashed.

---

If your current stack lacks any of this — monitoring, rollback paths, or escalation procedures — talk to the TierZero team. We build and operate these systems in production and can close the gaps before the next 3 AM wakeup call.