Building RPC Failover Architecture for Solana Trading Bots

RPC reliability is the unglamorous bottleneck that kills otherwise well-designed Solana trading bots. A slot-confirmation timeout at the wrong moment — mid-arb, mid-liquidation — is not recoverable. The market moves on, your position does not. What follows is the architecture we run in production to eliminate single-provider dependency without introducing the coordination overhead that kills latency.

Why Active-Passive Over Round-Robin

The temptation is to load-balance across providers in round-robin. Resist it. Round-robin distributes requests across all backends equally, which means when one provider degrades — not fails outright, but starts returning stale slots or high-latency responses — you spray a percentage of your traffic into a slow path on every cycle. For a bot sending getLatestBlockhash and sendTransaction calls in tight loops, that averages your worst-case latency into your baseline.

Active-passive keeps a single primary in the hot path at all times. A secondary (or tertiary) sits warm and idle, ready to absorb traffic within seconds of a failover trigger. Your p99 latency stays pinned to your best provider until that provider genuinely fails, at which point you accept a brief transition window rather than a continuous tax.

HAProxy as the Traffic Layer

HAProxy handles layer-4/layer-7 proxying well and has a mature health-check subsystem. A minimal haproxy.cfg for an RPC pool looks like this:

frontend solana_rpc
    bind *:8545
    default_backend rpc_pool

backend rpc_pool
    balance first
    option httpchk GET /health
    http-check expect status 200
    server primary   helius-mainnet.endpoint:443 ssl verify none check inter 1000 rise 2 fall 3
    server secondary triton-mainnet.endpoint:443 ssl verify none check inter 1000 rise 2 fall 3 backup
    server tertiary  self-hosted-rpc:8899          check inter 1000 rise 2 fall 3 backup

Key parameters: balance first routes all traffic to the first available healthy server rather than distributing it. inter 1000 runs health checks every 1,000 ms. fall 3 marks a server down after three consecutive failures — roughly 3 seconds of failure tolerance before failover triggers. rise 2 requires two consecutive successes to restore a server to active rotation, preventing flapping.

The Health-Check Sidecar

HAProxy's built-in httpchk checks whether the endpoint is reachable, not whether the Solana node behind it is healthy. A provider's HTTP endpoint can return 200 while the underlying validator is lagging 50 slots behind the tip. That gap is enough to invalidate blockhashes and cause transaction expiry under load.

We run a lightweight sidecar — a small Go binary, under 200 lines — that exposes /health on a local port and performs three checks on each probe cycle:

1. Slot lag check: calls getSlot with commitment: "confirmed" and compares the result against a reference slot fetched from a second provider. If the delta exceeds 8 slots, the check fails.
2. Blockhash freshness: calls getLatestBlockhash and verifies the lastValidBlockHeight is within expected bounds of the current slot.
3. Response latency: measures wall-clock time for each call. If either call exceeds 400 ms, the check returns 503 regardless of the data.

The sidecar runs one instance per provider. HAProxy checks each sidecar, not the provider directly. This means your failover logic is decoupled from raw TCP reachability and actually reflects the quality of data your bot will receive.

Circuit Breakers at the Application Layer

HAProxy failover handles sustained outages. It does not handle the case where a provider is healthy by all check metrics but starts returning errors on a specific RPC method — simulateTransaction latency spikes, for instance, while getSlot stays fast. Application-level circuit breakers close that gap.

Implement a per-method circuit breaker in your bot's RPC client wrapper. Track a rolling error rate over a 10-second window. If sendTransaction error rate exceeds 15%, open the circuit for that method and re-route those calls to the secondary provider directly, bypassing HAProxy. After 30 seconds, enter half-open state: allow one call through and close or stay open based on the result.

This is more granular than what a proxy layer can express and catches the degraded-but-not-down failure mode that causes the most strategy damage in practice.

Blockhash Management Under Failover

Blockhashes are valid for approximately 150 slots — around 60 seconds at current Solana block times. During a failover, if your bot has pre-fetched a blockhash from the primary provider, that hash remains valid and your in-flight transactions are fine. The risk is in the window where you fetch a new hash from a secondary that is lagging.

Cache your most recent valid blockhash with a timestamp. On failover, continue using the cached hash for up to 30 seconds while the secondary warms up and your sidecar confirms slot lag is within bounds. After 30 seconds, fetch fresh from the secondary. This prevents the race condition where a freshly failed-over secondary returns a stale hash that causes your next batch of transactions to expire.

Observability You Actually Need

Log three things per RPC call: provider name, method, and wall-clock latency. Nothing else is useful at call granularity — you will generate too much noise. Aggregate into per-provider, per-method p50/p95/p99 with a 60-second rolling window and expose them as Prometheus metrics. Alert on p95 sendTransaction latency exceeding 600 ms sustained for more than 90 seconds, and on failover events. Everything else is dashboard decoration.

Self-hosted RPC — even a single validator on bare metal with 32 GB RAM and NVMe — changes the economics of this stack considerably. It eliminates per-call pricing on high-throughput strategies and gives you a low-latency anchor for your slot-lag checks. The operational overhead is real, but for strategies doing more than 50,000 RPC calls per day it becomes the right trade.

---

If you want a production-ready failover stack deployed alongside your strategy rather than built from scratch, talk to us — infrastructure like this is a standard part of what we ship.