Building a Multi-Provider RPC Failover Layer for Solana Bots

Building a multi-provider RPC failover layer is one of the first things we do before any Solana bot goes to mainnet. A single endpoint is a single point of failure, and RPC providers go dark at the worst possible moments — high-volatility opens, launch windows, or exactly when you're mid-position. This post walks through the architecture we ship in production: a weighted round-robin proxy in Rust with active health checks, latency-aware routing, and circuit breakers calibrated for HFT tolerances.

Why You Cannot Rely on a Single Endpoint

Most Solana providers — QuickNode, Helius, Triton, Alchemy, your own private validator node — have SLAs in the 99.9% range. That sounds fine until you do the math: 99.9% uptime is roughly 44 minutes of downtime per month. On a launch sniper or a market-making bot, 44 minutes of downtime over a month is not acceptable — most of the losses will be concentrated in the few minutes when your bot is blind and the market is moving.

Provider-side degradation is subtler than full outages. You'll see endpoints start returning stale slot numbers, sendTransaction RPC calls taking 800ms instead of 80ms, or getAccountInfo returning null for accounts that clearly exist. These are not errors your bot will catch with a simple connection check — they require active measurement.

Proxy Architecture

The proxy is a Tokio async server that sits between your bot and the cluster. Your bot sends every JSON-RPC call to localhost:8899 and the proxy handles provider selection, failover, and metrics. This is cleaner than embedding provider-switching logic in the bot itself — the bot code stays simple and you can swap providers without a redeploy.

Bot → local proxy (Rust/Tokio) → [provider A, provider B, provider C]

Each provider entry has:

• weight — integer from 1–10, controls how much traffic it gets during normal operation
• latency_p99 — rolling 99th-percentile latency across the last 1,000 requests, updated every second
• slot_lag — difference between this provider's reported slot and the highest slot seen across all providers
• circuit_state — Closed, Open, or HalfOpen

Routing uses weighted random selection over providers with a Closed circuit. The weights are adjusted dynamically: if a provider's latency_p99 drifts more than 2x above the fleet median, its effective weight is halved. This isn't a hard cutover — it just reduces traffic to the degraded node while keeping it warm.

Health-Check Intervals

The health checker runs on a separate Tokio task, firing getSlot every 200ms per provider. This is where most guides get it wrong — they use 5-second or 10-second intervals that are completely inappropriate for a bot reacting to sub-second events.

With 200ms health checks:

• You detect a provider going stale within one slot (Solana produces a slot every ~400ms)
• You know within one check cycle if slot lag is growing
• The overhead is negligible — getSlot is a cheap call and 5 requests/second per provider costs nothing in quota terms

The circuit breaker thresholds we use in production:

Metric	Warn	Open circuit
latency_p99	> 150ms	> 400ms
Slot lag	> 5 slots	> 15 slots
Error rate	> 2%	> 10%

When a circuit opens, the provider is removed from the rotation immediately. The HalfOpen probe fires a single getSlot after 30 seconds; if the response is healthy (latency < 150ms, slot lag < 3), the circuit closes and the provider re-enters the pool with weight 1, ramping back to full weight over the next 60 seconds.

Handling sendTransaction Differently

Not all RPC methods need the same failover logic. getAccountInfo, getMultipleAccounts, and getSlot are reads — stale or slow is bad but retriable. sendTransaction is different. Submitting the same transaction to multiple providers is not just acceptable, it is the right move — Solana deduplicates at the validator level using the recent blockhash, so a duplicate submission will not double-execute.

For sendTransaction we fanout to the top two providers by weight simultaneously and take the first success. This reduces the tail latency on transaction landing significantly. In our benchmarks, dual-fanout cut p99 landing time from ~320ms to ~190ms on the same set of providers, because you're not serializing on a single provider's queue.

Blockhash Freshness

One subtle issue: if your primary and fallback providers report different slot heights, you might be fetching a blockhash from a slot that some providers haven't seen yet. We solve this by maintaining a shared latest_blockhash cache that only updates when at least two-thirds of active providers agree on the same slot range (within 2 slots of each other). If consensus breaks — e.g., one provider surges ahead or falls behind — the cache holds the last agreed value rather than trusting the outlier.

Blockhashes are valid for 150 blocks (~60 seconds), so being conservative here costs almost nothing in practice.

Production Numbers

On a three-provider setup (one primary, two backups):

• Mean time to failover: 200–400ms (one health-check cycle)
• Transaction fanout overhead: ~2ms additional latency from the parallel dispatch
• Circuit breaker false-positive rate: < 0.1% over 30 days of mainnet operation
• Effective uptime observed vs. any single provider: 99.97% vs. ~99.91%

The delta sounds small but it compounds — three nines is roughly 8.5 hours of aggregate downtime per year versus two-nines (87 hours). For a bot that makes money in specific windows, that difference matters enormously.

Trade-offs to Know Before You Build

This architecture is not free. Running three providers at real traffic volume costs real money — expect $300–800/month for three high-performance nodes with generous rate limits. You also add a local network hop, though at loopback speeds this is sub-millisecond.

The proxy itself is a new failure surface. Running it as a supervised process (systemd with Restart=always) and keeping the binary stateless (all state in memory, no disk writes in the hot path) keeps it simple to operate. We've shipped this pattern inside our broader infra and data pipelines service, and it underpins every high-frequency Solana bot we run — including the MEV and arbitrage systems where milliseconds of downtime translate directly to missed opportunity.

The proxy also gives you a central place to add rate-limit shaping, request logging, and latency traces — all without touching your bot code. Once it's in place, debugging provider issues goes from "add logging everywhere and redeploy" to "look at the proxy dashboard."

---

If you're building Solana bots that need this level of reliability — or want an existing setup hardened — get in touch. We scope and ship production-grade infrastructure fast.