Building a Real-Time Solana Data Pipeline with Kafka and ClickHouse

Solana produces around 2,500 transactions per second at peak, and a meaningful fraction of those touch DEX programs, lending protocols, or perp markets that feed directly into trading signal engines. If your latency budget for a signal is under 50 ms, you cannot afford to poll an RPC node — you need a streaming architecture that puts raw chain data in front of your consumers within a single slot. This post walks through the exact setup we run in production: Geyser → Kafka → a lightweight transformer → ClickHouse.

Geyser Plugin: The Only Sane Ingestion Point

The Agave validator exposes a Geyser plugin interface that fires callbacks for every account update, slot status change, and transaction confirmation as they happen inside the validator loop — no polling, no RPC overhead. You compile a shared library that implements the GeyserPlugin trait and pass it via --geyser-plugin-config at validator startup.

The two plugins worth knowing are Yellowstone gRPC (Dragon's Mouth) and plerkle (Kafka-native). We use Yellowstone because it gives you a gRPC subscription stream you can filter server-side by account owner or program ID before a single byte crosses the network. Filtering matters: unfiltered mainnet traffic is roughly 1.2 GB/s of raw protobuf. Scoped to Raydium, Orca, and Phoenix order-book programs, that drops to around 18 MB/s — manageable.

One operational note: run your Geyser node on the same physical host as the validator, or at minimum on the same rack with a 25 GbE link. Any extra hop adds jitter that compounds downstream.

Kafka: Topic Layout and Retention

We publish to three topics:

• sol.transactions.raw — full confirmed transaction protobuf, keyed by signature
• sol.accounts.updates — account state deltas, keyed by pubkey
• sol.slots — slot status events (processed / confirmed / finalized), keyed by slot number

Partition count matters here. We run 32 partitions on sol.transactions.raw with the key hashed mod partitions, which spreads write load evenly and lets us scale consumers horizontally without rebalancing. Retention is set to 4 hours — long enough to replay a bad consumer group without burning storage on a topic that generates 60 GB/hour.

Replication factor of 3, min.insync.replicas=2, acks=all on the producer side. Losing a confirmed Solana transaction from the pipeline is a harder bug to trace than the latency cost of waiting for two ISR acks.

Transformer: From Protobuf to Analytical Schema

The raw Geyser protobuf is not ClickHouse-friendly — it nests inner instructions, token balances, and log messages in ways that make columnar storage painful. A stateless transformer service (we use Rust with rdkafka and prost) consumes sol.transactions.raw, decodes it, and emits a flat JSON record to sol.transactions.decoded.

Key fields extracted per transaction:

• slot, block_time, signature
• fee, compute_units_consumed
• program_ids[] — every unique program invoked, including CPIs
• inner_instructions[] — flattened with a depth column
• pre_token_balances[] / post_token_balances[] — joined by account index

The transformer also emits a separate topic per program (sol.program.675kPXyEAT for Raydium AMM v4, etc.) so downstream consumers can subscribe to only what they care about without writing Kafka Streams topology.

ClickHouse: Schema Design for Low-Latency Reads

ClickHouse ingests from Kafka via the Kafka table engine paired with a materialized view that writes into a ReplicatedMergeTree. The materialized view does the final type coercion and drops fields your signal engine will never read.

The transactions table primary key is (slot, program_id, signature). This ordering is deliberate: slot as the first column means range scans over recent history hit a single part; program_id second means filtering by DEX is a prefix scan, not a full table scan. We see sub-5 ms p99 on SELECT ... WHERE slot BETWEEN X AND Y AND program_id = '675kPX...' against a table with 3 billion rows.

Use LowCardinality(String) for program_id — it cuts dictionary encoding overhead roughly in half compared to plain String when your cardinality is under 10,000 distinct values, which it always is for program IDs.

Partition by toYYYYMMDD(fromUnixTimestamp(block_time)), TTL 30 days. Hot analytical queries never need more than 72 hours of history; the TTL keeps disk usage flat.

Handling Slot Gaps

This is the part most pipeline writeups skip. Solana skips slots — a leader may time out, or a fork gets orphaned. If your signal engine assumes monotonically increasing slots it will silently miscalculate rolling windows.

The fix lives in the sol.slots topic. The transformer maintains a slot cursor in Redis. On every slot event, it checks whether the incoming slot is cursor + 1. If it is not, it emits a synthetic gap record with gap=true and the missing slot range into sol.slot_gaps. The ClickHouse signal layer JOINs against this table before computing any window aggregate and returns NULL (not a stale value) when the window overlaps a gap.

You also want a Grafana alert on slot_lag — the difference between the highest slot in ClickHouse and the current chain tip. Anything over 3 slots at steady state means something in the pipeline is backed up.

Latency Budget in Practice

End-to-end from validator confirmation to a queryable ClickHouse row:

Stage	Median	p99
Geyser callback → Kafka produce	1.2 ms	4 ms
Kafka → transformer consume	2 ms	6 ms
Transformer → decoded topic	0.8 ms	3 ms
ClickHouse Kafka engine flush	100 ms	200 ms

The ClickHouse flush interval is the dominant term and is configurable via kafka_flush_interval_ms. We set it to 100 ms; dropping it to 50 ms roughly doubles CPU on the ClickHouse ingestion node without meaningfully changing signal alpha, so 100 ms is where we sit. For sub-100 ms queries, the transformer output topics are also consumed directly by the signal engine in-process — ClickHouse is for historical analysis and backtesting, not the hot path.

---

If you want this pipeline running under your own trading strategy rather than building it from scratch, reach out to us — infrastructure like this is the foundation of everything we deploy at TierZero.