Real-Time P&L Alerting for Trading Bots: Telegram + Grafana Setup

Real-time P&L alerting for trading bots is the difference between catching a drawdown at -3% and waking up to a -30% account. Most teams bolt on alerting as an afterthought — a cron script that checks a database row and fires a message. That works until it doesn't: stale data, duplicate alerts at 3 AM, or silence during the one incident that mattered. This guide walks through a production-grade setup that ships with TierZero's bot builds: a time-series Grafana stack feeding threshold-based Telegram alerts, with cooldown windows and deduplication so the alerts mean something when they fire.

Streaming Position and Realized P&L into a Time-Series Store

The first decision is the data model. Grafana wants time-series data, so every P&L and position event needs a timestamp, a label set, and a numeric value — not a blob in a relational table.

The cleanest path is writing directly to Prometheus (via a push gateway or an exposed /metrics endpoint) or to InfluxDB if you need higher cardinality. For a single-bot setup, Prometheus is simpler. Expose these metrics from your bot process:

• bot_realized_pnl_usd — cumulative realized P&L since session start, labeled by market and strategy
• bot_unrealized_pnl_usd — mark-to-market on open positions
• bot_position_size — current net position, signed
• bot_fee_paid_usd — cumulative fees, labeled by fee_type (taker, maker, gas)
• bot_drawdown_pct — rolling drawdown from session high-water mark, as a negative float

The high-water mark calculation belongs in the bot, not in Grafana. Keep a running session_peak_equity variable and update drawdown_pct = (current_equity - session_peak) / session_peak on every fill. Pushing a pre-computed drawdown metric makes the alert rule trivial and avoids time-window gymnastics in PromQL.

For Hyperliquid and Polymarket bots, realized P&L is straightforward — you have fill events. For Solana bots (sniper, copytrading, market-maker), you need to track entry cost per mint. Store a cost_basis map keyed by mint address and compute realized P&L on each sell fill.

Grafana Dashboard Layout

One panel per concern, not one panel per metric. A useful layout for a single bot:

Row 1 — The scoreboard

• Stat panel: cumulative realized P&L (big number, green/red threshold coloring at 0)
• Stat panel: current drawdown % (red at -5%, dark red at -10%)
• Stat panel: open position size
• Stat panel: fees paid today vs daily budget

Row 2 — Time-series context

• Graph: realized + unrealized P&L over time (dual series)
• Graph: position size over time (useful for spotting stuck positions)
• Graph: fee burn rate (rolling 1h sum of bot_fee_paid_usd)

Row 3 — Debug layer

• Table: last 20 fills with market, side, size, price, realized P&L per trade
• Graph: drawdown % over time with threshold band overlay

Use Grafana's Unified Alerting (not the legacy channel-based system). Set the datasource to Prometheus and write alert rules against the same metrics your panels display — this way, a firing alert corresponds to exactly what you see on the dashboard.

Threshold-Based Telegram Alerts

Grafana's Unified Alerting routes alerts to contact points. Create a Telegram contact point under Alerting → Contact Points, paste your bot token and chat ID, and test it before writing a single rule.

Alert rules that actually matter in production:

Drawdown breached

bot_drawdown_pct < -0.05

Pending for 30s (avoids noise from a single bad mark), then fires. Message: [BOT] Drawdown -{{ $values.A.Value | printf "%.1f" }}% — session high-water: ${{ $values.B.Value | printf "%.0f" }}. Include the high-water mark so you know the absolute context, not just the percentage.

Flat fee overrun

increase(bot_fee_paid_usd[1h]) > 50

If your strategy budget is $50/hour in fees, this fires when you've already burned that in the last rolling hour. Critical for volume bots and market-makers where fee drag is a slow poison rather than a sudden event.

Position stuck

abs(bot_position_size) > 0 and (time() - bot_last_fill_timestamp) > 1800

An open position with no fill in 30 minutes. Either the bot is frozen, the market is dead, or you're carrying risk you forgot about. This alert has saved more money than any P&L threshold in my experience.

Realized P&L floor

bot_realized_pnl_usd < -200

Hard stop level. Below this, the right action is human review before the bot continues. Pair it with a kill-switch in your Trading Dashboard so the alert can link directly to the stop control.

Avoiding Alert Fatigue with Cooldowns and Deduplication

A drawdown alert that fires every 30 seconds during a bad market is worse than no alert — you train yourself to dismiss them. Grafana's Unified Alerting has two mechanisms:

Evaluation interval and for duration: Set your evaluation interval to 60s and the for duration to 2m for most rules. The alert only fires if the condition has been true for 2 continuous minutes. This filters almost all transient noise without meaningful delay on real events.

Repeat interval on notification policies: Under Alerting → Notification Policies, set Repeat interval to 4h for drawdown alerts. Once the alert has fired and the situation is known, you don't need it every 60 seconds. Set it shorter (15m) for stuck-position alerts since those can change faster.

Resolved notifications: Enable Send resolved for all rules. A resolved Telegram message is signal too — it confirms the bot recovered without manual intervention.

Inhibition rules: If a hard drawdown floor alert (-$200) is firing, silence the softer -5% drawdown alert. They're the same event at different severity. Set up an inhibition rule matching on bot_instance label so high-severity alerts suppress lower ones.

One more thing: deduplicate at the message level. Telegram doesn't deduplicate by default, but you can hash the alert name + labels and compare to a Redis or in-memory set before sending. If the same alert hash fired in the last N minutes, skip the send. This is especially useful when running multiple bots in the same alert group.

Structuring Alert Messages for Fast Triage

The Telegram message format determines how quickly you can act at 2 AM. Skip the generic boilerplate. Every alert message should answer in 5 seconds: what happened, how bad, what to check first.

A working template:

⚠️ [{{ $labels.strategy }}] {{ $labels.alertname }}
Market: {{ $labels.market }}
Value: {{ $values.A.Value | printf "%.2f" }}
Threshold: {{ $labels.threshold }}
Session P&L: ${{ $values.pnl.Value | printf "%.0f" }}
Dashboard: https://grafana.yourdomain.com/d/abc123

Keep the dashboard link in every message. The 10 seconds it saves navigating there matters when you're half awake. Use Grafana's annotation API to write an event marker to the dashboard the moment an alert fires — this puts a vertical line on your P&L chart exactly when the threshold was breached, which is invaluable during post-mortems.

Production Deployment Notes

Run Prometheus, Grafana, and Alertmanager in a single Docker Compose stack on the same host as your bot, or on a dedicated $10/month VPS. The latency of metrics scraping (typically 15s) is acceptable for P&L alerting — this is not the path for sub-second execution signals.

Persist Grafana dashboards as JSON in version control. A crashed Grafana container should come back with the same panels in under two minutes. Use GF_SECURITY_ADMIN_PASSWORD via environment variable, not the web UI default.

For bots running across Solana, Hyperliquid, and Polymarket simultaneously, label every metric with bot_instance and chain. This lets you build a single multi-bot overview dashboard and route alerts by instance to different Telegram chats — one channel per strategy, one aggregated ops channel for the on-call.

---

If you want this wiring done right the first time — metrics endpoint, Grafana stack, and Telegram alerting configured before the bot goes live — talk to us about a build.