Set up alerts

The alert engine runs inside the binary and is on by default. It evaluates persisted rules every FANOUT_ALERTS_EVALUATION_INTERVAL (30s) against service rollups, and delivers by webhook.

Rules are managed over HTTP. There is no browser page for them yet — see capabilities.

Endpoint	Does
GET /api/alerts	Current alert state
GET /api/alerts/summary	Counts by state
GET /api/rules	List rules
POST /api/rules	Create and compile a rule
PUT /api/rules/:id	Replace and recompile a rule

Reading takes telemetry:read; creating and replacing takes alerts:manage, which starts at the operator role. See roles.

What a rule contains

An expression, an optional service to scope it to, timing, and delivery:

Field	Means
expression	The condition, in expr-lang
service	Evaluate only for this service; omit for all
for_seconds	How long the condition must hold before firing
cooldown_s	Minimum quiet period after resolving before firing again
repeat_interval_s	How often to re-deliver while still firing
webhook_url	Where to deliver
webhook_headers	Extra headers, for a shared secret or routing key
webhook_template	Custom payload body; omit for the default
notify_on_resolve	Also deliver when the alert clears

Variables available to an expression

An expression reads one service’s rollup for the current window. The full list — error_rate, p95, throughput, the baseline deltas and the rest — is generated from the evaluation environment itself, so it cannot drift from what the evaluator actually binds.

So a rule body reads like the thing it means:

error_rate > 0.02 and throughput > 10

The throughput clause matters more than it looks. A service handling two requests a minute reaches a 50% error rate on one failure, and without a floor every quiet service pages you.

The lifecycle

An alert moves through three states:

• pending — the condition is true but has not held for for_seconds yet. With for_seconds: 0 a rule skips this and fires immediately.
• firing — it has held. The webhook is delivered, and re-delivered every repeat_interval_s while it stays true.
• resolved — the expression went false. If notify_on_resolve is set, that is delivered too.

cooldown_s then holds the rule quiet for that long after resolving, which is what stops a metric hovering at the threshold from delivering repeatedly.

Each alert records its last delivery status and timestamp, so a webhook endpoint that started rejecting is visible in GET /api/alerts rather than silent.

The namespace field on a rule is not applied

A rule carries a namespace selector and it is stored, but evaluation reads instance-wide service rollups and does not filter on it. A rule you believe is scoped to one namespace is evaluating across all of them.

Scope with service instead, which is applied. This is a real gap rather than a subtlety — it is marked here because the field’s presence implies otherwise.

Why alerts lag the raw data

Evaluation reads rollups, not raw telemetry, and rollups deliberately trail the newest data by a safety margin because telemetry arrives out of order. An alert therefore fires seconds after the condition is visible in a raw query. That is a correctness choice — how storage works explains the margin — and it means for_seconds should not be set below the rollup interval expecting finer resolution than the data has.