Tune retention

Three settings govern what is on disk and how it is shaped. They do different jobs and are worth separating before changing any of them.

How long data is kept

FANOUT_RETENTION_DAYS=30

Zero means keep everything. Changing it does not delete anything immediately — expired data goes when maintenance next runs, so disk use falls on the maintenance cycle rather than at the moment you change the setting.

The maintenance cycle

FANOUT_MAINTENANCE_INTERVAL=1h

This is the expensive pass: retention deletes plus full compaction. Lower it to reclaim space sooner, at the cost of running the heavy work more often.

The merge pass

FANOUT_MERGE_INTERVAL=1m

This is the cheap one, and it is the setting that most often matters for latency. It consolidates the newest small Parquet files and deletes nothing. Running it often keeps the queryable file count continuously low, which is what bounds rollup and query scan time — without the churn, the deletion race or the catalog cost of the full maintenance pass. 0s disables it.

If query latency is climbing on an instance whose data volume has not changed, this is the first thing to look at: a high file count from many small writes costs more per scan than the same bytes in fewer files.

Flush behaviour

How often data reaches disk in the first place:

FANOUT_FLUSH_INTERVAL=15s
FANOUT_FLUSH_BATCH_SIZE=50000

A longer flush interval produces fewer, larger files — less compaction work, at the cost of newly ingested telemetry taking longer to become queryable, and more of it being lost if the process dies uncleanly.

What the knobs interact with

Maintenance and rollups serialise against ingest through the same write gate. Running maintenance much more often on a busy instance therefore trades disk against ingest headroom rather than being free. Change one setting at a time and watch /-/metrics.

The full list, with defaults and types, is in the storage settings reference.