Storage settings
Every setting below is read from the environment variable in the second
column, or from the same key in a YAML configuration file. An unrecognised
FANOUT_-prefixed variable is a startup error, so a renamed setting surfaces
as a refusal to start rather than as a default nobody chose.
| Setting | Environment variable | Type | Default |
|---|---|---|---|
storage.data_dir | FANOUT_DATA_DIR | string | ./data |
storage.duckdb.max_connections | FANOUT_DUCKDB_MAX_CONNECTIONS | integer | 0 |
storage.duckdb.memory | FANOUT_DUCKDB_MEMORY | string | — |
storage.duckdb.threads | FANOUT_DUCKDB_THREADS | integer | — |
storage.maintenance_interval | FANOUT_MAINTENANCE_INTERVAL | duration | 1h |
storage.merge_interval | FANOUT_MERGE_INTERVAL | duration | 1m |
storage.retention_days | FANOUT_RETENTION_DAYS | integer | 30 |
storage.rollup_interval | FANOUT_ROLLUP_INTERVAL | duration | 1m |
storage.rollup_skip_to_latest | FANOUT_ROLLUP_SKIP_TO_LATEST | boolean | false |
storage.duckdb.max_connections
Section titled “storage.duckdb.max_connections”Caps the DuckDB connection pool. A value of 1 serializes everything through one handle; the machine-sized default lets read queries run concurrently with each other and with ingest flushes. Two things make >1 safe: the DuckLake SQLite catalog is opened in WAL mode (enableCatalogWAL), so readers don’t collide with the single writer and a crashed writer can’t leave the catalog permanently locked; and write commits are serialized by the shared write gate (Duck.WriteGate, wired into the writer via UseWriteGate in cmd/fanout/main.go, enforced at startup). Without the WAL mode, pool >1 fails with “database is locked”. Zero means “size it from the machine” — the same spelling DuckDBThreads uses for deferring to a default. Resolution happens in resolveSizing and is reported in the startup configuration log.
storage.duckdb.memory
Section titled “storage.duckdb.memory”Caps DuckDB’s memory (e.g. “8GB”). Empty means Fanout sizes it from detected memory, reserving headroom for the Go runtime; see resolveSizing. DuckDB’s own default is 80% of detected RAM, which is calculated as though DuckDB owned the machine and has been observed to get the process OOM-killed once the Go heap is added on top.
storage.duckdb.threads
Section titled “storage.duckdb.threads”Caps DuckDB’s global query worker pool. Zero leaves DuckDB’s own default in place (one worker per core). Set it to leave cores free for ingest on a query-heavy co-tenant host.
storage.maintenance_interval
Section titled “storage.maintenance_interval”Throttles the DuckLake maintenance cycle (retention deletes + compaction). Default 1h. Lower it to compact more aggressively, or for soak tests that need to observe file-count staying bounded within minutes rather than hours.
storage.merge_interval
Section titled “storage.merge_interval”The cadence for the cheap, frequent DuckLake file compaction pass (ducklake_merge_adjacent_files only — it consolidates the newest small parquet files and deletes nothing). Run often (default 1m) it keeps the queryable file count continuously low, which is what bounds rollup/query scan latency — WITHOUT the churn, deletion race, or catalog cost of the full hourly maintenance pass (expire + cleanup). 0 disables it.
storage.rollup_skip_to_latest
Section titled “storage.rollup_skip_to_latest”RollupSkipToLatest, set once at boot, advances every rollup watermark to the current max ingested timestamp so existing data is treated as already-rolled-up instead of aggregated as a backlog. Stands up a large pre-seeded historical dataset (benchmarks, restores) without a multi-minute first-rollup catch-up that holds the write gate and starves ingest. Off in normal operation.