Architecture¶

Siglake separates ingest, storage maintenance, and query into three roles. They share a write-ahead log (WAL), an Iceberg catalog, and object storage.

OTLP and Elasticsearch clients
            |
            v
     ingest server -> WAL segments -> compactor -> Iceberg tables
                         |                            |
                         +---------- query server ---+

This split lets you scale each role on its own. It also creates four distinct states for an event:

• The default acknowledgement follows a local sync of the segment bytes and directory entries for the tenant, index, active segment, and sealed segment. Its power-loss guarantee assumes ext4 or xfs on a node-attached volume; a network filesystem provides whatever its fsync(2) and rename semantics guarantee. The acknowledgement does not wait for a mirror upload or Iceberg commit.
• With commit=auto, the ingest server acknowledges after the WAL write while bytes can remain in the kernel page cache.
• The query server can read sealed WAL segments before the compactor commits them to an ingest target.
• Other Iceberg readers see the event only after the Iceberg commit.

Ingest and the WAL explains these boundaries.

The central idea: the WAL is the streaming substrate¶

The WAL is a directory of Arrow IPC segments on a shared filesystem. The ingest server appends to an active segment, then seals it by size or age. Sealed segments do not change.

The compactor drains sealed segments into Iceberg. The query server can read the same segments to include recent rows before an Iceberg commit. External processes can consume them through SegmentConsumer, with a cursor for each consumer.

The WAL therefore handles buffering and replay without a separate message broker. This keeps the data path short, but every process that reads the WAL needs access to its shared filesystem. WAL acknowledgement does not depend on an object-store mirror.

The roles¶

The ingester can run the compactor in the same process with --with-compactor. Production deployments can run the roles separately.

Storage: open by construction¶

Committed data lives in Apache Iceberg format-version-2 tables backed by Parquet files. Each index has its own table. Siglake partitions these tables by event day and sorts rows by the table's declared timestamp sort order.

The catalog can use SQLite for local work or PostgreSQL for shared deployments. The data files remain ordinary Iceberg and Parquet files, so external Iceberg readers do not need Siglake in their query path. See External query engines for tested readers and catalog limits.

What makes queries fast¶

Siglake avoids or reduces data-file reads in four ways:

1. Exact aggregate metadata can answer supported count and grouping queries.
2. Parquet statistics, bloom filters, and optional inverted indexes prune scan work.
3. Timestamp ordering lets a compatible ORDER BY ... LIMIT stop after a prefix of the table.
4. Query replicas can divide eligible large scans by file and merge the partial results.

Each path has a fallback. Missing or stale aggregate metadata causes a slower exact path. Missing pruning data causes a scan. An unsafe distributed plan runs on one replica. Query engine describes those choices.

Reading order¶

The crate map maps these concepts to the Rust workspace.