Skip to content

Freshness

The optional write-ahead log (WAL) buffer lets the query server read sealed segments before the compactor commits them to Iceberg. It supports the built-in events table and managed user indexes. This shortens query visibility without changing the ingest acknowledgement or Iceberg commit boundary.

The problem

The compactor batches WAL segments before an Iceberg commit. Its default batch target is 32 MiB, and its default maximum age is 10 seconds. Fewer commits reduce catalog work, but external Iceberg readers cannot see a row until its batch commits.

Reducing the batch size would trade commit throughput for lower visibility delay. The WAL buffer provides another choice for Siglake SQL queries over ingest targets.

Configure freshness for uncommitted WAL data

With Helm, enable the query WAL buffer:

query:
  walBuffer:
    enabled: true

The query pods must mount the same WAL volume as the ingester and compactor. Use a ReadWriteMany volume when those pods can run on different nodes. For a direct binary deployment, set --query-wal-buffer-dir to that shared WAL directory.

The buffer does not change the ingest acknowledgement. It also does not make active segments visible. The limitations below apply after you enable it.

The mechanism

When --query-wal-buffer-dir is set, the query server reads sealed and processing WAL segments from that directory. It exposes each segment through the same logical table as the current Iceberg snapshot. Managed-index batches pass through that index's document mapping before the union.

events query
    |
    +-- committed rows from the Iceberg snapshot
    |
    +-- uncommitted rows from sealed WAL segments

The query server does not read an active segment. A row becomes eligible for the buffer after the ingest server seals its segment. It can also inspect recently committed segments during the snapshot-cache transition and excludes the ones named by the serving snapshot. Query pods therefore need read access to the shared WAL filesystem.

See Helm for the mount and setting. The command-line interface is listed in the CLI reference.

Avoiding double-counting

An Iceberg commit records the WAL segments it consumed in siglake.consumed_segments. The buffer compares its segment list with the snapshot used by the query and excludes segments recorded by that snapshot.

This ties de-duplication to the serving snapshot. A segment moving from the WAL into Iceberg does not appear in both inputs to one query.

Folding into fast paths

Aggregate metadata describes committed Parquet files. A metadata-only answer would omit buffered rows unless the query server handled them separately.

For supported aggregate shapes, the WAL buffer computes the same aggregate over its rows and folds that delta into the committed answer. The query server falls back when it cannot produce a safe delta.

The delta path has byte and segment limits. The byte limit defaults to 2 GiB, and the segment limit defaults to 4,096 entries. If the relevant buffer exceeds either limit, the query returns the committed Iceberg view. This is a partial freshness fallback: uncommitted rows appear as the drain commits them.

These counters report the fallback:

  • siglake_query_wal_buffer_refused_bytes_total
  • siglake_query_wal_buffer_refused_segment_cap_total

SIGLAKE_BUFFER_DELTA_MAX_BYTES and SIGLAKE_BUFFER_DELTA_MAX_SEGMENTS set the limits. A zero value disables the corresponding guard. Time predicates can exclude a framed segment from its header timestamps before Arrow decoding.

OTLP-to-SQL visibility delay

The buffer cannot expose an active segment. By default, the ingest server seals a segment after 4,096 events or five seconds, whichever happens first.

A busy stream usually reaches the event limit first. An idle stream waits for the age limit. With the default five-second age, a lone OTLP record becomes visible to a Siglake SQL query in about five seconds. Lowering --wal-max-age-secs reduces that wait but creates more small segments for the drain and compactor. External Iceberg readers wait for the later compactor commit.

Freshness limitations

The WAL buffer applies to events and managed user indexes referenced by the query. It does not add uncommitted rows to query_audit or other system tables.

The feature is off when --query-wal-buffer-dir is unset. It also depends on shared filesystem access. Without a readable WAL mount, queries cannot provide the intended pre-commit visibility.

These metrics report buffer state and use:

  • siglake_query_wal_buffer_rows
  • siglake_query_wal_buffer_used_total
  • siglake_query_wal_buffer_segments_excluded_total

Live tail

Siglake has two other ways to follow new data:

  • GET /api/v1/stream sends server-sent events from the ingest process. It has no replay or durability. The server publishes before the WAL append, so a later ingest failure can leave a streamed event unacknowledged. A client retry can publish it again.
  • siglake subscribe polls a committed Iceberg table by time column. It can resume, but it refuses to skip an interval after the required snapshots expire.

Use SegmentConsumer when WAL retention must follow the consumer's cursor. The WAL consumer guide compares the replay and recovery boundaries.