Changelog¶

The authoritative changelog is CHANGELOG.md in the source repository. This page includes runtime changes through Siglake merge commit ca8d598 from 2026-09-08. It also includes a 2026-09-09 correction to the Jaeger name-list cache limits. The overview reflects the 2026-09-11 durability and OTLP/gRPC defaults from merge commits b23bffa and 94ea42e.

0.1.0: initial public release¶

Siglake is a horizontally-scalable, OTLP-native log analytics platform on Parquet v2, Apache Iceberg and DataFusion.

Timestamp handling, Jaeger reads, and what to do on upgrade¶

0.1.0 is the first public release, so there is no earlier release line to upgrade from. Five changes inside the line need an action if you are moving a pre-release install onto the release binary.

Rolling ingesters and drains across a version change also takes one retention setting, applied before the first new binary starts. See Consumed-proof rolling upgrades.

2026-09-09: Jaeger name-list cache correction¶

Later 0.1.0 changes replaced the cache limits in the 2026-09-08 entry. A complete Jaeger name list is cached when its retained arena and stored lookup string together fit the 512 KiB per-entry cap. The shim's derived name ceiling supplies the row bound. The SQL cache's 128-row entry cap does not apply to these lists. See the Jaeger shim reference.

2026-09-08: bounded Jaeger reads¶

The Jaeger HTTP shim now derives trace, span-row, accumulated Arrow-byte, and distinct-name ceilings from each request's admission reservation. An excessive search limit is rejected with 400 before admission or planning. A result that crosses another ceiling is refused whole with 413, no data, and no Retry-After. The four refusal causes are exposed as jaeger_* values on siglake_query_breaker_trips_total. See the Jaeger shim reference. Shipped in Siglake merge commit d4b60e1.

The span plan now fetches at most one row beyond its row ceiling. A refused request no longer materializes the full match before returning 413.

2026-09-08: release hardening¶

• An unknown field in a SQL request's limits object now returns 422 before planning, execution or batch enqueue. Unknown top-level keys remain allowed, and known limits above their tier ceiling remain clamped. The row-limit field is max_rows_returned, not the response field max_rows. See the SQL query reference.
• Delete sweeps now report claimed tasks that remain non-terminal for more than twice SIGLAKE_DRAIN_WATCHDOG_SECS. The siglake_compactor_delete_tasks_stalled_total{state} counter drives the SiglakeDeleteTaskStalled alert. The dashboard-only siglake_compactor_delete_tasks_nonterminal{state} gauge records the last complete observation. See Monitor Siglake.
• The Jaeger service and operation name routes now cache complete lists by table snapshot. Lists above 128 rows or 4 MiB remain uncached. See the Jaeger shim reference.

2026-09-06: portable timestamp contract¶

Siglake tables now use Iceberg format version 2. events.timestamp is a microsecond timestamptz (Parquet INT64 TIMESTAMP(MICROS, UTC)), and the required timestamp_ns long sibling preserves the OTLP nanosecond verbatim. Trino 483, Spark 3.5.9 with Iceberg 1.11.0, DuckDB 1.5.5 with core iceberg extension 45163a28, and PyIceberg 0.12.0 all read a fresh 2,000-row fixture and agreed on its exact timestamp_ns bounds. See External query engines.

Release overview¶

Ingest accepts OTLP/HTTP logs over POST /v1/logs and an Elasticsearch-compatible _bulk surface with per-index document mappings. The ingester also listens for OTLP/gRPC logs and traces on 0.0.0.0:4317 by default. The WAL force-seals on graceful shutdown. Backpressure lanes and per-tenant rate budgets bound the accept path. Auth is Authorization: Bearer <token> or OIDC.

Durability rests on the WAL. A request is acked once its rows are in the WAL. By default, the ingester calls fsync(2) before it acknowledges the request. The sync covers segment bytes and the directory entries that name the segment. A seal publishes the sealed name durably before unlinking the active copy. The power-loss guarantee assumes ext4 or xfs on a node-attached volume. A network filesystem provides whatever its fsync(2) and rename semantics guarantee.

Send ?commit=auto to accept an acknowledgement after write(2) instead. A node-level crash or power loss can discard rows that the kernel has not flushed in that mode. Sealed WAL segments are mirrored to object storage by default, but the asynchronous upload is outside the acknowledgement path. Neither mode waits for the rows to become queryable; that follows the drain's cadence (measured p50 ~5.5 s).

Multi-tenancy is single-tenant by default. Every ingest request routes to default. An X-Scope-OrgID naming another tenant is refused with 403 over HTTP or PermissionDenied over OpenTelemetry Protocol (OTLP)/gRPC. Set ingester.oidc.tenantClaim to route by a verified JSON Web Token (JWT) claim, or ingester.trustScopeHeader to trust the header. The claim option requires a usable claim, and a present header may only agree with it. Optional --allowed-tenants and --max-tenants bound what a client can create. The query server applies the claim rule through its own --oidc-tenant-claim: a claim that is missing, blank, non-string, over 128 characters or outside [A-Za-z0-9_-] is a 403 before routing rather than a fall back to the default namespace, and identifiers are validated rather than repaired, so acme.corp cannot reach acmecorp's data.

Storage is Iceberg tables on any object store: physically time-ordered Parquet with per-file group-count footers (typed columns included), token and trigram bloom filters, and continuous leveled compaction with overlap-depth convergence. The compactor's siglake_table_live_data_files gauge is read from the snapshot summary each cycle, so it stays exact on tables too large to walk inside the sampler's per-table budget. The level and depth gauges can lag, and siglake_table_gauges_sampled_at_seconds says when they were last walked. Group-count delta writes get four attempts with retry and failure metrics. An exhausted write leaves a durable marker for the maintenance compactor to rebuild automatically. siglake_group_count_auto_rebuilds_total{table,outcome} records the result, and SiglakeGroupCountDeltaLost fires only when that rebuild fails or remains incomplete. As the operator fallback, siglake rebuild-group-counts repairs the aggregate from committed files without double-folding late deltas; its --admit-typed-columns adds the typed columns a table created before typed side aggregates never carried, so no rewrite is needed for those.

Query is DataFusion SQL (/api/v1/sql) with a transparent distributed coordinator. Ordered-scan early stop in both time directions (reversed tail-chunk decode). Zero-scan fast paths for counts, group-bys, histograms, distinct counts, and dimensional or typed filters served from footers and snapshot-keyed side aggregates. Snapshot-keyed result and decoded-chunk caches, invalidated by commit and never by TTL. Per-request scan and cost attribution (stats.scan). Bounded query spill uses SIGLAKE_QUERY_SPILL_DIR and SIGLAKE_QUERY_SPILL_MAX_BYTES, the chart's query.spill.* ladder, and a matching query-pod ephemeral-storage limit. Memory-pool or spill-cap refusals return 503 plus Retry-After, including refusals forwarded from workers, and increment siglake_query_breaker_trips_total{breaker="pool_exhausted"}.

Freshness comes from sealed-WAL buffer serving. Records are queryable in seconds, before commit, and are folded exactly into every fast path.

Attribute capture is lossless. OTLP resource and log attributes stay queryable via attr_get(). Hot keys auto-promote to typed columns with backfill and query rewrite (opt-in).

Streaming consumers read the WAL directly. siglake_wal::consumer::SegmentConsumer is a supported interface for reading WAL segments as they seal, with a durable cursor, at-least-once delivery, retention that waits for slow consumers (bounded, so a stuck consumer degrades to "you missed some" rather than filling the disk), and CRC integrity. See docs/CONSUMING_SEGMENTS.md in the source repository. The four-tier semantic detection pipeline that shipped inside Siglake through 2026-08-29 was moved out to run entirely on top of this interface and is maintained as its reference consumer.

Schema evolution is additive and gated. A table records the schema version it is at. When the running binary is newer, the columns the table lacks cannot be written, so the write is refused, naming the column and the remedy, rather than silently dropped. Run siglake migrate-schema --all-tables --all-namespaces; it is additive-only and idempotent, and --all-namespaces matters on any multi-tenant install because events exists once per tenant namespace. Each control plane runs it on a trigger. The chart renders the pre-upgrade hook while schemaMigration.enabled is true, its default. The operator renders the Job when you raise the monotonic spec.schemaVersion counter, and holds the workload rollout until the Job finishes; the counter says "run a migration" rather than naming a target version, and leaving it unset or at 0 renders no Job. See Schema migrations. Anywhere neither trigger fires, run the command yourself before the new binary serves writes.

The operational surface is Helm charts and a Kubernetes operator (SiglakeCluster CRD), with per-tier spec.resources.<tier>, the chart's 4Gi query-pod default, schema-migration jobs, and offline Helm-release adoption. Invalid configurations set InvalidSpec with one of ten reasons rather than being partly honored; the former QueryAutoscalingIgnored condition is removed. A Terraform/EKS reference deployment covers BYOC installs. The release also carries Prometheus metrics throughout, Prometheus alert rules for the silent-loss counters, audit logging, and retention, GC and delete sweeps.

Interoperability rests on the open format. The warehouse is plain Iceberg-on-Parquet at format version 2, with no v3-only type in any schema. Event time is timestamp, a microsecond timestamptz (Parquet INT64 TIMESTAMP(MICROS, UTC)) that every Iceberg reader maps. events also carries timestamp_ns, a required long holding the OTLP time_unix_nano value verbatim, so nanosecond exactness is available externally and Siglake's own scans lose nothing. Each engine reconstructs the nanosecond timestamp with its own conversion, listed under Reconstruct nanosecond timestamps from timestamp_ns. Trino, Spark, DuckDB and PyIceberg all read the warehouse without Siglake in the path; the versions tested and the measured output are in External query engines . Warehouses written before this change are format version 3 and must be recreated, not migrated (docs/DESIGN_time_ordered_storage.md, "Timestamp contract").

See the Performance section of README.md for measured comparisons against Quickwit, Elasticsearch, ClickHouse, and a vanilla-Parquet DuckDB baseline.

Before 0.1.0¶

The project was named knulps until 2026-06-12. Pre-rename docs are kept in the internal history, and older git history uses the old name.

Development was phased: the storage engine, ingest, and query (phases 1 to 4); scale-out, multi-tenancy, and hardening; the former in-tree detector pipeline (phase 5, retired in source commit e2a6ab2 on 2026-08-29); and the 2026-06/07 performance arc. See About.