Configuration reference¶

CLI flags and SIGLAKE_* environment variables configure Siglake. If both set the same option, the CLI flag takes precedence. Siglake's own OTLP export is configured with the standard OTEL_* variables instead, under Observability.

Object-store credentials use the provider's standard environment variables. For example, S3-compatible stores use AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION and AWS_ENDPOINT_URL.

Core addressing¶

Every role needs to know where the warehouse and catalog are.

Authentication¶

Ingest¶

WAL¶

Segment roll thresholds are flags, not env vars: --wal-max-events (4096) and --wal-max-age-secs (5). The age bound is the freshness floor.

Drain and commit¶

Compaction¶

Catalog claim¶

The catalog claim mechanism protects a write-ahead log (WAL) segment from being drained by two compactor pods. Two settings control stale catalog claims. Catalog claims have no renewable lease or claim time to live (TTL). SIGLAKE_CLAIM_RECLAIM_MAX_AGE_SECS controls when recovery considers a processing claim abandoned. SIGLAKE_CLAIM_RECLAIM_INTERVAL_SECS controls how often recovery checks.

Catalog claim settings¶

These settings control recovery and retention for catalog-claim drains.

Indexing and promotion¶

Where an inverted index lands: Parquet footer or Puffin sidecar¶

SIGLAKE_INDEX_FOOTER_MAX_BYTES is the per-column threshold. A serialized index within it rides in the Parquet footer's metadata. A column whose index exceeds it goes to a Puffin sidecar instead, and the rewrite commit does not register that sidecar.

A streamed rewrite writes no footer inverted index at all, so every merge past the in-memory caps leaves its output unindexed until something registers a sidecar. That something is the post-rewrite pass SIGLAKE_INDEX_REBUILD turns on. The pass skips a file and column whose Parquet footer already carries the index or whose table metadata already registers a blob for the pair. Reads do not depend on the switch: a query uses whatever index a file carries and scans a file with none. Post-compaction index rebuilding gives the cost behind that default.

Segmented text indexes: the seg2 write and read switches¶

SIGLAKE_SEGMENTED_INDEX_WRITES and SIGLAKE_SEGMENTED_INDEX_READS move independently, and both are off by default. Writing a seg2 sidecar does not make a query read one, and reading is safe on a table no writer has touched.

Set writes to 1 and the compactor's streaming re-cluster builds a segmented (seg2) text index one Parquet row group at a time and registers it with the rewrite commit. Any other value, including an unset variable, leaves it off. The in-memory merge arm, ingest flush and delete-task rewrites write v1 indexes either way.

Set reads to 1, true, yes or on and a query answers a text predicate from a seg2 sidecar with partial reads. Any other value leaves reads off, and a file whose only text index is seg2 is scanned. The value is read once per process, and nothing consults the seg2 directory cache while reads are off. That cache budget, SIGLAKE_SEGMENTED_INDEX_DIRECTORY_CACHE_MAX_BYTES, is counted across every warehouse the process reads; at 0 every lookup rereads the trailer and the directory.

Neither switch has a chart value or an operator field; set them through compactor.extraEnv and query.extraEnv. Segmented text indexes (seg2) has the format and the metadata names.

Auto-promotion sampling bounds¶

SIGLAKE_AUTO_PROMOTE_SAMPLE_FILES and SIGLAKE_AUTO_PROMOTE_SAMPLE_ROWS decide what a promotion is inferred from and what the sampling pass costs. Their product is the number of residual JSON documents one pass parses, every 300 seconds, on a compactor whose SIGLAKE_AUTO_PROMOTE_MIN_PCT is above zero. At the defaults that is 4 files x 4,096 rows. Sample more rows when a wanted attribute sits below the threshold only because the sample is small; raise the file count when recent files differ in shape. Neither bound turns promotion off: SIGLAKE_AUTO_PROMOTE_MIN_PCT=0 does, and SIGLAKE_AUTO_PROMOTE_MAX_COLUMNS=0 leaves the pass inert with the threshold in place. Neither has a chart value; set them through compactor.extraEnv.

Query¶

Row caps¶

SIGLAKE_QUERY_MAX_ROWS is not the only bound on rows returned. Each priority tier carries its own default and its own ceiling, and the variable is folded into the ceiling only:

The cap a request actually runs under is

effective_ceiling = min(tier ceiling, SIGLAKE_QUERY_MAX_ROWS)
resolved_cap      = min(limits.max_rows_returned or the tier default, effective_ceiling)

so the variable can only tighten those numbers, never loosen them. Two consequences an operator raising it should expect:

• Raising it above 10000000 changes nothing on the interactive tier, and above 1000000000 nothing on the batch tier: the tier ceiling wins. At the shipped default of 1000000 the variable is the binding interactive cap, so it is the one to raise for a records response that came back truncated up to 10,000,000. Above that ceiling, the request needs a smaller result rather than a larger cap. It is also one possible binding ceiling on a Jaeger read refused with 413; the Jaeger-specific trace, span-row, Arrow-byte, and name ceilings instead follow the admission reservation, which moves with SIGLAKE_QUERY_ADMISSION_BUDGET_BYTES, the share divisor, and pod memory.
• A batch request that sets no limits.max_rows_returned resolves to 100,000 rows however high the variable is set, because the binding number there is the tier default and the variable does not touch defaults. A batch caller who wants more has to ask per request; that ask is then clamped by the batch ceiling and by this variable.

Clipped-limit admission ramp¶

An unordered LIMIT scan starts one partition and widens the admitted set each time the admitted partitions have emitted their own number of batches or one of them ends. SIGLAKE_SCAN_CLIPPED_ADMISSION_WAVE is the factor it widens by, 2 by default, so the admitted count doubles per step and the scan reaches full fan-out in log2(partitions) steps. 0 turns the ramp off and starts every partition at once. 1 resolves to 2, and an unparseable value falls back to 2. The ramp is scheduling only: a partition that waits still reads every row it would have read, and the limit still applies above the residual filter, so no qualifying row is lost. It has no chart value or operator field; set it through query.extraEnv. Pruning names the two metrics that report it.

Guardrails¶

Spill¶

Sorts and aggregates that reach the shared memory pool spill to local disk instead of growing until the kernel intervenes; see Execution tiers. The parsing rules differ from the Helm chart's packaged values, which are described under Query spill and ephemeral storage.

Caches¶

Observability¶

OTLP export of Siglake's own logs and traces¶

Since 0.2.0, every Siglake process can export its own log records and spans over OTLP/HTTP. Export is off until OTEL_EXPORTER_OTLP_ENDPOINT is set, and metrics are not part of it: they stay on /metrics for Prometheus. Each variable is read once, at startup. Export Siglake's own logs and traces has the deployment steps and the collector configuration an OTLP-only backend needs.

RUST_LOG selects which log records export, the same filter it applies to the console. HOSTNAME, HOST and POD_NAME are not Siglake variables: a Kubernetes pod already carries HOSTNAME, and POD_NAME is one you inject yourself with the downward API.

Full inventory¶

Every SIGLAKE_* variable referenced anywhere in the public source tree, plus the OTEL_* variables the telemetry initializer reads, with the crates that read each one. A crate is named by itself; the vendored Iceberg forks are named by their third_party/<fork> path. Variables not documented above are internal tuning knobs or test hooks. Treat them as unstable.

The generator takes an OTEL_* name only from a quoted lookup in the source, so the table holds the standard OpenTelemetry environment and not the unrelated constants the load generator names OTEL_*. HOSTNAME, HOST and POD_NAME are read too and carry no prefix the generator can match, so they appear in OTLP export of Siglake's own logs and traces above and not here.