Data model¶

Committed Siglake data uses three kinds of Iceberg table: the built-in events table, managed user indexes, and system tables. The events table has a fixed core schema. User indexes use mappings supplied by the operator.

The events table shape and required columns¶

The core columns and their stable Iceberg field IDs come from the schema declaration:

Promoted columns follow these fields. The generated schema reference also lists their Arrow types.

timestamp stores event time with microsecond precision. Siglake partitions on day(timestamp) and uses it as the first sort field. timestamp_ns keeps the original nanosecond value and breaks microsecond ties. Storage layout explains how this order affects scans.

Sending data lists the OpenTelemetry Protocol (OTLP) field mapping.

All eight core columns are present on every row except nullable attributes. timestamp, timestamp_ns, and raw come directly from the OTLP record. host, source, sourcetype, and index use resource or record attributes with fixed fallbacks. Columns numbered 9 and above are optional promoted attributes.

Choose residual, automatic, or declared attribute promotion¶

Residual attributes stay in the nullable attributes JSON string. They need no schema change, but each query must extract and cast the value.

Declared promotion adds a typed Parquet column for a named attribute. Use it when that attribute is common in filters or groups. Automatic promotion samples frequent attributes and chooses columns after ingestion. It is off until SIGLAKE_AUTO_PROMOTE_MIN_PCT is greater than zero.

Both promotion paths keep the original value in attributes. Declared promotion gives you a predictable schema. Automatic promotion trades that control for less setup and can trigger later backfill work.

Residual attributes¶

OTLP attributes that do not map to core columns remain in the nullable attributes column as a JSON object string. attr_get reads a named value:

SELECT attr_get(attributes, 'k8s.namespace') AS namespace, count(*)
FROM events
GROUP BY namespace;

attr_get returns text. Cast it when the source value is numeric or boolean:

WHERE CAST(attr_get(attributes, 'http.status_code') AS INT) >= 500

Tables created before the attributes field was added need an additive schema migration. New tables include it when Siglake creates them.

Promoted columns¶

A promoted column copies one named attribute into a typed Parquet column. The supported types are string, int, float, and bool.

Promotion gives the value Parquet statistics and its declared Arrow type. String columns can also use dictionary encoding and group-count metadata. The original attribute stays in attributes.

The table property siglake.promoted.v1 records the mapping from attribute name to column name and type. The query planner can rewrite an attr_get predicate for that attribute to the promoted column. Existing SQL therefore keeps working after a promotion.

Declared vs automatic promotion¶

Declared promotion is the direct path: you provide an attribute name, output column, and type.

Automatic promotion samples attribute names in the compactor. It is off until SIGLAKE_AUTO_PROMOTE_MIN_PCT is greater than zero.

The compactor backfills older files after adding an automatic promotion. It sets siglake.promotion_backfill_complete.v1 only after that work completes, so readers can distinguish a partial backfill from a complete one.

The cap counts the table's whole promoted list, so columns you declared by hand occupy it too. A table that reaches the cap stops sampling: the pass could add nothing, so it skips the sample rather than paying for a verdict it cannot act on, and it reports no candidate count for that table. Raising the cap admits more schema additions you cannot undo. Setting SIGLAKE_AUTO_PROMOTE_MIN_PCT to 0 stops future automatic additions and keeps the columns already added.

Automatic promotion is observable per table. Each pass records one of four outcomes, and the compactor logs the attribute names it promoted along with the ones it refused for the cap, a schema name collision or mixed sampled types. See What is attribute auto-promotion doing to my schema? for the metrics and the near-cap alert.

User indexes¶

The Elasticsearch _bulk endpoint can write to managed user indexes. Each index has its own Iceberg table and document mapping.

Field types are text, long, double, bool, datetime, bytes, and json. Text fields support the default, raw, and stem tokenizers.

The mapping mode controls fields that are absent from field_mappings:

siglake_compactor_strict_residual_rows_total counts rows that strict mode would reject. Do not use strict mode as a data-quality boundary until the documented limitation is removed.

The user indexes guide covers index creation and mapping examples.

Choose index_at_flush for ingest cost or query latency¶

index_at_flush chooses when Siglake builds enabled raw-text search metadata. It does not enable an index type.

If it is true, ingest can build the metadata in each new Parquet file. If it is false, compaction can build the enabled metadata when it rewrites that file. Until then, queries scan files that lack an accelerator, so results stay correct while latency can increase.

Leave it false when ingest throughput matters more than immediate search acceleration. Set it true when new files must be searchable with enabled raw-text indexes before compaction reaches them. Either choice affects cost and latency, not query results.

Inverted indexes have a separate switch, compactor.invertedIndex.enabled or SIGLAKE_INVERTED_INDEX. That switch is on by default; set the Helm value to false or the variable to 0 to opt out. The per-index index_at_flush field applies to user indexes. The built-in events table uses the deployment-wide SIGLAKE_INDEX_AT_FLUSH setting.

See Search acceleration for the switches that control each artifact.

Use query audit to investigate slow or expensive queries¶

Retained records in the query_audit Iceberg table contain the request identity, SQL, cost, outcome, and timing fields listed in the schema reference.

Query it like any other table:

SELECT query, count(*) AS runs, avg(duration_ms) AS avg_ms,
       max(estimated_bytes_scanned) AS max_bytes
FROM query_audit
WHERE timestamp >= now() - INTERVAL '24 hours'
GROUP BY query
ORDER BY max_bytes DESC
LIMIT 20;

Metadata versions on the data path¶

Siglake versions its own persisted metadata. Accelerators such as group counts are optional: if a reader cannot use them, it runs the exact fallback.

Pruning metadata requires a stricter rule because a false negative could omit matching rows. Readers ignore an unknown bloom or inverted-index format and scan the data instead of interpreting unknown bytes.

Compaction can replace old metadata while rewriting a file for layout. Siglake does not require a file rewrite only because an optional accelerator format changed. The source tree's docs/DESIGN_file_formats.md records the format keys and versions.

The Iceberg table format and the schema are separate contracts from these artifact versions. How the storage format is versioned covers all three and what each one costs you at upgrade time.