Metrics reference¶

Each Siglake role exposes Prometheus metrics on a separate /metrics port. This page lists the main signals, labels and export types.

The Helm chart can render a ServiceMonitor with serviceMonitor.enabled and a 39-alert PrometheusRule with prometheusRule.enabled. Both values default to off and require the Prometheus Operator. Monitoring lists the alerts and operator actions. The chart does not render deploy/grafana/siglake-overview.json; import that dashboard separately. See Starter Grafana dashboard for its panels and related alerts.

Selected operational metrics¶

These tables group selected metrics by operational purpose.

Build provenance¶

siglake_build_info exposes the binary's version and source commit labels. These build provenance labels identify the running binary. Confirm that it matches a released version by comparing both labels with the intended release and across every scrape target.

Build information metric¶

Ingest health¶

All six siglake_ingest_tenant_denied_total{reason} series are pre-registered, so the first denial after enabling ingester.oidc.tenantClaim is visible rather than hidden behind an empty graph, and the chart's SiglakeTenantsDenied alert watches the counter. It is the ingest counterpart of siglake_query_tenant_denied_total; see Monitoring.

siglake_wal_ipc_framing_refused_total carries no labels and is pre-registered at 0 on the ingester, the compactor and the query server, so the first refusal is visible to increase() on whichever role read the segment. It covers sealed, legacy and recovered partial segments alike, and sits beside two counters it never overlaps: siglake_wal_crc_mismatch_total for a body that failed its CRC check before any framing was walked, and siglake_wal_partial_tail_dropped_total for a recovered partial whose complete prefix decoded and whose torn tail was discarded. A decode that fails after the framing walk passed increments none of the three. The chart's critical SiglakeWalIpcFramingRefused alert fires on any increase over 15 minutes; see Monitoring.

Drain and commit¶

Layout quality¶

The manifest-walk gauges, siglake_table_level_files, siglake_table_leading_edge_small_files, siglake_table_leading_edge_small_bytes and siglake_table_overlap_depth, share a per-table manifest-walk budget set by SIGLAKE_GAUGE_TABLE_TIMEOUT_SECS (default 20 seconds). The sampler updates them and siglake_table_gauges_sampled_at_seconds only after a complete walk; siglake_table_gauge_table_timeouts_total counts walks that did not finish. siglake_table_live_data_files does not share that staleness: the snapshot summary's total-data-files value is published before the walk. siglake_table_live_data_files_sample_failures_total retains its broader meaning that the sampler sweep did not finish; it does not mean that a published snapshot-summary count was approximate.

Query¶

siglake_query_breaker_trips_total has a breaker label with these values: admission, timeout, shard_timeout, preflight_bytes, pool_exhausted, midflight_rows_shard, midflight_rows_scanned, and midflight_rows_scanned_ndjson, plus jaeger_trace_limit, jaeger_span_rows, jaeger_render_bytes, and jaeger_name_rows. Its priority label is either interactive or batch. The pool_exhausted series covers both memory-pool and spill-cap refusals; the three midflight_rows_* series identify SQL row-ceiling trips, while the four jaeger_* series identify the Jaeger ceiling that refused a read and always have priority="interactive".

Read attempted first: running means the store refused the run's start publication, so the query was dropped unpolled and nothing was executed; any of succeeded, failed and timeout means a completed run's verdict was refused and its computed output discarded. The cause label then makes the conflict actionable:

• cancellation: The run finished just after a client cancellation landed. attempted="succeeded", actual="cancelled" is the common shape, and a low rate is expected anywhere clients cancel.
• recovery: Recovery installed the monotonic failed state before the executor finished. If the executor had not started, its refused running publication prevented the query from running at all; if it had, the computed output was discarded. SiglakeBatchCompletionRejectedByRecovery covers both shapes.
• gone: The row was deleted before the run finished, usually by the job TTL sweep.
• result_too_large: A success body exceeded the job-row limit and was installed as failed. This is the only conflict that also counts in siglake_query_jobs_total, as failed.
• write_deferred: The terminal write never landed, so actual="unknown", but the run's own executor parked the id and retries the same conditional write every five seconds. The row reads running until that pass resolves it, which needs no operator; watch siglake_query_jobs_unreconciled fall back to zero.
• write_abandoned: The terminal write never landed and nothing is retrying it: reconciliation bookkeeping was full, so the row stays non-terminal until this replica exits and lease-expiry recovery on a peer condemns it. actual="unknown" and the answer the run computed is gone. This is the second cause covered by a chart alert, SiglakeBatchRowStrandedNonTerminal.
• other: Another terminal state won without matching one of the cases above; inspect actual and the query-server warning.

Do not put a rate threshold on either counter as a whole: siglake_query_jobs_cancel_propagated_total is a normal-operation counter, and an unlabelled siglake_query_job_terminal_conflict_total rate pages on ordinary client cancellations and TTL expiry. Watch both on the starter dashboard's Batch job cancellation across replicas panel. The chart warns whenever recovery refused executor work, of either shape: increase(siglake_query_job_terminal_conflict_total{cause="recovery"}[10m]) > 0, and pages per pod on the abandoned write (SiglakeBatchRowStrandedNonTerminal, which also reads siglake_query_jobs_unreconciled_dropped_total). Cancellation, TTL expiry and write_deferred do not alert. Seven series are pre-registered so the first conflict of each is visible to increase(): the four cause="recovery" ones (attempted is running, succeeded, failed, or timeout; actual is failed) and the three cause="write_abandoned" ones (attempted is succeeded, failed, or timeout; actual is unknown). Only a finished run's verdict can be deferred or abandoned: a job-store error on the start publication is no conflict at all, and the run executes as before. The unlabelled siglake_query_jobs_unreconciled_dropped_total is pre-registered too, for the other half of that alert. Other causes appear on first occurrence.

Which WARN line to read follows from attempted. A refused completion logs batch completion was refused; computed output was discarded with the job id, owner, attempted outcome, superseding status, cause, and dropped rows and bytes. A refused start logs batch lifecycle transition was refused with phase="running", the job id, owner, superseding status, and whether the row was recovered; it reports no dropped rows, because there were none. A write that never landed logs neither: write_deferred logs the WARN batch terminal state could not be persisted; deferred to owner-local reconciliation, and write_abandoned the ERROR batch terminal state could not be persisted and could not be tracked, both with the job id, owner and attempted verdict. See Batch jobs for completion and start-refusal behavior.

Cache effectiveness¶

SIGLAKE_FOOTER_CACHE_CAP, SIGLAKE_AGG_RESULT_CACHE_CAP and SIGLAKE_DATA_FILE_LIST_CACHE_CAP set the corresponding cache capacities. SIGLAKE_QUERY_RESULT_CACHE controls whole-response caching. Siglake does not define a healthy cache hit rate. Interpret the cache effectiveness metrics against query repetition and snapshot churn; the configuration options change capacity, not a target hit rate.

Cache counters¶

Text-index startup stages¶

siglake_iceberg_text_index_startup_seconds{stage,storage} times what a text query spends on a planned indexed file before that file produces rows. The name ends in _seconds, so it has _bucket series and histogram_quantile() applies. storage is puffin or footer_kv.

Read the sample counts as well as the quantiles. decode's count is the number of cold loads, and selection's count is the number of files the index pruned. So decode climbing toward selection means the plan is not staying warm, and each stage's share of the total says which of the four to chase.

Parsed text-index cache outcomes¶

Both gauges are published where the bounds are enforced, after a successful insert into the cache; an oversized refusal and a zero bound publish nothing. They are absent until the pod's first indexed text query, and they hold their last value while no text query runs. Do not read either one as a liveness signal.

Puffin and footer-KV indexes share this cache under the same bounds, so storage splits both arms of the lookup counter. The three eviction reasons map to the three knobs described in Tune the Puffin and parsed text-index caches.

Puffin blob cache outcomes¶

The Puffin blob cache holds the compressed index bytes a decode reads, one layer below the parsed cache. A parsed miss is what brings a query here, so read these three counters beside siglake_iceberg_parsed_index_cache_lookups_total.

Blob hits climbing with parsed misses is a working set past the parsed budget but inside the blob budget: a re-decode that needed no re-read. A fetch rate that tracks the parsed miss rate is the opposite reading: the refetch shape the coupled eviction rule fixed. A fetch rate above the miss rate is index reading that consulted no cache: a rebuild that bypasses it, or a pod with SIGLAKE_PUFFIN_BLOB_CACHE_MAX_BYTES or SIGLAKE_PUFFIN_BLOB_CACHE_MAX_ENTRIES at 0. All seven series are pre-registered at 0 on the query server, so a tier serving no text query charts zero rather than no data. Both knobs are described in Tune the Puffin and parsed text-index caches.

Footer text-index checksum refusals¶

siglake_index_footer_checksum_refused_total{reason} counts footer-KV inverted indexes the reader refused because the sibling CRC-32 stored beside them did not check out. The check runs before the parsed-cache lookup, so a warm index is covered as well as a cold one. The reader then tries a Puffin index for that column, and scans the file exactly when there is none. A rising rate is correctness holding at the price of scan work.

Both series are pre-registered at 0 on query pods, so the first refusal charts as a step rather than replacing No data. A file written before 0.2.0 carries no sibling checksum, so zero on a fleet mid-upgrade says nothing about the files it has not rewritten. The dashboard graphs both reasons on Footer text-index checksum refusals in the Fast paths row, and ships no alert for them.

Decoded-file cache outcomes¶

siglake_query_scan_file_cache_requests_total{outcome} counts what the experimental decoded-file cache did with each file task a scan opened, and with each population that task started, under the nine outcomes below.

A column comparison, an IN list, IS [NOT] NULL or a prefix LIKE bypasses whenever the Iceberg predicate converter accepts it, including a timestamp range. Order-preserving scans never reach this path, so a newest-first browse records neither outcome. A predicate query is still served by an entry a predicate-free scan left behind, and DataFusion's residual filter keeps that answer exact.

The outcomes do not partition miss. miss is charged before the population stream is built, and hit and bypass never build one, so a stream that fails to construct leaves a miss with no second outcome. A population that went oversized charges skip_oversized and never abandoned, and a population that ended in an error counts as finished rather than abandoned. A population the shared byte bound refuses records population_refused once. The label does not distinguish a full budget from a busy replacement lock. See What the decoded-file cache byte limit covers.

siglake_query_scan_file_cache_accounted_bytes is the total the shared byte bound enforces: completed entries plus admitted live populations. It starts at 0 and updates wherever that total moves. siglake_query_scan_file_cache_bytes reports completed entries only, and siglake_query_scan_file_cache_entries reports their count.

The cache is on only when both SIGLAKE_QUERY_SCAN_FILE_CACHE_MAX_BYTES and SIGLAKE_QUERY_SCAN_FILE_CACHE_MAX_ENTRIES are positive, and the chart, the Compose file and the operator all set them to 0. All nine outcome series are pre-registered on the query server, so a default install reads 0 on every arm rather than absent. The knobs, and the narrow set of scan shapes that fill an entry, are described in Cache tuning.

Decoded-file cache population depth¶

siglake_query_scan_file_cache_populate_rows{outcome} is a histogram with one observation per population of the decoded-file cache: the rows the reader handed that population before it stopped.

The count is cumulative over the stream and is taken before the residual filter above the scan drops any rows. It keeps rising after a candidate crosses the entry bound, so it measures how far the read got, not what the cache kept and not what the query returned.

The buckets carry an edge at 131,071, so +Inf minus le="131071" is the exact count of populations handed 131,072 rows or more, the write path's row-group floor. Reaching that floor does not prove a row group was closed: a file whose groups are larger closes none at that depth, and a read that does not start on a group boundary closes none at any depth.

Unlike the nine counter arms above, this histogram is not pre-registered. It is absent until the cache is on and a population ends, so a default install exports no series for it. The same depth per request is stats.scan.file_cache_populate_rows.

Storage maintenance¶

Attribute auto-promotion¶

Five metrics report what the compactor's attribute auto-promotion pass did to one table's schema. Every series carries iceberg_namespace and table, and a compactor publishes one set per table the pass visits: the events table plus every user index it can list. The gauges refresh on each re-clustering pass; sampling itself runs at most once per 300 seconds per process, so the pass counter moves on that cadence. Auto-promotion is off by default (SIGLAKE_AUTO_PROMOTE_MIN_PCT=0; see Indexing and promotion), and a disabled compactor still publishes the gauges and pre-registers all four pass outcomes at 0.

The restored cost series price sampling separately from the backfill it can trigger. They do not mean that automatic promotion is qualified for an object-store workload.

The attribute names are in the compactor's log, not in a metric. A pass that sampled logs auto-promotion pass finished at INFO with its outcome, the candidate count, the column count before and after, and up to 32 names per disposition: promoted, declined_ceiling, declined_name_collision and declined_mixed_type. Each list carries a _truncated count of the names it left out, so the totals stay exact while one line stays bounded. A table at its cap logs auto-promotion pass declined sampling with candidates=unavailable instead, and a failed pass logs a WARN naming whether the metadata read or the sampling was what broke.

SiglakeAutoPromotionNearCeiling warns when a table with promotion enabled has used at least 80% of its configured cap for 10 minutes. It reads each table against that compactor's own cap, not against a universal column ceiling. See Saturation and What is attribute auto-promotion doing to my schema?.

Deferred text-index registration¶

siglake_index_registration_deferred_total{reason="snapshot_has_statistics"} counts registration attempts that published no index. Iceberg allows one statistics file per snapshot, so a registration that finds the snapshot already carrying one keeps the registered file and drops the blobs it just built. The compactor logs the deferred data-file paths beside each increment. Those files stay readable and answer text predicates by scan, so a rising count costs query time on them, not correctness; they gain an index from a later rewrite or from a CLI index rebuild against a later snapshot. The counter is pre-registered with its one reason, so a healthy process reads 0 rather than absent. The dashboard graphs it by reason over one hour, and the chart ships no alert for it.

Statistics retirement¶

Two counters report the pass that removes Iceberg statistics entries whose Puffin index blobs describe no data file a retained snapshot still references. The elected snapshot-expiry pass runs it, and so does siglake gc-orphans --apply; a dry run classifies the entries and increments nothing. Removal only makes the Puffin object unreachable. The orphan sweep's min_age gate still decides when the object is deleted, and siglake_gc_bytes_reclaimed_total is where its bytes show up. Both counters are pre-registered, so a table with nothing to retire reads 0.

Segmented text-index writes¶

Three metrics report the segmented (seg2) sidecar build a compactor re-cluster runs when SIGLAKE_SEGMENTED_INDEX_WRITES=1. That switch is off by default in 0.2.0; see Indexing and promotion. All four (outcome, reason) series of the counter are pre-registered on every compactor, so a default install reads 0 on each of them rather than absent. The two _bytes families export as summaries, which have nothing to pre-register, so each stays absent until that pod closes its first sidecar. Neither is among the bucketed families: select a quantile label directly instead of calling histogram_quantile().

Segmented text-index reads¶

Four histograms price the store traffic one seg2 lookup makes: the rounds of waits it took, the reads its reader made across them, the ranges the store served, and the bytes those ranges carried. A lookup runs only when SIGLAKE_SEGMENTED_INDEX_READS is on, which it is not by default in 0.2.0 (see Segmented text indexes: the seg2 write and read switches), and only on a scanned file that carries a seg2 sidecar for the predicate's column. All four record one sample per (file, column) lookup that opened a sidecar, whatever that lookup concluded, so a declined file sits in the numbers beside an answered one: a lookup that finds the blob compressed records four zeros, and one refused for row_group_order records nothing at all. All four export as summaries and are not among the bucketed families: select a quantile label instead of calling histogram_quantile(), and expect them absent on a pod that has read no sidecar yet. None has a dashboard row or an alert; they report what the staged reader costs, not a fault.

Segmented text-index lookups: used and declined¶

Two counters and two histograms say what seg2 lookups concluded. A lookup that answers returns a row selection for the file and is counted once at siglake_iceberg_segmented_index_used_total; one that concludes nothing about the file is counted once at siglake_iceberg_segmented_index_declined_total, under the reason it stopped for. The ratio between them is what the read switch buys. No series here is pre-registered, so a pod with SIGLAKE_SEGMENTED_INDEX_READS off, or one whose files carry no seg2 sidecar, exports none of them rather than zeros. The two histograms export as summaries, like the cost families above: select a quantile label rather than calling histogram_quantile().

Why a seg2 lookup declines a file¶

siglake_iceberg_segmented_index_declined_total{reason} carries one of nine reasons. A decline is the safe outcome: the file falls back to its v1 Puffin or footer index for that column, and to a scan when it has none. A LIMIT query that reached seg2 through the clipped path has no v1 fallback and scans the file.

Segmented directory cache¶

The reader holds a parsed seg2 directory between lookups, keyed by sidecar path and blob offset, so a repeat lookup on the same blob skips the trailer and directory stages. SIGLAKE_SEGMENTED_INDEX_DIRECTORY_CACHE_MAX_BYTES sets the process-wide budget, 64 MiB by default; at 0 the cache is off, no lookup is counted and neither gauge is published. This cache is separate from the parsed text-index cache, holds seg2 directories alone, and charges each one its _resident_bytes plus the path it is filed under.

Both gauges are published on admission, so they stay absent until the cache accepts its first directory and hold their last value while no seg2 lookup runs. A refused oversized directory publishes neither. Do not read either one as a liveness signal.

Group-count aggregate labels¶

Five aggregate-maintenance counters name the table they report on with an Iceberg namespace as well as a table name. One compactor maintains the base namespace and every tenant_* namespace, each with its own events, so a bare table="events" merged every tenant onto one series.

The namespace label is iceberg_namespace, not namespace, because Prometheus attaches the Kubernetes namespace under namespace and renames a colliding metric label to exported_namespace. siglake_inline_coverage_unproven carries it for the same reason. All four alerts on these counters report <iceberg_namespace>.<table>. Three of them - SiglakeGroupCountDeltaLost, SiglakeGroupCountAggregateShort and SiglakeSideAggregatePublicationLost - render siglake rebuild-group-counts --namespace <ns> --table <table> with both labels filled in. The fourth, SiglakeGroupCountDeltaRetrying, has no repair command to render: a retry is a transient object-store or credential failure, so it names the pair and the pod label Prometheus adds, and sends you to that pod's warehouse access.

All five counters gained iceberg_namespace in the 0.2.0 line, the retry counter last; before that they carried table alone, and one table name used in several namespaces shared a series. A dashboard or alert that selects on table alone still matches, because a PromQL matcher ignores the labels it does not name, but it now returns one series per namespace where it used to return one in total. Series identity changes with the label: the pre-upgrade series and the per-namespace ones that replace it have separate histories, so rate() over a window that spans the upgrade reports each of them that still holds two samples in the window. A recording rule, a join or a by clause that pairs these counters with another series on an exact label set needs iceberg_namespace added to it; one that aggregates across namespaces on purpose keeps the label out.

Inline aggregate repair¶

Two maintenance paths rewrite a table's inline aggregate object outside the commit path. The compactor re-roots the coverage edge off ancestry that a snapshot expiry is about to drop. You run siglake rebuild-time-aggregates --namespace <ns> --table <table> to recompute the time aggregates from committed files. Both paths are fenced on the object they read, so each counts the writes it published apart from the writes it abandoned. A third path only reads: every 15 minutes (SIGLAKE_INLINE_COVERAGE_SCAN_INTERVAL_SECS) the maintenance compactor censuses each maintained table for an object whose coverage edge no longer reaches the current snapshot, and reports the tables that need the rebuild command. The census is the only signal here with an alert, SiglakeInlineCoverageUnproven.

The two source splits say where a rebuild's work went: a decode file reads column data where a footer file reads metadata, so a decode-heavy run is the slow one. The two cost families put a number on that per component. All four rebuild series record as soon as the pass reads, so an attempt the table committed under records the seconds and bytes it spent before its work was thrown away.

This output is scheduled for Siglake 0.3.0 and is not present in 0.2.x. The command's final stdout line starts with rebuild_cost and ends with one JSON object.

rebuild_cost {"measurement":"complete","publications":1,"conflicts":0,"components":{"time_buckets":{"seconds":0.021,"decoded_bytes":27136,"files":{"footer":0,"decode":16}},"time_group_counts":{"seconds":0.004,"decoded_bytes":196864,"files":{"footer":0,"decode":16}}}}

publications counts writes completed by this invocation. conflicts counts attempts discarded because the table committed before publication. The command makes up to three attempts. Every total accumulates across those attempts, including work later discarded by a conflict.

Each component reports the sum of its attempt durations in seconds. decoded_bytes is the projected Arrow batches' in-memory size, not bytes read from object storage. files.footer counts files answered from Parquet footer metadata. files.decode counts files read from column data because the footer was missing or invalid.

measurement="complete" means the command finished measuring, not that it published a write. An already-covered table is a successful no-op with zero publications, conflicts, seconds, bytes and files. A command error keeps the observations collected before the error, prints measurement="incomplete", then retains its nonzero exit status.

The command installs a recorder for its own invocation and writes the captured totals to stdout. It starts no metrics listener or network exporter, and these observations do not appear on any role's /metrics endpoint. No flag changes that boundary.

The re-root pair and the two census series belong to the maintenance compactor, which does export. Read those as the standing signal for this section; the census never rebuilds anything itself.

Table subscriptions¶

v1 index rebuild cost¶

Three metrics report the post-rewrite v1 inverted-index rebuild, the opt-in pass that reads a committed data file back whole and indexes the columns a rewrite left without one (SIGLAKE_INDEX_REBUILD=1 or compactor.indexRebuild; see Indexing and promotion and Post-compaction index rebuilding). All three carry tenant and table, known only at the increment, so a table nothing has rebuilt is absent rather than 0. Increments land after the registration commits: a pass that rebuilt nothing, or whose registration was deferred, records nothing here.

The pass holds the compaction cycle it runs in, so a p99 near that interval is a compactor whose cycle is the rebuild.

histogram_quantile(0.99, sum by (le, table) (rate(siglake_index_rebuild_seconds_bucket[5m])))

Suggested alerts¶

The chart provides 39 alert rules. Monitoring lists each expression and operator action.

Histogram export¶

The histogram export format is fixed in code. Siglake metrics ending in *_seconds and five named count families use the Prometheus histogram format with bucket series. Other histogram metrics use the Prometheus summary format. You do not configure buckets at scrape time. Scrape the buckets as emitted and use histogram_quantile() only with metrics that expose _bucket series.

Bucketed metric families¶

The inventory below uses histogram for every metric recorded through the histogram API, but not every such metric is exported as a Prometheus histogram. Only names ending in *_seconds and these five count histograms have bucket series:

• siglake_group_count_deltas_folded
• siglake_group_count_tier2_files_per_call
• siglake_compactor_mirror_sync_objects
• siglake_compactor_mirror_sync_rotation_objects
• siglake_query_scan_file_cache_populate_rows

The first four share one per-call count layout. siglake_query_scan_file_cache_populate_rows has its own row-count layout, which places an edge at 131,071 next to one at 131,072 so that the fraction at or above the write path's row-group floor is exact.

Other histogram families, including _bytes, _rows, ratios, and other counts, are exported as summaries with a quantile label and no _bucket series. There is no per-metric switch that changes this export type. Use histogram_quantile() only with the bucketed families above; if a future cross-pod quantile needs another family, that metric first needs an explicit bucket layout in Siglake. The monitoring examples follow this rule: their histogram_quantile() expressions use _seconds_bucket metrics. This export split comes from Siglake's September 3, 2026 Prometheus histogram-bucket correction.

Full inventory¶

A row means the source tree registers the metric, not that your binary emits it. Three rows are behind a cargo feature nothing in the workspace turns on, experimental-exact-point-rollup in siglake-storage: siglake_exact_point_rollup_build_nanoseconds_total, siglake_exact_point_rollup_query_total and siglake_exact_point_boundary_data_files_read_total. A default build never exports those three series.