Metrics reference¶

Each Siglake role exposes Prometheus metrics on a separate /metrics port. This page lists the main signals, labels and export types.

The Helm chart can render a ServiceMonitor with serviceMonitor.enabled and a 33-alert PrometheusRule with prometheusRule.enabled. Both values default to off and require the Prometheus Operator. Monitoring lists the alerts and operator actions. The chart does not render deploy/grafana/siglake-overview.json; import that dashboard separately. See Starter Grafana dashboard for its panels and related alerts.

Selected operational metrics¶

These tables group selected metrics by operational purpose.

Build provenance¶

siglake_build_info exposes the binary's version and source commit labels. These build provenance labels identify the running binary. Confirm that it matches a released version by comparing both labels with the intended release and across every scrape target.

Build information metric¶

Ingest health¶

All six siglake_ingest_tenant_denied_total{reason} series are pre-registered, so the first denial after enabling ingester.oidc.tenantClaim is visible rather than hidden behind an empty graph, and the chart's SiglakeTenantsDenied alert watches the counter. It is the ingest counterpart of siglake_query_tenant_denied_total; see Monitoring.

Drain and commit¶

Layout quality¶

The manifest-walk gauges, siglake_table_level_files, siglake_table_leading_edge_small_files, siglake_table_leading_edge_small_bytes and siglake_table_overlap_depth, share a per-table manifest-walk budget set by SIGLAKE_GAUGE_TABLE_TIMEOUT_SECS (default 20 seconds). The sampler updates them and siglake_table_gauges_sampled_at_seconds only after a complete walk; siglake_table_gauge_table_timeouts_total counts walks that did not finish. siglake_table_live_data_files does not share that staleness: the snapshot summary's total-data-files value is published before the walk. siglake_table_live_data_files_sample_failures_total retains its broader meaning that the sampler sweep did not finish; it does not mean that a published snapshot-summary count was approximate.

Query¶

siglake_query_breaker_trips_total has a breaker label with these values: admission, timeout, shard_timeout, preflight_bytes, pool_exhausted, midflight_rows_shard, midflight_rows_scanned, and midflight_rows_scanned_ndjson, plus jaeger_trace_limit, jaeger_span_rows, jaeger_render_bytes, and jaeger_name_rows. Its priority label is either interactive or batch. The pool_exhausted series covers both memory-pool and spill-cap refusals; the three midflight_rows_* series identify SQL row-ceiling trips, while the four jaeger_* series identify the Jaeger ceiling that refused a read and always have priority="interactive".

Read attempted first: running means the store refused the run's start publication, so the query was dropped unpolled and nothing was executed; any of succeeded, failed and timeout means a completed run's verdict was refused and its computed output discarded. The cause label then makes the conflict actionable:

• cancellation: The run finished just after a client cancellation landed. attempted="succeeded", actual="cancelled" is the common shape, and a low rate is expected anywhere clients cancel.
• recovery: Recovery installed the monotonic failed state before the executor finished. If the executor had not started, its refused running publication prevented the query from running at all; if it had, the computed output was discarded. SiglakeBatchCompletionRejectedByRecovery covers both shapes.
• gone: The row was deleted before the run finished, usually by the job TTL sweep.
• result_too_large: A success body exceeded the job-row limit and was installed as failed. This is the only conflict that also counts in siglake_query_jobs_total, as failed.
• write_deferred: The terminal write never landed, so actual="unknown", but the run's own executor parked the id and retries the same conditional write every five seconds. The row reads running until that pass resolves it, which needs no operator; watch siglake_query_jobs_unreconciled fall back to zero.
• write_abandoned: The terminal write never landed and nothing is retrying it: reconciliation bookkeeping was full, so the row stays non-terminal until this replica exits and lease-expiry recovery on a peer condemns it. actual="unknown" and the answer the run computed is gone. This is the second cause covered by a chart alert, SiglakeBatchRowStrandedNonTerminal.
• other: Another terminal state won without matching one of the cases above; inspect actual and the query-server warning.

Do not put a rate threshold on either counter as a whole: siglake_query_jobs_cancel_propagated_total is a normal-operation counter, and an unlabelled siglake_query_job_terminal_conflict_total rate pages on ordinary client cancellations and TTL expiry. Watch both on the starter dashboard's Batch job cancellation across replicas panel. The chart warns whenever recovery refused executor work, of either shape: increase(siglake_query_job_terminal_conflict_total{cause="recovery"}[10m]) > 0, and pages per pod on the abandoned write (SiglakeBatchRowStrandedNonTerminal, which also reads siglake_query_jobs_unreconciled_dropped_total). Cancellation, TTL expiry and write_deferred do not alert. Seven series are pre-registered so the first conflict of each is visible to increase(): the four cause="recovery" ones (attempted is running, succeeded, failed, or timeout; actual is failed) and the three cause="write_abandoned" ones (attempted is succeeded, failed, or timeout; actual is unknown). Only a finished run's verdict can be deferred or abandoned: a job-store error on the start publication is no conflict at all, and the run executes as before. The unlabelled siglake_query_jobs_unreconciled_dropped_total is pre-registered too, for the other half of that alert. Other causes appear on first occurrence.

Which WARN line to read follows from attempted. A refused completion logs batch completion was refused; computed output was discarded with the job id, owner, attempted outcome, superseding status, cause, and dropped rows and bytes. A refused start logs batch lifecycle transition was refused with phase="running", the job id, owner, superseding status, and whether the row was recovered; it reports no dropped rows, because there were none. A write that never landed logs neither: write_deferred logs the WARN batch terminal state could not be persisted; deferred to owner-local reconciliation, and write_abandoned the ERROR batch terminal state could not be persisted and could not be tracked, both with the job id, owner and attempted verdict. See Batch jobs for completion and start-refusal behavior.

Cache effectiveness¶

SIGLAKE_FOOTER_CACHE_CAP, SIGLAKE_AGG_RESULT_CACHE_CAP and SIGLAKE_DATA_FILE_LIST_CACHE_CAP set the corresponding cache capacities. SIGLAKE_QUERY_RESULT_CACHE controls whole-response caching. Siglake does not define a healthy cache hit rate. Interpret the cache effectiveness metrics against query repetition and snapshot churn; the configuration options change capacity, not a target hit rate.

Cache counters¶

Storage maintenance¶

Table subscriptions¶

Suggested alerts¶

The chart provides 33 alert rules. Monitoring lists each expression and operator action.

Histogram export¶

The histogram export format is fixed in code. Siglake metrics ending in *_seconds and four named count families use the Prometheus histogram format with bucket series. Other histogram metrics use the Prometheus summary format. You do not configure buckets at scrape time. Scrape the buckets as emitted and use histogram_quantile() only with metrics that expose _bucket series.

Bucketed metric families¶

The inventory below uses histogram for every metric recorded through the histogram API, but not every such metric is exported as a Prometheus histogram. Only names ending in *_seconds and these four per-call count histograms have bucket series:

• siglake_group_count_deltas_folded
• siglake_group_count_tier2_files_per_call
• siglake_compactor_mirror_sync_objects
• siglake_compactor_mirror_sync_rotation_objects

Other histogram families, including _bytes, _rows, ratios, and other counts, are exported as summaries with a quantile label and no _bucket series. There is no per-metric switch that changes this export type. Use histogram_quantile() only with the bucketed families above; if a future cross-pod quantile needs another family, that metric first needs an explicit bucket layout in Siglake. The monitoring examples follow this rule: their histogram_quantile() expressions use _seconds_bucket metrics. This export split comes from Siglake's September 3, 2026 Prometheus histogram-bucket correction.

Full inventory¶