Monitoring¶

Configure Prometheus to scrape every role, install the chart's alert rules, then work from the question that matches the incident. Most investigations reduce to four: is ingest healthy, is the drain keeping up, is the layout converging, and why is a query slow. The metrics reference lists every metric and the smaller set you need for routine operations.

Wiring it up¶

serviceMonitor:
  enabled: true      # Prometheus Operator
prometheusRule:
  enabled: true      # 33 alert rules; off by default
  labels:
    release: kube-prometheus-stack

Both objects require the Prometheus Operator. Set prometheusRule.labels to the labels your Prometheus installation uses for rule discovery. The chart only installs alert definitions; Prometheus and Alertmanager remain responsible for evaluation, routing, delivery, acknowledgement, and the on-call workflow.

Or scrape directly:

Starter Grafana dashboard¶

The deploy/grafana/siglake-overview.json file is a starter dashboard that you import by hand; the chart does not render it.

No panel alerts or pages anyone. Only the chart's PrometheusRule does that, so use the alert table below to configure what pages.

The rows and panels worth knowing before you need them:

The cause label on siglake_query_job_terminal_conflict_total is the one that needs reading rather than watching:

• attempted="running" is a start the store refused, so the query never executed. succeeded, failed and timeout are refused completions whose output was discarded.
• Cancellation and gone are expected races.
• recovery means recovery refused executor work of either shape.
• write_deferred and write_abandoned are terminal writes that never landed. The first is being retried by its own executor; the second is not.
• result_too_large is an applied failed state.

The four cause="recovery" and three cause="write_abandoned" series are pre-registered, because the chart's alerts read them. Other causes appear on first occurrence; see Query metrics.

The local Compose stack ships Prometheus at http://localhost:9090 with deploy/prometheus.yml already scraping every role.

Query server probes¶

On the query server, /healthz is a process-liveness check that returns a constant 200, and /readyz only round-trips the Iceberg catalog. Neither probe executes a query or checks the cache-warm loop, so keep SiglakeQueryWarmCycleStalled enabled: it is the shipped signal for the known case where browse requests time out while both probes stay green. Do not treat successful probes as end-to-end query-path health. See Query probes do not detect query-path degradation for why Siglake does not automatically restart or remove the pod from service. This behavior comes from Siglake's September 3, 2026 review of query degradation and probe behavior.

Which build is running?¶

Every long-running service publishes siglake_build_info with version and commit labels. In Kubernetes, list the build scraped from each pod with:

max by (namespace, pod, version, commit) (siglake_build_info)

Each series has a value of 1; its labels are the useful part. Different version or commit values make a mixed-version fleet visible before any behavior differs between pods.

Is ingest healthy?¶

rate(siglake_events_accepted_total[5m])
rate(siglake_ingest_backpressure_rejected_total[5m])
siglake_ingest_backpressure_queue_events

Rejections mean Siglake is shedding client requests. Compare them with queue depth. If both rise, the writer cannot keep up, so add ingester CPU or replicas. If the queue stays shallow, a burst filled the lane before it could drain.

siglake_wal_crc_mismatch_total should always be zero.

Is the drain keeping up, and is the layout converging?¶

siglake_compactor_sealed_pending_oldest_age_seconds and SiglakeDrainBacklogGrowing tell you whether the WAL drain is keeping up. siglake_table_overlap_depth and SiglakeTableNotConverging tell you whether the layout is converging. The supporting metrics identify why either stage is behind, and the chart ships every alert named here.

Is the WAL drain keeping up?¶

Is the layout converging?¶

The drain commits sealed WAL segments to Iceberg. Compaction then merges what the drain wrote into a layout queries can scan. If the drain falls behind, rows are durable but not yet queryable. If the layout stops converging, the rows are queryable and scans get slower as depth grows. More compactor replicas clear a drain backlog. They do not help a table whose merge passes keep aborting at their deadline.

Read the drain signals¶

siglake_compactor_sealed_pending_oldest_age_seconds
sum by (namespace) (
  max by (namespace, tenant) (
    siglake_compactor_sealed_pending
  )
)
siglake_compactor_drain_inflight
histogram_quantile(0.95, rate(siglake_compactor_commit_duration_seconds_bucket[5m]))
increase(siglake_compactor_watchdog_trips_total[30m])
siglake_catalog_claim_quarantined

With two or more catalog-claim workers, every worker publishes the same queue total. Deduplicate the copies with max by (namespace, tenant) before you sum tenants. Dashboard panel 111 does this for both siglake_compactor_sealed_pending and siglake_compactor_sealed_pending_bytes, then sums each result by namespace.

If the oldest age climbs and sealed_pending grows with it, ingest is running faster than the drain: rows are durable but not yet queryable, and the WAL volume is filling. Investigate commit duration or add compactor replicas with catalogClaim.enabled.

Rising commit duration with flat throughput usually means catalog pressure: check RDS.

A segment the drain cannot commit at all is quarantined rather than retried forever, which SiglakeSegmentsQuarantined reports; its row under Stalled carries the triage.

WAL segments from a deleted index¶

Delete an index and recreate it under the same name, and the drain can still find WAL segments written for the old table. It refuses to commit those rows as the replacement's, and counts each refusal:

Both are refusal events rather than a count of distinct objects held back. A filesystem segment whose move under stale/ fails stays in sealed/ and is counted again on the next cycle that examines it, and requeueing a quarantined mirror claim runs the identity check again, which produces another refusal. So read the hourly increase as an event rate, and do not read the cumulative total as a backlog depth. The metrics reference carries both descriptions.

The WAL owner mismatches (dropped incarnation held back) panel graphs both series in the starter dashboard's Drain and compaction row. It plots the hourly increase by tenant and index alongside both cumulative totals, because a series that appears with its first increment has no rate to read yet.

Neither counter has an alert, and that is deliberate. A refused mirrored object ends up as a quarantined claim, which SiglakeSegmentsQuarantined and the Segments quarantined panel already report.

Resolve the identity mismatch before you try to recover the rows. On the mirror side, requeueing alone does not repair it: follow the SiglakeSegmentsQuarantined row, which covers what the requeue does and does not adopt. On the filesystem side the refused segments stay visible under stale/<dropped-uuid>/ in the index's WAL directory. Nothing on either path is deleted, so recovering or discarding those rows stays your decision.

Delete tasks that stay non-terminal¶

Delete-task observation runs as part of the compactor's delete sweep:

The claim object's age is an upper bound on execution duration, not a running duration. Neither metric says that an executor died or whether its rewrite committed. The state set is exactly running and pending_claimed.

Read the layout signals¶

siglake_table_overlap_depth
siglake_table_live_data_files
siglake_table_gauges_sampled_at_seconds
rate(siglake_compactor_maintenance_throttled_backpressure_total[5m])

Use overlap_depth as the layout-health metric. A converged 2 B-row table settles in the low twenties. A flat plateau over hours means compaction is not converging, and query performance is silently degraded: an unconverged 1 TB layout collapsed 32-way query throughput from 115 QPS to 28. Because this gauge comes from the budgeted manifest walk, it can lag; first confirm that siglake_table_gauges_sampled_at_seconds is advancing. The same freshness signal covers the level and leading-edge gauges and changes only when a table's walk completes.

live_data_files growing without bound means compaction is losing to ingest. Unlike the walked gauges, it is read exactly from the snapshot summary every sampler cycle, regardless of table size. The throttled-backpressure counter tells you compaction is deliberately yielding. That is expected under load, but it is a problem if it never stops.

Why is a query slow?¶

Start with the response. Every response carries cost, stats.scan and stats.phases. See Query engine.

Then, cluster-wide:

histogram_quantile(0.95, rate(siglake_query_request_duration_seconds_bucket[5m]))
siglake_query_exec_pool_queue_seconds
siglake_query_in_flight
rate(siglake_query_scan_output_ordering_total[5m])

Query peer discovery and table-cache publication¶

When SRV peer discovery is enabled, each refresh increments siglake_query_peer_discovery_refresh_total. The changed and unchanged outcomes mean the pod resolved a usable membership; empty, unmatched, and error are failures. A failed refresh keeps the last good membership, which protects in-flight queries from a DNS blip but means a pod broken since startup can keep answering every query locally without an obvious request failure. siglake_query_peer_discovery_members shows the published member count, and siglake_query_peer_discovery_last_success_seconds records the last usable answer.

SiglakeQueryPeerDiscoveryStalled is the warning for a pod cut off from its membership. One failed refresh is normal during a rollout. The trigger instead means that refreshes are being attempted and none are succeeding. The failing outcomes increase over a ten-minute window while neither changed nor unchanged increases, then the rule holds for ten minutes. Failures alongside successes are a cluster in churn or a partially answering resolver; that pod still has a membership, so the rule deliberately stays quiet. With the default --query-peer-discovery-interval-secs of 5, ten minutes without one usable answer is roughly 120 consecutive attempts.

Split the refresh counter by outcome to act on it: for error, check CoreDNS, NetworkPolicy, and that --query-peer-discovery-srv names the headless Service's http port; for empty, check that the Service has Ready endpoints; for unmatched, compare SIGLAKE_QUERY_PEER_SELF_NAME with the SRV targets. A pod that cannot find its own name in the answer is also left without a failover target.

What the stall costs fan-out depends on whether the pod ever published a membership. One that has not answers every query itself, single-pod, using none of the query tier's other replicas, and no query-side counter marks it as degraded. One that published earlier keeps serving its last good membership, so its fan-out is stale. It will not see replicas added since, and can briefly address a member that has gone away. Same-shard failover turns that into a latency event instead of a wrong answer.

The query server and compactor also count table-metadata cache publication races in siglake_iceberg_table_cache_fenced_total. Its pre-registered action="superseded" and action="reload" series are healthy contention under concurrent commits. action="unpublished" means the reload exhausted its attempts, left that table's cache entry empty, and made later reads pay for a full load_table.

SiglakeTableCacheUnpublished pages only when unpublished continues, rather than on the healthy arms. If superseded or reload is moving too, reduce the commit rate by raising compactor.commitBatch.targetMb or maxAgeSecs. If metadata loads themselves are slow, shrink metadata.json by lowering compactor.snapshotExpire.retainLast, or run snapshot expiry more frequently by lowering its intervalSecs.

External consumer health¶

Processes built on the WAL consumer interface must export their own lag, error, and result telemetry. Their alert names depend on that separate deployment and are not part of Siglake's metric inventory.

Disaster-recovery readiness¶

rate(siglake_wal_mirror_failures_total[5m])
rate(siglake_wal_mirror_segments_total[5m])
siglake_wal_mirror_queue_depth
rate(siglake_wal_mirror_queue_wait_seconds_sum[5m])
  / rate(siglake_wal_mirror_queue_wait_seconds_count[5m])

Mirror failures move your recovery point further back without affecting ingest. If mirroring is enabled, alert on any failure and restore uploads before the WAL volume becomes the only copy.

siglake_wal_mirror_queue_depth counts in-memory segments waiting for the uploader. It excludes the active upload and does not inventory durable mirror-pending/ pins. A stuck active upload can therefore coincide with a queue depth of zero.

siglake_wal_mirror_queue_wait_seconds records each segment's time from enqueue to dequeue. It does not measure upload duration or the current oldest pending segment's age. Keep the failure alert and WAL volume free-space monitoring alongside both queue signals. A prolonged remote failure retains successfully pinned bytes across local compaction and retention, which can exhaust the volume.

The ingester checks sealed/ and mirror-pending/ at startup and every 300 seconds to resume uploads after an outage or process exit. Set SIGLAKE_WAL_MIRROR_SWEEP_SECS=0 to disable both checks. A failed pin appears as siglake_wal_mirror_failures_total{reason="pin"} and does not carry the retention protection described above.

Suggested alerts¶

The chart's PrometheusRule is the authoritative alert set. Siglake CI checks that every rule uses a metric exported by the build. The groups describe the operator action, not the component that emits the metric.

Alert counters whose label sets are known at startup are pre-registered at zero, so Prometheus increase() can observe the first event after a fresh pod starts. The exceptions have labels that are discovered only when the event occurs: siglake_storage_schema_drift_total{column=...} and the index-table series for siglake_group_count_delta_write_failures_total, siglake_side_aggregate_publish_failures_total and siglake_group_count_auto_rebuilds_total. The events-table series are pre-registered, including the automatic-rebuild counter's success, incomplete, and failed outcomes. For a dynamically discovered series, the first increment is not visible to increase(); the schema-drift alert therefore fires on the second refusal, which the next drain cycle produces.

Data loss or durable inconsistency¶

Page on the critical rules in this group. The warning rules expose precursors or recoverable inconsistencies that still need investigation.

Stalled¶

Refusing work¶

Saturation¶

Logs¶

RUST_LOG controls filtering; the chart's logLevel default is info,siglake=info. For debugging, info,siglake=debug.

Container runtimes capture the server roles' tracing logs from stderr; the server roles write nothing operational to stdout.

SIGLAKE_LOG_EXEC_PLANS logs physical query plans: very verbose, useful when a fast path isn't engaging and you need to see what DataFusion actually built.

Query audit as an observability surface¶

query_audit is an ordinary Iceberg table, so your query history is queryable:

-- most expensive query shapes in the last day
SELECT query, count(*) AS runs, avg(duration_ms) AS avg_ms,
       max(estimated_bytes_scanned) AS max_bytes
FROM query_audit
WHERE timestamp >= now() - INTERVAL '24 hours'
  AND complexity IN ('large', 'huge')
GROUP BY query
ORDER BY avg_ms DESC
LIMIT 20;

-- rejections and errors
SELECT status, count(*) FROM query_audit
WHERE timestamp >= now() - INTERVAL '1 hour'
GROUP BY status;

The table contains literal SQL text. Treat it as sensitive and rotate it with siglake audit-rotate --max-age-secs N.