Monitoring¶

Configure Prometheus to scrape every role, install the chart's alert rules, then work from the question that matches the incident. Most investigations reduce to four: is ingest healthy, is the drain keeping up, is the layout converging, and why is a query slow. The metrics reference lists every metric and the smaller set you need for routine operations.

Wiring it up¶

serviceMonitor:
  enabled: true      # Prometheus Operator
prometheusRule:
  enabled: true      # 39 alert rules; off by default
  labels:
    release: kube-prometheus-stack

Both objects require the Prometheus Operator. Set prometheusRule.labels to the labels your Prometheus installation uses for rule discovery. The chart only installs alert definitions; Prometheus and Alertmanager remain responsible for evaluation, routing, delivery, acknowledgement, and the on-call workflow.

Or scrape directly:

The metrics port checks no token of its own. Bearer tokens and OIDC guard the query API on 8089, not this listener, and --metrics-bind defaults to 0.0.0.0, so each role answers /metrics on every interface its pod has. The chart's networkPolicy.enabled renders egress rules only, so nothing shipped restricts who may reach 9100, 9101 or 9105: reachability is whatever your cluster allows between pods. Keep the port off any Ingress or LoadBalancer, and if your cluster's default is open, write the ingress restriction yourself. See Network policies.

Metrics are Prometheus only: no Siglake process exports them as OTLP, whatever its OTel configuration says. An OTLP-only backend gets them from a collector with a Prometheus receiver scraping these ports, which Siglake's own telemetry configures. That page also covers the logs and traces Siglake exports about itself, which are not on this listener.

Starter Grafana dashboard¶

The deploy/grafana/siglake-overview.json file is a starter dashboard that you import by hand; the chart does not render it.

No panel alerts or pages anyone. Only the chart's PrometheusRule does that, so use the alert table below to configure what pages.

The rows and panels worth knowing before you need them:

The cause label on siglake_query_job_terminal_conflict_total is the one that needs reading rather than watching:

• attempted="running" is a start the store refused, so the query never executed. succeeded, failed and timeout are refused completions whose output was discarded.
• Cancellation and gone are expected races.
• recovery means recovery refused executor work of either shape.
• write_deferred and write_abandoned are terminal writes that never landed. The first is being retried by its own executor; the second is not.
• result_too_large is an applied failed state.

The four cause="recovery" and three cause="write_abandoned" series are pre-registered, because the chart's alerts read them. Other causes appear on first occurrence; see Query metrics.

The local Compose stack ships Prometheus at http://localhost:9090 with deploy/prometheus.yml already scraping every role.

Query server probes¶

On the query server, /healthz is a process-liveness check that returns a constant 200, and /readyz only round-trips the Iceberg catalog. Neither probe executes a query or checks the cache-warm loop, so keep SiglakeQueryWarmCycleStalled enabled: it is the shipped signal for the known case where browse requests time out while both probes stay green. Do not treat successful probes as end-to-end query-path health. See Query probes do not detect query-path degradation for why Siglake does not automatically restart or remove the pod from service. This behavior comes from Siglake's September 3, 2026 review of query degradation and probe behavior.

Which build is running?¶

Every long-running service publishes siglake_build_info with version and commit labels. In Kubernetes, list the build scraped from each pod with:

max by (namespace, pod, version, commit) (siglake_build_info)

Each series has a value of 1; its labels are the useful part. Different version or commit values make a mixed-version fleet visible before any behavior differs between pods.

Is ingest healthy?¶

rate(siglake_events_accepted_total[5m])
rate(siglake_ingest_backpressure_rejected_total[5m])
siglake_ingest_backpressure_queue_events

Rejections mean Siglake is shedding client requests. Compare them with queue depth. If both rise, the writer cannot keep up, so add ingester CPU or replicas. If the queue stays shallow, a burst filled the lane before it could drain.

siglake_wal_crc_mismatch_total and siglake_wal_ipc_framing_refused_total should always be zero. The first counts segment bodies that failed their CRC check; the second counts WAL reads refused because the Arrow IPC length framing does not describe a message the segment's bytes can back.

Is the drain keeping up, and is the layout converging?¶

siglake_compactor_sealed_pending_oldest_age_seconds and SiglakeDrainBacklogGrowing tell you whether the WAL drain is keeping up. siglake_table_overlap_depth and SiglakeTableNotConverging tell you whether the layout is converging. The supporting metrics identify why either stage is behind, and the chart ships every alert named here.

Is the WAL drain keeping up?¶

Is the layout converging?¶

The drain commits sealed WAL segments to Iceberg. Compaction then merges what the drain wrote into a layout queries can scan. If the drain falls behind, rows are durable but not yet queryable. If the layout stops converging, the rows are queryable and scans get slower as depth grows. More compactor replicas clear a drain backlog. They do not help a table whose merge passes keep aborting at their deadline.

Read the drain signals¶

siglake_compactor_sealed_pending_oldest_age_seconds
sum by (namespace) (
  max by (namespace, tenant) (
    siglake_compactor_sealed_pending
  )
)
siglake_compactor_drain_inflight
histogram_quantile(0.95, rate(siglake_compactor_commit_duration_seconds_bucket[5m]))
increase(siglake_compactor_watchdog_trips_total[30m])
siglake_catalog_claim_quarantined
siglake_compactor_segments_poisoned

With two or more catalog-claim workers, every worker publishes the same queue total. Deduplicate the copies with max by (namespace, tenant) before you sum tenants. Dashboard panel 111 does this for both siglake_compactor_sealed_pending and siglake_compactor_sealed_pending_bytes, then sums each result by namespace.

If the oldest age climbs and sealed_pending grows with it, ingest is running faster than the drain: rows are durable but not yet queryable, and the WAL volume is filling. Investigate commit duration or add compactor replicas with catalogClaim.enabled.

Rising commit duration with flat throughput usually means catalog pressure: check RDS.

A segment the drain cannot commit at all is set aside rather than retried forever, and SiglakeSegmentsQuarantined reports both drains doing it: the claim table quarantines the segment, and the local filesystem drain moves the file under <wal>/poison/. Its row under Stalled carries the triage for each.

WAL segments from a deleted index¶

Delete an index and recreate it under the same name, and the drain can still find WAL segments written for the old table. It refuses to commit those rows as the replacement's, and counts each refusal:

Both are refusal events rather than a count of distinct objects held back. A filesystem segment whose move under stale/ fails stays in sealed/ and is counted again on the next cycle that examines it, and requeueing a quarantined mirror claim runs the identity check again, which produces another refusal. So read the hourly increase as an event rate, and do not read the cumulative total as a backlog depth. The metrics reference carries both descriptions.

The WAL owner mismatches (dropped incarnation held back) panel graphs both series in the starter dashboard's Drain and compaction row. It plots the hourly increase by tenant and index alongside both cumulative totals, because a series that appears with its first increment has no rate to read yet.

Neither counter has an alert, and that is deliberate. A refused mirrored object ends up as a quarantined claim, which SiglakeSegmentsQuarantined and the Segments quarantined panel already report.

Resolve the identity mismatch before you try to recover the rows. On the mirror side, requeueing alone does not repair it: follow the SiglakeSegmentsQuarantined row, which covers what the requeue does and does not adopt. On the filesystem side the refused segments stay visible under stale/<dropped-uuid>/ in the index's WAL directory. Nothing on either path is deleted, so recovering or discarding those rows stays your decision.

Delete tasks that stay non-terminal¶

Delete-task observation runs as part of the compactor's delete sweep:

The claim object's age is an upper bound on execution duration, not a running duration. Neither metric says that an executor died or whether its rewrite committed. The state set is exactly running and pending_claimed.

Read the layout signals¶

siglake_table_overlap_depth
siglake_table_live_data_files
siglake_table_gauges_sampled_at_seconds
rate(siglake_compactor_maintenance_throttled_backpressure_total[5m])

Use overlap_depth as the layout-health metric. A converged 2 B-row table settles in the low twenties. A flat plateau over hours means compaction is not converging, and query performance is silently degraded: an unconverged 1 TB layout collapsed 32-way query throughput from 115 QPS to 28. Because this gauge comes from the budgeted manifest walk, it can lag; first confirm that siglake_table_gauges_sampled_at_seconds is advancing. The same freshness signal covers the level and leading-edge gauges and changes only when a table's walk completes.

live_data_files growing without bound means compaction is losing to ingest. Unlike the walked gauges, it is read exactly from the snapshot summary every sampler cycle, regardless of table size. The throttled-backpressure counter tells you compaction is deliberately yielding. That is expected under load, but it is a problem if it never stops.

What is attribute auto-promotion doing to my schema?¶

Auto-promotion is off by default. Once SIGLAKE_AUTO_PROMOTE_MIN_PCT is above zero, the compactor widens table schemas from a sample with no operator in the loop, and that widening cannot be undone, so watch the column count and what each pass decided.

siglake_auto_promotion_enabled
sum by (iceberg_namespace, table, outcome) (increase(siglake_auto_promotion_passes_total[1h]))
siglake_auto_promotion_columns{kind="used"} / ignoring (kind) siglake_auto_promotion_columns{kind="limit"}
siglake_auto_promotion_candidates and on (namespace, pod, iceberg_namespace, table) siglake_auto_promotion_candidates_available == 1

Names are in the log: auto-promotion pass finished at INFO carries up to 32 attribute names per disposition, promoted and declined, with a _truncated count for the rest. The dashboard's Auto-promotion panel graphs these four families, and SiglakeAutoPromotionNearCeiling warns at 80% of the cap. See Attribute auto-promotion for each metric and Declared vs automatic promotion for what a promotion costs.

Why is a query slow?¶

Start with the response. Every response carries cost, stats.scan and stats.phases. See Query engine.

Then, cluster-wide:

histogram_quantile(0.95, rate(siglake_query_request_duration_seconds_bucket[5m]))
siglake_query_exec_pool_queue_seconds
siglake_query_in_flight
rate(siglake_query_scan_output_ordering_total[5m])

Footer text-index checksum refusals¶

siglake_index_footer_checksum_refused_total{reason} above zero means a query found a footer-KV inverted index that did not match the sibling CRC-32 stored with it, and refused to prune with it. Answers stay exact: the reader tries the column's Puffin index and otherwise scans the file. The cost is pruning on that file, and it stays until something rewrites the file.

Read the two reasons differently. mismatch is the corruption case: the blob in the Parquet footer no longer hashes to the value written beside it. Any rewrite of that file, a compactor re-cluster or a delete-task rewrite, replaces the index it carries, and that is what clears the refusal. malformed is a value the reader cannot parse at all: a checksum that is not eight hexadecimal characters, or a blob that is not even-length hexadecimal. The Siglake writer emits neither shape, so it is damage that landed on the characters themselves, or a footer some other tool rewrote.

Neither series carries a file label and no log line names the refused path, so the metric says that this is happening and not where. The rate follows query traffic rather than the number of damaged files: one file read by ten queries counts ten times. The Footer text-index checksum refusals panel in the Fast paths row graphs it by reason, and ships no alert.

Query peer discovery and table-cache publication¶

When SRV peer discovery is enabled, each refresh increments siglake_query_peer_discovery_refresh_total. The changed and unchanged outcomes mean the pod resolved a usable membership; empty, unmatched, and error are failures. A failed refresh keeps the last good membership, which protects in-flight queries from a DNS blip but means a pod broken since startup can keep answering every query locally without an obvious request failure. siglake_query_peer_discovery_members shows the published member count, and siglake_query_peer_discovery_last_success_seconds records the last usable answer.

SiglakeQueryPeerDiscoveryStalled is the warning for a pod cut off from its membership. One failed refresh is normal during a rollout. The trigger instead means that refreshes are being attempted and none are succeeding. The failing outcomes increase over a ten-minute window while neither changed nor unchanged increases, then the rule holds for ten minutes. Failures alongside successes are a cluster in churn or a partially answering resolver; that pod still has a membership, so the rule deliberately stays quiet. With the default --query-peer-discovery-interval-secs of 5, ten minutes without one usable answer is roughly 120 consecutive attempts.

Split the refresh counter by outcome to act on it: for error, check CoreDNS, NetworkPolicy, and that --query-peer-discovery-srv names the headless Service's http port; for empty, check that the Service has Ready endpoints; for unmatched, compare SIGLAKE_QUERY_PEER_SELF_NAME with the SRV targets. A pod that cannot find its own name in the answer is also left without a failover target.

What the stall costs fan-out depends on whether the pod ever published a membership. One that has not answers every query itself, single-pod, using none of the query tier's other replicas, and no query-side counter marks it as degraded. One that published earlier keeps serving its last good membership, so its fan-out is stale. It will not see replicas added since, and can briefly address a member that has gone away. Same-shard failover turns that into a latency event instead of a wrong answer.

The query server and compactor also count table-metadata cache publication races in siglake_iceberg_table_cache_fenced_total. Its pre-registered action="superseded" and action="reload" series are healthy contention under concurrent commits. action="unpublished" means the reload exhausted its attempts, left that table's cache entry empty, and made later reads pay for a full load_table.

SiglakeTableCacheUnpublished pages only when unpublished continues, rather than on the healthy arms. If superseded or reload is moving too, reduce the commit rate by raising compactor.commitBatch.targetMb or maxAgeSecs. If metadata loads themselves are slow, shrink metadata.json by lowering compactor.snapshotExpire.retainLast, or run snapshot expiry more frequently by lowering its intervalSecs.

External consumer health¶

Processes built on the WAL consumer interface must export their own lag, error, and result telemetry. Their alert names depend on that separate deployment and are not part of Siglake's metric inventory.

Disaster-recovery readiness¶

rate(siglake_wal_mirror_failures_total[5m])
rate(siglake_wal_mirror_segments_total[5m])
siglake_wal_mirror_queue_depth
rate(siglake_wal_mirror_queue_wait_seconds_sum[5m])
  / rate(siglake_wal_mirror_queue_wait_seconds_count[5m])
sum(rate(siglake_wal_mirror_pin_duration_seconds_sum[5m]))
  / sum(rate(siglake_wal_mirror_pin_duration_seconds_count[5m]))
histogram_quantile(0.99,
  sum by (le) (rate(siglake_wal_mirror_pin_duration_seconds_bucket[5m])))

Mirror failures move your recovery point further back without affecting ingest. If mirroring is enabled, alert on any failure and restore uploads before the WAL volume becomes the only copy.

siglake_wal_mirror_queue_depth counts in-memory segments waiting for the uploader. It excludes the active upload and does not inventory durable mirror-pending/ pins. A stuck active upload can therefore coincide with a queue depth of zero.

siglake_wal_mirror_queue_wait_seconds records each segment's time from enqueue to dequeue. It does not measure upload duration or the current oldest pending segment's age. Keep the failure alert and WAL volume free-space monitoring alongside both queue signals. A prolonged remote failure retains successfully pinned bytes across local compaction and retention, which can exhaust the volume.

siglake_wal_mirror_pin_duration_seconds records each seal's whole synchronous pin_segment call, whether the call succeeds or fails. Its first four buckets are below 1 ms, where healthy pins commonly finish. The average and p99 show how long sealing waits, but they do not attribute time to local lookup retries, hard-link creation, or the mirror-pending/ directory fsync.

The ingester checks sealed/ and mirror-pending/ at startup and every 300 seconds to resume uploads after an outage or process exit. Set SIGLAKE_WAL_MIRROR_SWEEP_SECS=0 to disable both checks. A failed pin appears as siglake_wal_mirror_failures_total{reason="pin"} and does not carry the retention protection described above.

Two reasons on the same counter report leaked storage rather than a worse recovery point. reason="active_cleanup" means three deletes of a sealed segment's _active/ partial failed, and reason="active_cleanup_stat" means an active upload could not check whether its segment had sealed. The sealed segment is uploaded and registered in both cases. The next catch-up sweep retries the delete while the local segment is still on the WAL volume; after that, only an object-store lifecycle expiry removes the partial. Treat a rising rate as a storage-growth signal on the mirror prefix, not as a lost recovery point.

Unreadable objects found by a wal-recover run¶

siglake wal-recover reports mirror damage in its printed output and in log events, and exports no metric for it. The command is a one-shot subcommand that installs no metrics recorder and binds no /metrics, so a counter it charged would be discarded when the process exits, and no row in the metric inventory covers a recovery run. What you read instead is the counts on the printed report and one WARN event per refused object and unrecognised name. Unreadable reporting arrives in Siglake 0.2.0, with the plan-first wal-recover.

  UNREADABLE: `default/siglake-ingester-0-01a0afc1-2901-7cc2-8b41-6f5f0f4fd83e.arrow` is not a WAL segment (WAL frame integrity check: <object-store segment> body length 972 != header 625); it will be left in the mirror
  totals: 467 segments, 3.5 GiB (1 unreadable: body does not decode, 17 keys skipped: unrecognised layout)

One UNREADABLE: line prints per refused object, capped at ten, and the count lands on totals:. Any nonzero count is rows you cannot restore from the mirror. The refused objects stay where they are and the local drain never sees them, so the count comes back every time you plan. For the disposition, and for what separates an unreadable candidate from a skipped name, read What an UNREADABLE: line in the plan means.

Each refused object and each unrecognised name also emits a WARN event carrying the object name and, for a body that did not decode, the error. These are the three messages a log rule matches on:

The events go to stderr, and over OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is set; the command flushes the exporter before it returns. So alerting on a damaged mirror is a rule in your log pipeline, not a Prometheus rule, and it depends on you collecting the run's logs: a restore whose stderr goes nowhere leaves no record once the terminal scrolls. Setting RUST_LOG replaces the default filter outright, so keep WARN enabled for siglake_wal if you set it.

Suggested alerts¶

The chart's PrometheusRule is the authoritative alert set. Siglake CI checks that every rule uses a metric exported by the build. The groups describe the operator action, not the component that emits the metric.

Alert counters whose label sets are known at startup are pre-registered at zero, so Prometheus increase() can observe the first event after a fresh pod starts. The exceptions have labels that are discovered only when the event occurs: siglake_storage_schema_drift_total{column=...} and most series of siglake_group_count_delta_write_failures_total, siglake_group_count_delta_write_retries_total, siglake_side_aggregate_publish_failures_total, siglake_group_count_auto_rebuilds_total and siglake_group_count_short_aggregates_total. Those five carry iceberg_namespace alongside table, and only {iceberg_namespace="siglake",table="events"} is pre-registered, including the automatic-rebuild counter's success, incomplete and failed outcomes and all four census outcomes. A tenant_* namespace, an index table and a base namespace moved off the default by SIGLAKE_TENANT_NAMESPACE are known only at the first increment. For a dynamically discovered series, the first increment is not visible to increase(); the schema-drift alert therefore fires on the second refusal, which the next drain cycle produces. SiglakeGroupCountDeltaRetrying reads rate() over 15 minutes and holds for 30, so it needs retries in two consecutive windows whether or not its series was pre-registered.

Data loss or durable inconsistency¶

Page on the critical rules in this group. The warning rules expose precursors or recoverable inconsistencies that still need investigation.

Stalled¶

Refusing work¶

Saturation¶

Logs¶

RUST_LOG controls filtering; the chart's logLevel default is info,siglake=info. For debugging, info,siglake=debug.

Container runtimes capture the server roles' tracing logs from stderr; the server roles write nothing operational to stdout.

SIGLAKE_LOG_EXEC_PLANS logs physical query plans: very verbose, useful when a fast path isn't engaging and you need to see what DataFusion actually built.

Query audit as an observability surface¶

query_audit is an ordinary Iceberg table, so your query history is queryable:

-- most expensive query shapes in the last day
SELECT query, count(*) AS runs, avg(duration_ms) AS avg_ms,
       max(estimated_bytes_scanned) AS max_bytes
FROM query_audit
WHERE timestamp >= now() - INTERVAL '24 hours'
  AND complexity IN ('large', 'huge')
GROUP BY query
ORDER BY avg_ms DESC
LIMIT 20;

-- rejections and errors
SELECT status, count(*) FROM query_audit
WHERE timestamp >= now() - INTERVAL '1 hour'
GROUP BY status;

The table contains literal SQL text. Treat it as sensitive and rotate it with siglake audit-rotate --max-age-secs N.