Skip to content

Crate map

The Siglake Rust workspace contains 13 crates. Three Iceberg forks under third_party/ are path dependencies rather than workspace members. This page maps source areas to their responsibilities.

Dependency shape

The table lists direct dependencies between Siglake crates. It omits external Rust dependencies.

Crate Direct Siglake dependencies
siglake-core None
siglake-bloom None
siglake-index siglake-bloom
siglake-wal siglake-core
siglake-storage siglake-bloom, siglake-core, siglake-index
siglake-compactor siglake-core, siglake-storage, siglake-wal
siglake-ingest siglake-compactor, siglake-core, siglake-storage, siglake-wal
siglake-query-server siglake-bloom, siglake-core, siglake-ingest, siglake-storage, siglake-wal
siglake-cli siglake-compactor, siglake-core, siglake-ingest, siglake-storage, siglake-wal
siglake-operator siglake-core
siglake-loadgen None
siglake-bench siglake-loadgen
siglake-openapi siglake-ingest, siglake-query-server

The crates

siglake-core

Shared event, schema, mapping, and tenant types.

Module Contents
lib.rs Event, the built-in events schema, and Arrow batch construction.
audit_schema.rs The query_audit schema and batch builder.
index_config.rs Managed-index mappings, field types, and validation.
mapping.rs Conversion from the carrier event batch to a managed-index mapping.
promote.rs Promoted-attribute declarations and typed extraction.
oidc.rs, tenant.rs Shared authentication and tenant identifier types.

siglake-wal

Arrow segment writing, framing, lifecycle, and object-store mirroring. consumer.rs contains SegmentConsumer, its cursor, and retention watermark.

siglake-ingest

HTTP and gRPC ingest routes that turn accepted requests into WAL appends.

Module Contents
otlp.rs OTLP log decoding and the events field mapping.
otlp_traces.rs OTLP trace decoding.
backpressure.rs Bounded tenant and index writer lanes.
rate_limit.rs In-memory and Redis token buckets.
mem_guard.rs Resident-memory circuit breaker.
lib.rs Routers, Elasticsearch bulk ingest, live stream, commit modes, and tenant middleware.

siglake-bloom

Trigram and token bloom formats. group_counts.rs contains the compact group-count footer encoding and targeted readers.

siglake-index

Builds and reads term-to-row-ordinal inverted indexes.

siglake-storage

Iceberg and Parquet storage behavior, DataFusion table registration, and managed-index state.

Module Contents
iceberg.rs Table writes, commits, level policy, aggregate metadata, expiry, retention, and garbage collection.
merge.rs In-memory, streaming, and page-bounded file merges.
query_provider.rs DataFusion scan pruning, ordering, and file sharding.
index_manager.rs Managed-index configuration and templates.
catalog_claim.rs SQL ownership records for mirrored WAL segments.
consumed_proof.rs Snapshot proof of consumed WAL segments.
subscribe.rs Incremental Iceberg subscriptions.
aws_credential.rs Object-store credential resolution.

siglake-compactor

The WAL drain and maintenance scheduler. Its single lib.rs contains commit batching, compactor roles, graded backpressure, automatic promotion, snapshot expiry, aggregate folding, and delete-task scheduling.

siglake-query-server

The HTTP query service and its execution support.

Module Contents
sql.rs SQL routes, metadata paths, limits, and complete-result caching.
coordinator.rs Distributed plan classification and result merging.
cost.rs, midflight.rs Cost estimates and the running scan breaker.
admission.rs, limits.rs Memory admission and per-tier ceilings.
wal_buffer.rs Uncommitted events rows and aggregate deltas.
jobs.rs, jobs_routes.rs Asynchronous batch jobs.
audit.rs Query audit writes.
auth.rs, oidc.rs, tenants.rs Authentication and tenant routing.
indexes_routes.rs Managed-index and template routes.
jaeger_routes.rs, jaeger_limits.rs Jaeger-compatible query routes and limits.
delete_tasks_routes.rs Delete-task API routes.
format.rs Response, scan, and phase statistics.
hot_cache.rs, udfs.rs Hot-cache table functions and SQL functions.

siglake-cli

main.rs defines the siglake binary, its server roles, and maintenance commands. It is the source for the generated CLI reference. sql_client.rs implements the interactive SQL client.

siglake-operator

The SiglakeCluster custom resource, reconciler, leader election, deployment rendering, autoscaling, and status reporting.

Tooling

Crate Purpose
siglake-loadgen Generates OTLP load and records latency histograms.
siglake-bench Runs the in-repository comparative benchmark workloads.
siglake-openapi Emits OpenAPI 3.1 documents for the ingest and query servers.

Vendored forks

Path Siglake use
third_party/iceberg Iceberg table, scan, pruning, ordered-write, and maintenance behavior.
third_party/iceberg-catalog-sql SQL catalog commits, including base-metadata updates.
third_party/iceberg-storage-opendal OpenDAL-backed storage and multipart upload controls.

These directories are path dependencies and are not listed in the workspace members. The root Cargo.toml also excludes third_party from workspace lints.

Design records

The source repository keeps design records under docs/.

Document Subject
DESIGN_file_formats.md Persisted metadata keys and versions.
DESIGN_time_ordered_storage.md Physical ordering and direction handling.
DESIGN_continuous_compaction_and_ingest.md Leveled compaction and backpressure.
DESIGN_compaction_drain_to_quiescence.md Drain scheduling and compaction disjointness.
DESIGN_distributed_query.md File sharding, classification, and merge rules.
DESIGN_inverted_index.md Inverted-index build and reads.
DESIGN_raw_content_index.md Raw-text trigram and token blooms.
DESIGN_group_count_footer_encoding.md Compact group-count footer format.
DESIGN_incremental_group_count_aggregate.md Base and delta aggregate maintenance.
DESIGN_approximate_group_counts.md Heavy-hitter sketches for wide group counts.
DESIGN_ws7_attr_get_pruning.md Rewriting attr_get predicates to promoted columns.
DESIGN_orphan_gc.md Orphan file reclamation.
DESIGN_bm25_scoring.md BM25 scoring design. It is not implemented.
DESIGN_operator_adoption.md Operator adoption of existing Helm releases.