Skip to content

Contributing

CONTRIBUTING.md in the source repository is authoritative. This page summarizes it and links to the relevant documentation.

Build the workspace

Siglake is a Rust workspace. The toolchain is pinned in rust-toolchain.toml, and rustup picks it up automatically.

git clone https://github.com/siglake/siglake.git
cd siglake
cargo build --workspace

Run the gate

Every change must pass the local CI gate before review:

scripts/ci-local.sh

The default gate runs thirteen jobs: build-env, fmt, shell, claude-md, set-var, dashboard, test, clippy, helm, public-tree, generated, deny, and fork-tests. The generated job verifies that the committed OpenAPI specifications and both copies of the CRDs match the code. The helm and dashboard jobs verify that the chart renders across its value matrix and that every dashboard metric is emitted by the code. The fork-tests job runs the vendored forks' own unit tests, which a workspace test run does not compile.

Use the extended gate for nightly and pre-release checks. Use strict mode when a job that cannot run on the local machine should fail instead of being skipped:

scripts/ci-local.sh --all       # adds operator-cluster (kind) and docker
scripts/ci-local.sh --strict    # treat unavailable jobs as failures

During development, the three commands you will reach for most are:

cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo fmt --all -- --check

Most tests are hermetic, with no network and no cloud access. The plain workspace cargo test above needs no external services; tests that want object storage use in-memory or file:// backends.

Run the local stack

docker compose -f deploy/docker-compose.yml up --build

See Local development for the scripts, the single-process demo mode, and test-data generation.

Follow the code conventions

Review enforces these, and the compiler enforces a few.

  • Ship complete states. Each pull request compiles, is clippy-clean, and is tested. Do not park half-states behind long-lived feature branches.
  • Document new limitations. User-visible gaps go in Siglake's docs/LIMITATIONS.md and, for this site, in Limitations. Deferring something is fine; hiding it is not.
  • Keep result caches snapshot-keyed, never TTL-expired. A cache entry must be a pure function of (table, snapshot, query) and invalidate on commit. A TTL'd result cache silently serves stale leading-edge answers. See Query engine.
  • Leave physical ordering to storage. Data files are written ordered by the table's declared Iceberg SortOrder. Do not add per-component re-sorts. See Storage.
  • Name env-var knobs SIGLAKE_* and document them where they are read. The configuration reference inventory is generated from the source tree, so a new variable appears there automatically. The doc comment at the read site is what makes it understandable.
  • Hold no lock guard across .await. clippy::await_holding_lock is denied workspace-wide: a std or parking_lot guard held across an .await serializes the executor and can deadlock under cancellation. tokio::sync::Mutex is designed for it and is exempt.

Change a persisted format

Take these steps when a patch changes a footer encoding, a bloom payload, an index blob, or the WAL frame. Read docs/DESIGN_file_formats.md first: it carries the registry of every Siglake-owned format and the class each one belongs to.

  1. Decide which class the artifact is in, because the two fail differently. An accelerator (group-count footer, time-bucket footer, snapshot aggregate, layout metadata) only has to detect a bad payload and fall back to a scan. A pruning artifact (trigram bloom, inverted index) decides what not to read, so it must fail closed: an unrecognized payload is ignored entirely and never probed on a guess, because a false negative silently drops rows from a result.
  2. Bump the version for any change to bytes or meaning, including one you consider compatible. A different hash function or tokenizer produces a structurally valid artifact with different semantics, which is what a version byte exists to catch.
  3. Put the version where the format needs it, and most formats need both places. A version in the artifact's name (siglake.group_counts.v1) is the unit of replacement: a .v1 reader does not find .v2 and behaves as though the artifact were absent. A magic string plus version byte in the payload is the unit of detection: it catches a truncated blob, a foreign blob, a future writer's blob, and a name whose meaning drifted.
  4. Write exactly one version. Dual-writing the old and the new encoding gives up the saving the change was made for.
  5. Leave the files already on disk alone. Never rewrite files for format reasons: a table converges on the current format as compaction rewrites files for its own reasons, and until then old files keep working or stop being accelerated.
  6. Add the artifact to the registry table in docs/DESIGN_file_formats.md, with its class and what a reader does with an unknown version.
  7. Run the full gate and sign off, as Submit a change describes.

The Iceberg schema version (EVENTS_SCHEMA_VERSION) is a different concept with its own additive migration path, siglake migrate-schema. Changing it is not a persisted-format change.

Change a vendored fork

third_party/iceberg, third_party/iceberg-catalog-sql, and third_party/iceberg-storage-opendal are first-class forks of Apache Iceberg 0.9.1 crates, wired in through [patch.crates-io]. They are neither submodules nor pinned copies, so change one in place, in the same pull request as the code that needs it, without waiting on an upstream release.

  1. Read third_party/README.md for what already diverged in that fork and why. The divergence surface is that file plus the git history since the 0.9.1 import.
  2. Make the change as small as the behavior needs. When upstream ships an equivalent facility, prefer migrating to it and shrinking the fork.
  3. Record the new divergence in third_party/README.md. A rebase diffs the fork against the pristine crates.io package, so an undocumented change is one a later rebase can drop without noticing.

  4. Run the fork's own tests:

    scripts/check-fork-tests.sh --fork iceberg

    The forks are [patch.crates-io] path dependencies rather than workspace members, so a workspace test run compiles them without the test configuration and never builds their #[cfg(test)] modules. scripts/ci-local.sh runs the script as its fork-tests job; name no fork to run all three. 5. Run the full gate and sign off, as Submit a change describes. Fork code follows the same review, gate, and DCO rules as the rest of the workspace.

Submit a change

  1. Fork and branch from main.
  2. Make your change, with tests.
  3. Run the full gate.
  4. Sign off every commit with git commit -s.
  5. Open a pull request describing what changed and why.

Small, focused pull requests review faster than large ones.

Contributions are Apache-2.0 inbound-equals-outbound. The Signed-off-by: trailer certifies the Developer Certificate of Origin 1.1, and every commit in a pull request must carry it. There is no contributor license agreement.

Report a bug

Use the issue templates at https://github.com/siglake/siglake/issues. Run siglake --version against the binary involved and include its full output in the report; the version string includes the source commit for that build.

Security issues

Do not open a public issue for a security vulnerability. See SECURITY.md in the source repository for the disclosure process.

Contribute to these docs

The documentation site lives in its own repository.

  • The mechanical reference pages (CLI, configuration inventory, metrics inventory) are generated from the source tree by scripts/gen-reference.sh. Edit the prose around the generated blocks, not the blocks themselves.
  • Narrative pages are pitched at invariants and design intent rather than line-by-line behavior, so they survive refactors. When a design changes, the matching docs/concepts/* page is the one to revisit.

Performance claims must carry their measurement context: scale, topology, warm or cold. See Performance for why.