Your first queries¶

Siglake's query surface is DataFusion SQL over Iceberg tables. There is no custom query language to learn: joins, window functions, CTEs and subqueries all work here because DataFusion supports them.

This tutorial walks one query from schema to response. It assumes the quickstart stack is running on localhost:8089 and that you have sent at least one record.

1. Look at the table you are querying¶

Every OTLP log record lands in events. Its core schema is generated from the source declaration:

host comes from OTLP resource attributes, index is the logical index name, raw is the log body, and attributes holds the residual OTLP attributes as a JSON object.

A table may carry promoted columns on top of the core schema: hot attribute names lifted into typed columns of their own. See Data model and the generated table schema reference.

The table is partitioned by day(timestamp) and physically sorted by (timestamp, timestamp_ns), which is what makes time-bounded queries and ordered browses cheap.

Both time columns describe the same instant. Convert the integer sibling when you need to display the exact event time:

SELECT timestamp, to_timestamp_nanos(timestamp_ns) AS exact_timestamp,
       host, raw
FROM events
ORDER BY timestamp DESC, timestamp_ns DESC
LIMIT 100

Keep timestamp in time predicates and as the leading sort column so the partition and ordering optimizations still apply. timestamp_ns preserves nanosecond exactness and orders events that share a microsecond.

2. Run your first SQL query¶

Post the SQL to /api/v1/sql on the query server. One request returns one result: the API is stateless, so there is nothing to open or close first.

curl -s http://localhost:8089/api/v1/sql \
  -H 'Content-Type: application/json' \
  -d '{"query": "SELECT count(*) FROM events"}' | jq

Other fields you can send with the query¶

The request body takes more than query:

{
  "query": "SELECT * FROM events WHERE host = 'web-01' LIMIT 100",
  "format": "records",
  "priority": "interactive",
  "default_order": true,
  "dry_run": false,
  "limits": { "max_rows_returned": 1000 }
}

The HTTP API reference has the complete surface, including the /local, /shard, /distributed and /explain variants.

3. What the response body contains besides rows¶

Every field a records response can contain besides rows: columns, row_count, a cost estimate the planner produced before any data I/O, and a stats block describing what the query run actually did. An ndjson response carries the same fields in its trailer. The x-siglake-server-micros header gives total server-side latency, so you can separate engine time from the network round trip.

{
  "columns": ["timestamp", "host", "raw"],
  "row_count": 10,
  "rows": [ /* ... */ ],
  "cost": {
    "files_to_scan": 3,
    "files_considered": 412,
    "estimated_bytes_scanned": 8123904,
    "estimated_rows_processed": 120000,
    "estimated_runtime_seconds": 0.15,
    "complexity_class": "small",
    "exact": true,
    "warnings": []
  },
  "stats": {
    "served_by": "scan",
    "rows_scanned": 8192,
    "bytes_scanned": 2097152,
    "spill_bytes": 0,
    "phases": { "plan_micros": 1200, "collect_micros": 40100, "render_micros": 300 },
    "scan": {
      "files_planned": 3,
      "files_read": 1,
      "files_pruned_bloom": 2,
      "row_groups_considered": 24,
      "row_groups_pruned_stats": 22,
      "row_groups_read": 2,
      "ordering": "advertised"
    }
  }
}

Read cost, stats and the scan counters¶

cost is a pre-flight estimate. It comes from a manifest walk, before any data I/O. files_considered minus files_to_scan is what planning-time pruning saved you, and exact: true means the underlying statistics were exact rather than heuristic.

Read stats.served_by and stats.rows_scanned together. A tier1_* value identifies a warm metadata fast path. rows_scanned: 0 on its own does not prove there was no data-file I/O: served_by: "materialized" can read per-file footers and decode raw pages while bypassing the DataFusion scan counter. See Query engine; if the materialized fallback is slow, follow the aggregate repair guidance.

stats.scan is the pruning-effectiveness view. It tells you what the plan assigned, what each pruning mechanism dropped, and what the read actually cost. ordering: "advertised" means the scan streamed in sorted order and an ordered LIMIT early-stopped without a blocking sort; any other value is the refusal reason.

4. Try the query patterns¶

Recent events¶

SELECT timestamp, host, raw
FROM events
ORDER BY timestamp DESC
LIMIT 100

This early-stops instead of sorting the table.

Time-bounded¶

SELECT timestamp, host, raw
FROM events
WHERE timestamp >= now() - INTERVAL '1 hour'
ORDER BY timestamp DESC
LIMIT 100

Partition and manifest bounds prune to the relevant files before any I/O.

Aggregate¶

SELECT host, count(*) AS n
FROM events
GROUP BY host
ORDER BY n DESC

Whole-table group-bys can use the group-count fast path. Check both stats.served_by and stats.rows_scanned: tier1_* identifies the warm metadata path, while materialized identifies the per-file fallback, which can be I/O-heavy even when rows_scanned is 0.

Substring search¶

SELECT timestamp, raw
FROM events
WHERE raw LIKE '%connection refused%'
ORDER BY timestamp DESC
LIMIT 50

File-level trigram blooms and per-row-group token blooms prune away files that cannot contain the substring.

Full-text search¶

SELECT timestamp, host, raw
FROM events
WHERE match_terms(raw, 'timeout upstream')
ORDER BY timestamp DESC
LIMIT 50

Working with attributes¶

Residual OTLP attributes live as a JSON string in attributes and come back as text, so they CAST cleanly:

SELECT attr_get(attributes, 'k8s.namespace') AS ns, count(*) AS n
FROM events
GROUP BY ns
ORDER BY n DESC;

SELECT timestamp, raw
FROM events
WHERE CAST(attr_get(attributes, 'http.status_code') AS INT) >= 500
ORDER BY timestamp DESC
LIMIT 100;

Pull structured fields out of unstructured log text¶

SELECT kv_extract(raw, 'status') AS status, count(*) AS n
FROM events
GROUP BY status

5. UDFs for attributes and unstructured log text¶

The match_* family engages inverted-index pruning on columns that have index blobs; on other columns it falls back to row evaluation.

6. What the guardrails are, and what happens when a query exceeds them¶

Guardrails apply to every query by default, and each one refuses differently:

The query server submits completed queries to query_audit on the bounded, best-effort basis described in Audit logging.

Next¶

• SQL cookbook: deeper query patterns.
• Query engine: how fast paths, early-stop and distributed coordination work.